Offset Pagination for LDAP (Lightweight Directory Access Protocol) (Java)

In my previous blog, I wrote on connecting a Java application to an LDAP server and performing your typical operations from fetching, searching, inserting and deleting users.

Manipulating an LDAP Server using Java

When we add thousands of users to an LDAP server such as ApacheDS, you will notice that it gets considerably slower when getting all the results. It is impractical to retrieve all the users with a single API call. As a remedy to that, in this article I will show you how to perform offset pagination with the data on the server.

Recently, I’ve been working with WSO2’s Identity Server which uses SCIM (System for Cross-Domain Identity Management) API to manage its users. There is no need to worry if you don’t understand what any of that means, because I’ll be detailing a general approach to perform offset-pagination, but said approach is based on the method used in WSO2’s Identity Server. The way that LDAP supports pagination, cannot be used with the Identity Server’s SCIM API because of the stateless behavior of HTTP. What I want to highlight is that this is true for any application using HTTP to make paginated requests from an LDAP server.

Lets get straight into how to get offset pagination working with an LDAP user store. Do note that you will need to have;

1. Installed an LDAP server like ApacheDS.
2. Added users into your server.

You can follow the steps I’ve laid out in LDAP — Lightweight Directory Access Protocol and Manipulating an LDAP Server using Java.

Then we just need the following methods to be able to perform pagination.

public void offsetPagedSearch(int limit, int offset) {

Hashtable<String, Object> env = new Hashtable<String, Object>(11);
env.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
/* Specify host and port to use for directory service */
//Authentication mechanism
env.put(Context.SECURITY_AUTHENTICATION, "simple");             
//Credentials - administrator's DN
env.put(Context.SECURITY_PRINCIPAL, "uid=admin,ou=system");     
//Credentials - password
env.put(Context.SECURITY_CREDENTIALS, "secret");                
env.put(Context.PROVIDER_URL, "ldap://localhost:10389");


try {
    LdapContext ctx = new InitialLdapContext(env, null);
// Number of results per page
    int pageSize = limit;
// Page number
    int pageIndex = -1;
// Cookie is automatically handled by the server.    
    byte[] cookie = null;   
    List finalUserList = new ArrayList<>();


//Here we use two controllers. PagedResultsControl signifies that we are using pagination and SortControl signifies we want the results to be sorted in ASC order based on the "uid"
    ctx.setRequestControls(new Control[]{new PagedResultsControl(pageSize, Control.CRITICAL),
            new SortControl("uid", Control.NONCRITICAL)});

    do {
//To store the intermediary result
        List tempUserList = new ArrayList<>();
/* perform the search */
// ctx.search(search base, filter, search controls)
        NamingEnumeration results = ctx.search("ou=users,ou=system",
                "(objectClass=Person)", new SearchControls());

/* for each entry add it to the tempUserList in each iteration.*/
        while (results != null && results.hasMore()) {
            SearchResult entry = (SearchResult) results.next();
            tempUserList.add(entry.getName());
        }
        pageIndex++;

//Check if these are the records we actually need based on the offset, if they are then add to the finalUserList, else discard them.
        generatePaginatedUserList(pageIndex, offset, pageSize, tempUserList, finalUserList);
        int needMore = pageSize - finalUserList.size();
        if (needMore == 0) {
            break;
        }

// Examine the paged results control response
        Control[] controls = ctx.getResponseControls();
        if (controls != null) {
            for (int i = 0; i < controls.length; i++) {
//We have given two controls. SortControl and PagedResultsControl.
//In the if condition we are checking if the control is an instance of PRRC.
//SortControl isn't, PagedResultsControl is.
                if (controls[i] instanceof PagedResultsResponseControl) {
                    PagedResultsResponseControl prrc = (PagedResultsResponseControl) controls[i];
//The cookie will be handled by the PagedResultsResponseControl
                    cookie = prrc.getCookie();
                }
            }
        } else {
            System.out.println("No controls were sent from the server");
        }

// Re-activate paged results for the next call.
        ctx.setRequestControls(new Control[]{
                new PagedResultsControl(pageSize, cookie, Control.CRITICAL),
                new SortControl("uid", Control.NONCRITICAL)});
    } while ((cookie != null) && (cookie.length != 0));
    ctx.close();
    for (int i = 0; i < finalUserList.size(); i++) {
        System.out.println(finalUserList.get(i));
    }

  } catch (NamingException e) {
      System.err.println("PagedSearch failed.");
      e.printStackTrace();
  } catch (IOException ie) {
      System.err.println("PagedSearch failed.");
      ie.printStackTrace();
  }
}

protected void generatePaginatedUserList(int pageIndex, int offset, int pageSize, List tempUserList, List finalUserList) {

    int needMore;
// Handle pagination depends on given offset, i.e. start index.
    if (pageIndex == (offset / pageSize)) {
        int startPosition = (offset % pageSize);
        if (startPosition < tempUserList.size() - 1) {
            finalUserList.addAll(tempUserList.subList(startPosition, tempUserList.size()));
        } else if (startPosition == tempUserList.size() - 1) {
            finalUserList.add(tempUserList.get(tempUserList.size() - 1));
        }
    } else if (pageIndex == (offset / pageSize) + 1) {
        needMore = pageSize - finalUserList.size();
        if (tempUserList.size() >= needMore) {
            finalUserList.addAll(tempUserList.subList(0, needMore));
        } else {
            finalUserList.addAll(tempUserList);
        }
    }
}

Let me explain the flow of the methods using a simple example. For this example, assume that and offset of 50 and a limit of 10 are passed into the offsetPagedSearch method when it is being called. That means the user wants 10 results starting after the 50th index.

1. Add the connection details and establish the connection.
2. Set the PagedResultsControls (for pagination) and SortControl (for ordering the data) and perform the search — The very first search will get the first 10 results (the amount mentioned as the limit) from the user store.
3. All the names of the users are added to the tempUserList.
4. pageIndex is incremented by 1 to show that this is the first page of the result set.
5. Call the generatePaginatedUserList method to check if these are the results we want. This is done by checking the pageIndex against the offset / pageSize. If the requirement is satisfied then exit the loop.
6. Else, use a cookie to keep track of the current position and make another call by passing in the cookie to the to create a new PagedResultControl to get the next page of results.
7. For our example, this loop will execute 5 times until an offset of 50 is reached and execute once more to get the results desired by the user.

From this scenario we can observe that even though offset-pagination is implemented, it is not as efficient as we could hope it to be. For example, an offset of 1000 and a page size of 10 would require 101 calls to be made to get the final result. A more efficient pagination solution is available in the form of cursor-based pagination and I’ll be going through how to implement it for an LDAP user store, in my next blog. I really hope this article helped you get an idea on how to perform offset pagination in LDAP and I’ll see you next time.