Month: October 2018

In the first post in this serie I covered the basics on how to create a list of Domino view data with pagination and sorting using the Domino JNA project. We can now get into the more interesting stuff: filtering the dataset.

Background on filtering with Domino JNA

Before diving into the code, it might be good te offer some background. Like I mentioned in the first post, Domino JNA surfaces some low level Domino (C) APIs. With those APIs it is possible to apply a selection/ filter on a view. Think of it as opening a view in the Notes client, and selecting a number of documents according to some selection criteria. The nice part about this is that it is really fast and is able to keep the sort order of the view, as well as doing pagination.

A selection is applied to a view by giving it a list of Note IDs:

Set<Integer> selectionList = new HashSet<Integer>();
collection.select(selectionList, true);

How you retrieve the list of Note IDs of documents that you want to select/ filter, is up to you: if a user wants to search for a specific last name, you can perform a view lookup to get a list of all the entries that match that name. But you can also get results from another view. Or, if you linked documents together using some ID, from related documents (e.g. performing JOIN-like queries).

Optimised lookups

To perform fast view lookups, JNA offers an optimised lookup function that performs better than the standard methods. Instead of using:

view.getColumnValues(colIndex);

You use:

collection.getColumnValues(colName, locale);

I used this optimised lookup function on the contacts list in the demo database to get a unique list of all cities and found the Domino JNA method to be about 7 times faster!

Note IDs as Integers, not Strings

When working with Domino JNA you’ll notice that it uses Integers to represent Note IDs in a database. This is done for efficiency reasons. Converting a Note ID from a String to an Integer can be done with a single line of code:

Integer id = Integer.parseInt(noteId, 16);

Filtering the contacts by cities

The demo application uses a list of 50,000 contacts. Every contact lives in a city, so let’s start with a filter on that. You could of course query the contacts by adding a Full Text index and using that, but I always run into issues. For example it isn’t really good at doing exact matches: if I only want people in city “X”, I don’t want them from “XY” (or “X Y”). Keeping the FT index up to date is also a challenge (BTW: looking forward to the changes in this area in Domino v10!).

For the cities filter I added to the third demo I used a Select2 item picker and populated that with all the available cities by performing a fast view lookup in the view (see the code below, rows 9-27). On my local VM it takes about 900 ms (for about 8,000 unique cities in the 50,000 contacts). The resulting list is cached in the applicationScope. Note that for the demo I populate the Select2 picker with a subset of all the available cities: 8,000 entries would really slow it down in the UI. In a real application you would use a server side lookup.

When the filter is applied, the list is reduced to those contacts that match the city/ cities: for every city selected in the filter, a lookup is performed to get the matching Note IDs (rows 95-99). Reading the entries that should be displayed on the current page is done in the loadEntries() function. It checks if there’s a filter applied (row 49), and if there is, it applies the list of matching Note IDs to the collection. It then iterates over the matching entries only (rows 54-56).

Adding more filters

Using this method, we can add more filters and combine the results. In demo 4 and demo 5 I added an extra filter on country (similar to the city filter) and on last name. Note that the demos do an “OR” lookup with all filters: if you select a city and country, the resulting list shows the results for the city as well as the country. The can easily be changed in the controller class by intersecting the lists of matching IDs.

The last name filter is a bit different: instead of doing a view entries/ column lookup to get an exact match, I do a lookup on a column that contains all last names and Note IDs. That list is cached in the applicationScope and used to find (partial) matches. The design of the ‘contacts’ view I used is available in the demo database.

The last en best demo (demo 6) shows how you can make filtering the list more user friendly/ intuitive: it updates the results after each change of one of the filters.

The source code for all demos is available on Github.

Finally

With just a few days ahead until Domino V10 is released, one of the first things that comes to my mind is: how does this all stack up against the new Domino Query Language. I guess that has to be the topic of a follow up post…

Every web application I build includes lists (or as Domino people like to call them: views) in one form of another. And then users start asking question like “But can it do sorting?” “How about a search function (on just these fields?)” “And filtering?”. With <insert-you-favorite-framework-here> that’s not too hard on short lists. I’m an Angular guy and can easily give them sorting and filtering.

But then the data grows. And you need to find a solution for paging. Or build an infinite scroll. So the fun starts: how do I get paged/ sorted/ filtered data from my Domino database? My favorite goto place for this always used to be the XPages REST controls. But they have limitations. Luckily I recently discovered an alternative: the Domino JNA project by Karsten Lehmann.

In short: it gives you an API you can use to access Domino data in ways you never did before. It does that by surfacing low-level Domino APIs that you normally don’t have access to. You’ll be amazed at all the gems hidden away in the product. One of the things I particularly like about the Domino JNA API is being able to work with Domino view data, perform JOIN-like queries  (yes, you read that right), and use the built-in sorting and pagination functions. That allowed me to build a faceted search. How I did that is the topic of this series of articles.

Note: the demos for this post are created using XPages as the front end. Everything that I write here will also benefit you when you’re creating a REST API on Domino. I might even do a follow-up article describing that.

In this first post I cover the basics: installing the plugin, setup a structure to read view data and create a first basic example of showing a sorted list of entries (from a view) using JNA.

Installation of JNA

Download the latest version of Domino JNA from the Github repo. Domino JNA is delivered as an OSGi plugin and needs to be installed on both the server and client. Restart the server and client and you’re good to go.

In the application properties of the database you want to use Domino JNA in, you need to enable it on the Page Generation tab:

Fake data database with 50,000 contacts

For testing I created a database with 50.000 contacts. The fake contacts were created by the Fake Name Generator and imported into an NSF that can be downloaded here. The NSF just contains the data and a view.

A basic list with pagination

Lets start with the basics: accessing a view and show the entries in a list. The demo can be viewed here (source code on Github) and consists of an XPage with a data table and a Java (controller) class to drive the backend. In the controller class you’ll find this:

 

Let’s walk through the code.

Opening a view (in JNA called a ‘collection’) is done similar as with the standard API: open the database and then the view/collection (lines 4 and 5).

Once you have the collection, you start to read entries from it. That’s done a little different then what you’re used to. You start of by specifying how to ‘walk’ the view entries. The simplest form is just move from one entry to the next using Navigate.NEXT (line 11). JNA also offers alternatives like moving to the next category or next ‘unread’ entry. More on that in another post.

Normally you retrieve a ViewEntry from a view (or view navigator) that includes all column data as well as metadata like the Note ID. You can’t control what data to read or ignore.  With JNA you can specify exactly what data to retrieve for every column. That allows you to code for performance: if you don’t need the column data, you just don’t read it. In the controller class in line 15 you can see that I’m reading the Note ID and all column values (ReadMask.SUMMARYVALUES) for each entry.

The API call on lines 21 to 24 is where the entries are actually read from the view. The skipEntries parameter allows me to start reading from a certain location in the view (useful for pagination or an infinite scrolling list) and with the NUM_PER_PAGE paramater I’m telling the API the number of entries to read in one request. The final parameter is the callback function that specifies what to do with every entry (a NotesViewEntryData object) retrieved from the view. In the example I’m using the built-in EntriesAsListCallback that returns a Java list containing the entries from the view. You can also write you own callback function to convert every entry in (for example) your own Java models.

Since I’m using XPages for the demo, I need the controller class to be serializable. The list of NotesViewEntryData object isn’t serializable, so in the last lines I’m converting the 15 entries read from the view to a list of Java maps, containing only the primitive values from the columns.

Pagination

If you look at the full code of the ListController class, you’ll see that I’m keeping track of a ‘skipEntries‘ variable. If a user navigates to the next page of data, I update this variable (incrementing it by the number of entries per page) and by adding that variable to the getAllEntries() method call I start reading at the new index. That, combined with the number of entries shown on every page allow me to add a pager.

Sorting

To enable sorting on the dataset, you need to make a change to the view design. For every column that should be sortable you enable the option “Click on column header to sort”. With that in place you can change the sort order of the view data with this API call:

Combine that with a click handler on the view columns and two variables in the controller class to store the current sort order and direction and this is what you get.

Finally

This article describes how you can start using Domino JNA to work view Domino view data. In the next article I’ll show you how you can use the built-in entry selection methods to filter the data (based on search results in the current or a different view), while maintaining the sort order and pagination functions.

(UPDATE: don’t forget to read part 2 and part 3 of this Domino JNA series)