XPages partial refresh issue in Domino 10.0.1

A couple of days ago I tweeted about an XPages partial refresh issue I ran into in Domino 10.0.1. I got it on a freshly installed Domino 10.0.1 server with our main XPages app on it. Within a couple of clicks it started throwing errors, so I was kind of disappointed.

Did some more investigation and was able to reproduce the issue in an isolated sample .

IBM/ HCL support read my tweet and got in touch (+1 for that!). Created a case for this issue and I must honestly say that they handled very well.

Turns out that the error handling for partial refreshes has changed in v10. I didn’t know that and you probably did neither. It’s more strict in v10 and will throw an error in some specific scenarios (like my example that worked perfectly in 9.0.1).

The solution/ workaround is to revert to the way errors are handled (or are actually NOT handled 😉 )  in 9.0.1 by adding this to your XSP properties in the NSF:

xsp.error.disable.detection.set01=true

They told me that a technote that explains this in more detail is on the way.

Query Domino data with Domino JNA (part 3): REST API and infinite scroll

With the demos from part 1 and part 2 we now have a list running in XPages that has sorting, paging and filtering. But what if you want to use Domino JNA in a REST API? To serve a JavaScript/ Angular/ React/ Vue application or, why not, an app running in Office 365?

The good news is that we already have most of the code for that. So we can take the code from demo 6, do a little refactoring, write a class for the REST API, and we’re done! In this post I’ll show you how to create the JNA-powered REST API and use it in a JavaScript grid component (for Domino people: a ‘view’) called AG Grid. Want to see the demo first?

REST API in Domino

The main reason why we need to refactor the controller class from demo 6. is that REST APIs are stateless, so we need to get rid of all Java class variables. We end up with one (public static) getEntries() method and add parameters to set the start index, number of entries to return, sort order and filter.

Creating a REST API in Domino can be done in multiple ways:

For the purpose of this demo, and to keep things simple, I’ve used the ExtLib REST control. In short: you  create an XPage (e.g. api.xsp), add the ExtLib control, set the path (the part in the URL after the XPage name, e.g. ‘contacts’), configure it as a custom REST service and point it to a Java class (eu.linqed.api.ContactsService). In that class you extend com.ibm.xsp.extlib.component.rest.CustomServiceBean and implement the doService() method to handle the request. See the demo database for an example. With that in place, we now have a working REST API serving the contacts data. The data can be sorted, paged and filtered by combining URL parameters (start, count, sortCol, sortAsc and filter).

Using the REST API in an app

For this post I created a small JavaScript demo app that uses ag-grid to show the contacts. ag-grid is a very feature-rich, open source component to build grids (sorry if that sounded like marketing – I’m not affiliated in any way with them). It comes with built-in support for sorting, filtering and infinite scrolling using a virtual row model. Plus it has versions for all the major JavaScript frameworks. Since I’m the most comfortable writing Angular I used that one.

After following the getting started tutorial I ended up with a working grid, linked to their sample data. We’re now ready to change that and link it to the Domino JNA-driven REST API we just created. The documentation describes what events we can use.

Most changes need to be made in the app.component.ts file: the component definition and configuration for the grid. We’ll first change the endpoint and immediately run into an error: the Angular app is running on a different server, so the browser shows a CORS security warning. You can fix that by (1) adding an Access-Control-Allow-Origin header to your servers’ response or (2) simply run the Angular on your Domino server. When you’re running Domino behind a proxy you can add it to the proxy configuration. Or you (normally) would create rule for your website document in names.nsf. Unfortunately those rules don’t seem to be added when you’re using an ExtLib REST Control, so you either need to use a proxy (=recommended) or add the header to the ContactsService class.

We then have the change the column definition to display properties from the contacts received from the REST API in the ‘columnDefs’ property of AG Grid (lines 25-30).

Infinite scrolling on the grid is enabled by setting the ‘infinite’ row model, according to this guide. The datasource can be found in rows 102-156. The basic idea is that when the user (almost) scrolled to the end of the list, it sends a request to the REST API to load more data. With that request it includes the current ‘state’ of the grid (sort column, filtering). You can see the request being made on line 135. Note that I’m connecting to the endpoint configured in the ‘environments’ configuration file.

Last thing is to handle a user entering something in the search field or sorting the grid. We change the ‘state’ object of the grid in the appropriate events (onGridSortChanged, onGridFilterChanged) and the grid takes care of the rest!

Have fun with the demo and don’t forget the explore the source code of the demo database and Angular demo app.

 

Query Domino data and faceted search with Domino JNA (part 2): filtering

In the first post in this serie I covered the basics on how to create a list of Domino view data with pagination and sorting using the Domino JNA project. We can now get into the more interesting stuff: filtering the dataset.

Background on filtering with Domino JNA

Before diving into the code, it might be good te offer some background. Like I mentioned in the first post, Domino JNA surfaces some low level Domino (C) APIs. With those APIs it is possible to apply a selection/ filter on a view. Think of it as opening a view in the Notes client, and selecting a number of documents according to some selection criteria. The nice part about this is that it is really fast and is able to keep the sort order of the view, as well as doing pagination.

A selection is applied to a view by giving it a list of Note IDs:

Set<Integer> selectionList = new HashSet<Integer>();
collection.select(selectionList, true);

How you retrieve the list of Note IDs of documents that you want to select/ filter, is up to you: if a user wants to search for a specific last name, you can perform a view lookup to get a list of all the entries that match that name. But you can also get results from another view. Or, if you linked documents together using some ID, from related documents (e.g. performing JOIN-like queries).

Optimised lookups

To perform fast view lookups, JNA offers an optimised lookup function that performs better than the standard methods. Instead of using:

view.getColumnValues(colIndex);

You use:

collection.getColumnValues(colName, locale);

I used this optimised lookup function on the contacts list in the demo database to get a unique list of all cities and found the Domino JNA method to be about 7 times faster!

Note IDs as Integers, not Strings

When working with Domino JNA you’ll notice that it uses Integers to represent Note IDs in a database. This is done for efficiency reasons. Converting a Note ID from a String to an Integer can be done with a single line of code:

Integer id = Integer.parseInt(noteId, 16);

Filtering the contacts by cities

The demo application uses a list of 50,000 contacts. Every contact lives in a city, so let’s start with a filter on that. You could of course query the contacts by adding a Full Text index and using that, but I always run into issues. For example it isn’t really good at doing exact matches: if I only want people in city “X”, I don’t want them from “XY” (or “X Y”). Keeping the FT index up to date is also a challenge (BTW: looking forward to the changes in this area in Domino v10!).

For the cities filter I added to the third demo I used a Select2 item picker and populated that with all the available cities by performing a fast view lookup in the view (see the code below, rows 9-27). On my local VM it takes about 900 ms (for about 8,000 unique cities in the 50,000 contacts). The resulting list is cached in the applicationScope. Note that for the demo I populate the Select2 picker with a subset of all the available cities: 8,000 entries would really slow it down in the UI. In a real application you would use a server side lookup.

When the filter is applied, the list is reduced to those contacts that match the city/ cities: for every city selected in the filter, a lookup is performed to get the matching Note IDs (rows 95-99). Reading the entries that should be displayed on the current page is done in the loadEntries() function. It checks if there’s a filter applied (row 49), and if there is, it applies the list of matching Note IDs to the collection. It then iterates over the matching entries only (rows 54-56).

Adding more filters

Using this method, we can add more filters and combine the results. In demo 4 and demo 5 I added an extra filter on country (similar to the city filter) and on last name. Note that the demos do an “OR” lookup with all filters: if you select a city and country, the resulting list shows the results for the city as well as the country. The can easily be changed in the controller class by intersecting the lists of matching IDs.

The last name filter is a bit different: instead of doing a view entries/ column lookup to get an exact match, I do a lookup on a column that contains all last names and Note IDs. That list is cached in the applicationScope and used to find (partial) matches. The design of the ‘contacts’ view I used is available in the demo database.

The last en best demo (demo 6) shows how you can make filtering the list more user friendly/ intuitive: it updates the results after each change of one of the filters.

The source code for all demos is available on Github.

Finally

With just a few days ahead until Domino V10 is released, one of the first things that comes to my mind is: how does this all stack up against the new Domino Query Language. I guess that has to be the topic of a follow up post…