A couple of days ago I tweeted about an XPages partial refresh issue I ran into in Domino 10.0.1. I got it on a freshly installed Domino 10.0.1 server with our main XPages app on it. Within a couple of clicks it started throwing errors, so I was kind of disappointed.

Did some more investigation and was able to reproduce the issue in an isolated sample .

IBM/ HCL support read my tweet and got in touch (+1 for that!). Created a case for this issue and I must honestly say that they handled very well.

Turns out that the error handling for partial refreshes has changed in v10. I didn’t know that and you probably did neither. It’s more strict in v10 and will throw an error in some specific scenarios (like my example that worked perfectly in 9.0.1).

The solution/ workaround is to revert to the way errors are handled (or are actually NOT handled 😉 )  in 9.0.1 by adding this to your XSP properties in the NSF:

xsp.error.disable.detection.set01=true

UPDATE

IBM/ HCL published a technote describing the changes partial refresh behaviour in v10. You can read it here.

With the demos from part 1 and part 2 we now have a list running in XPages that has sorting, paging and filtering. But what if you want to use Domino JNA in a REST API? To serve a JavaScript/ Angular/ React/ Vue application or, why not, an app running in Office 365?

The good news is that we already have most of the code for that. So we can take the code from demo 6, do a little refactoring, write a class for the REST API, and we’re done! In this post I’ll show you how to create the JNA-powered REST API and use it in a JavaScript grid component (for Domino people: a ‘view’) called AG Grid. Want to see the demo first?

REST API in Domino

The main reason why we need to refactor the controller class from demo 6. is that REST APIs are stateless, so we need to get rid of all Java class variables. We end up with one (public static) getEntries() method and add parameters to set the start index, number of entries to return, sort order and filter.

Creating a REST API in Domino can be done in multiple ways:

• Using an XAgent (don’t want to go that way anymore)
• XPages ExtLib REST control (relatively simple)
• running a servlet in the NSF (little harder)
• running a servlet in an OSGi plugin (harder)

For the purpose of this demo, and to keep things simple, I’ve used the ExtLib REST control. In short: you  create an XPage (e.g. api.xsp), add the ExtLib control, set the path (the part in the URL after the XPage name, e.g. ‘contacts’), configure it as a custom REST service and point it to a Java class (eu.linqed.api.ContactsService). In that class you extend com.ibm.xsp.extlib.component.rest.CustomServiceBean and implement the doService() method to handle the request. See the demo database for an example. With that in place, we now have a working REST API serving the contacts data. The data can be sorted, paged and filtered by combining URL parameters (start, count, sortCol, sortAsc and filter).

Using the REST API in an app

For this post I created a small JavaScript demo app that uses ag-grid to show the contacts. ag-grid is a very feature-rich, open source component to build grids (sorry if that sounded like marketing – I’m not affiliated in any way with them). It comes with built-in support for sorting, filtering and infinite scrolling using a virtual row model. Plus it has versions for all the major JavaScript frameworks. Since I’m the most comfortable writing Angular I used that one.

After following the getting started tutorial I ended up with a working grid, linked to their sample data. We’re now ready to change that and link it to the Domino JNA-driven REST API we just created. The documentation describes what events we can use.

Most changes need to be made in the app.component.ts file: the component definition and configuration for the grid. We’ll first change the endpoint and immediately run into an error: the Angular app is running on a different server, so the browser shows a CORS security warning. You can fix that by (1) adding an Access-Control-Allow-Origin header to your servers’ response or (2) simply run the Angular on your Domino server. When you’re running Domino behind a proxy you can add it to the proxy configuration. Or you (normally) would create rule for your website document in names.nsf. Unfortunately those rules don’t seem to be added when you’re using an ExtLib REST Control, so you either need to use a proxy (=recommended) or add the header to the ContactsService class.

We then have the change the column definition to display properties from the contacts received from the REST API in the ‘columnDefs’ property of AG Grid (lines 25-30).

Infinite scrolling on the grid is enabled by setting the ‘infinite’ row model, according to this guide. The datasource can be found in rows 102-156. The basic idea is that when the user (almost) scrolled to the end of the list, it sends a request to the REST API to load more data. With that request it includes the current ‘state’ of the grid (sort column, filtering). You can see the request being made on line 135. Note that I’m connecting to the endpoint configured in the ‘environments’ configuration file.

Last thing is to handle a user entering something in the search field or sorting the grid. We change the ‘state’ object of the grid in the appropriate events (onGridSortChanged, onGridFilterChanged) and the grid takes care of the rest!

Have fun with the demo and don’t forget the explore the source code of the demo database and Angular demo app.

In the first post in this serie I covered the basics on how to create a list of Domino view data with pagination and sorting using the Domino JNA project. We can now get into the more interesting stuff: filtering the dataset.

Background on filtering with Domino JNA

Before diving into the code, it might be good te offer some background. Like I mentioned in the first post, Domino JNA surfaces some low level Domino (C) APIs. With those APIs it is possible to apply a selection/ filter on a view. Think of it as opening a view in the Notes client, and selecting a number of documents according to some selection criteria. The nice part about this is that it is really fast and is able to keep the sort order of the view, as well as doing pagination.

A selection is applied to a view by giving it a list of Note IDs:

Set<Integer> selectionList = new HashSet<Integer>();
collection.select(selectionList, true);

How you retrieve the list of Note IDs of documents that you want to select/ filter, is up to you: if a user wants to search for a specific last name, you can perform a view lookup to get a list of all the entries that match that name. But you can also get results from another view. Or, if you linked documents together using some ID, from related documents (e.g. performing JOIN-like queries).

Optimised lookups

To perform fast view lookups, JNA offers an optimised lookup function that performs better than the standard methods. Instead of using:

view.getColumnValues(colIndex);

You use:

collection.getColumnValues(colName, locale);

I used this optimised lookup function on the contacts list in the demo database to get a unique list of all cities and found the Domino JNA method to be about 7 times faster!

Note IDs as Integers, not Strings

When working with Domino JNA you’ll notice that it uses Integers to represent Note IDs in a database. This is done for efficiency reasons. Converting a Note ID from a String to an Integer can be done with a single line of code:

Integer id = Integer.parseInt(noteId, 16);

Filtering the contacts by cities

The demo application uses a list of 50,000 contacts. Every contact lives in a city, so let’s start with a filter on that. You could of course query the contacts by adding a Full Text index and using that, but I always run into issues. For example it isn’t really good at doing exact matches: if I only want people in city “X”, I don’t want them from “XY” (or “X Y”). Keeping the FT index up to date is also a challenge (BTW: looking forward to the changes in this area in Domino v10!).

For the cities filter I added to the third demo I used a Select2 item picker and populated that with all the available cities by performing a fast view lookup in the view (see the code below, rows 9-27). On my local VM it takes about 900 ms (for about 8,000 unique cities in the 50,000 contacts). The resulting list is cached in the applicationScope. Note that for the demo I populate the Select2 picker with a subset of all the available cities: 8,000 entries would really slow it down in the UI. In a real application you would use a server side lookup.

When the filter is applied, the list is reduced to those contacts that match the city/ cities: for every city selected in the filter, a lookup is performed to get the matching Note IDs (rows 95-99). Reading the entries that should be displayed on the current page is done in the loadEntries() function. It checks if there’s a filter applied (row 49), and if there is, it applies the list of matching Note IDs to the collection. It then iterates over the matching entries only (rows 54-56).

Adding more filters

Using this method, we can add more filters and combine the results. In demo 4 and demo 5 I added an extra filter on country (similar to the city filter) and on last name. Note that the demos do an “OR” lookup with all filters: if you select a city and country, the resulting list shows the results for the city as well as the country. The can easily be changed in the controller class by intersecting the lists of matching IDs.

The last name filter is a bit different: instead of doing a view entries/ column lookup to get an exact match, I do a lookup on a column that contains all last names and Note IDs. That list is cached in the applicationScope and used to find (partial) matches. The design of the ‘contacts’ view I used is available in the demo database.

The last en best demo (demo 6) shows how you can make filtering the list more user friendly/ intuitive: it updates the results after each change of one of the filters.

The source code for all demos is available on Github.

Finally

With just a few days ahead until Domino V10 is released, one of the first things that comes to my mind is: how does this all stack up against the new Domino Query Language. I guess that has to be the topic of a follow up post…

Every web application I build includes lists (or as Domino people like to call them: views) in one form of another. And then users start asking question like “But can it do sorting?” “How about a search function (on just these fields?)” “And filtering?”. With <insert-you-favorite-framework-here> that’s not too hard on short lists. I’m an Angular guy and can easily give them sorting and filtering.

But then the data grows. And you need to find a solution for paging. Or build an infinite scroll. So the fun starts: how do I get paged/ sorted/ filtered data from my Domino database? My favorite goto place for this always used to be the XPages REST controls. But they have limitations. Luckily I recently discovered an alternative: the Domino JNA project by Karsten Lehmann.

In short: it gives you an API you can use to access Domino data in ways you never did before. It does that by surfacing low-level Domino APIs that you normally don’t have access to. You’ll be amazed at all the gems hidden away in the product. One of the things I particularly like about the Domino JNA API is being able to work with Domino view data, perform JOIN-like queries  (yes, you read that right), and use the built-in sorting and pagination functions. That allowed me to build a faceted search. How I did that is the topic of this series of articles.

Note: the demos for this post are created using XPages as the front end. Everything that I write here will also benefit you when you’re creating a REST API on Domino. I might even do a follow-up article describing that.

In this first post I cover the basics: installing the plugin, setup a structure to read view data and create a first basic example of showing a sorted list of entries (from a view) using JNA.

Installation of JNA

Download the latest version of Domino JNA from the Github repo. Domino JNA is delivered as an OSGi plugin and needs to be installed on both the server and client. Restart the server and client and you’re good to go.

In the application properties of the database you want to use Domino JNA in, you need to enable it on the Page Generation tab:

Fake data database with 50,000 contacts

For testing I created a database with 50.000 contacts. The fake contacts were created by the Fake Name Generator and imported into an NSF that can be downloaded here. The NSF just contains the data and a view.

A basic list with pagination

Lets start with the basics: accessing a view and show the entries in a list. The demo can be viewed here (source code on Github) and consists of an XPage with a data table and a Java (controller) class to drive the backend. In the controller class you’ll find this:

Let’s walk through the code.

Opening a view (in JNA called a ‘collection’) is done similar as with the standard API: open the database and then the view/collection (lines 4 and 5).

Once you have the collection, you start to read entries from it. That’s done a little different then what you’re used to. You start of by specifying how to ‘walk’ the view entries. The simplest form is just move from one entry to the next using Navigate.NEXT (line 11). JNA also offers alternatives like moving to the next category or next ‘unread’ entry. More on that in another post.

Normally you retrieve a ViewEntry from a view (or view navigator) that includes all column data as well as metadata like the Note ID. You can’t control what data to read or ignore.  With JNA you can specify exactly what data to retrieve for every column. That allows you to code for performance: if you don’t need the column data, you just don’t read it. In the controller class in line 15 you can see that I’m reading the Note ID and all column values (ReadMask.SUMMARYVALUES) for each entry.

The API call on lines 21 to 24 is where the entries are actually read from the view. The skipEntries parameter allows me to start reading from a certain location in the view (useful for pagination or an infinite scrolling list) and with the NUM_PER_PAGE paramater I’m telling the API the number of entries to read in one request. The final parameter is the callback function that specifies what to do with every entry (a NotesViewEntryData object) retrieved from the view. In the example I’m using the built-in EntriesAsListCallback that returns a Java list containing the entries from the view. You can also write you own callback function to convert every entry in (for example) your own Java models.

Since I’m using XPages for the demo, I need the controller class to be serializable. The list of NotesViewEntryData object isn’t serializable, so in the last lines I’m converting the 15 entries read from the view to a list of Java maps, containing only the primitive values from the columns.

Pagination

If you look at the full code of the ListController class, you’ll see that I’m keeping track of a ‘skipEntries‘ variable. If a user navigates to the next page of data, I update this variable (incrementing it by the number of entries per page) and by adding that variable to the getAllEntries() method call I start reading at the new index. That, combined with the number of entries shown on every page allow me to add a pager.

Sorting

To enable sorting on the dataset, you need to make a change to the view design. For every column that should be sortable you enable the option “Click on column header to sort”. With that in place you can change the sort order of the view data with this API call:

Combine that with a click handler on the view columns and two variables in the controller class to store the current sort order and direction and this is what you get.

Finally

This article describes how you can start using Domino JNA to work view Domino view data. In the next article I’ll show you how you can use the built-in entry selection methods to filter the data (based on search results in the current or a different view), while maintaining the sort order and pagination functions.

(UPDATE: don’t forget to read part 2 and part 3 of this Domino JNA series)

I’m pretty happy with FP10: great that with we now have up-to-date Eclipse/ Java/ OSGi version. Despite some obvious glitches it works reasonably well for me. But if you read the blogs and follow Twitter: a lot of people are having issues. The current ‘gold’ version would have been a great beta (2) and with a couple of fixes a great release.

I can confirm that I also have most of the issues described in this blog post (and more). I also can/ really want to elaborate on them by providing more info/ screenshots just to make FP10 better.
​
​But… although I appreciate IBM (ArnazR) asking for more info in one of the more recent comments I don’t think that is the place. I also support software and absolutely hate it when people introduces new issues in a comments section.
​
​Creating PMR’s is way too much overhead. It is mentioned that a PMR exists for most issues. Can I simply add more info with just a line of text? Add a screenshot in under a minute? I don’t think so. Since I’m a consultant/ business partner and not a customer, I can’t even create PMR’s (seriously?). Don’t want to bother one of my customers for it neither.

Wouldn’t it be great if we had a simple way to log the issue we’re having with FP10? Just have a look at the Github issues section for inspiration. ​The delivery model for Notes/ Domino has changed in light of what the rest of the world is doing. I think it’s time the support model follows.

PS. For ArnazR: have a look at the screenshot. I would also say that this help screen looks ‘curious’).

I’m into performance at the moment, trying to solve some nasty issues in one of the applications I work on. While researching the topic I came across Karsten Lehmann’s Domino JNA project that allows you to use some low-level C-API methods using Java. It contains functions that are very useful in my scenario, but I also wondered how the library performed. So I wrote some basic tests, comparing JNA with the ‘standard’ (or ‘legacy’) Domino API and, while I was at it, the org.openntf.domino (ODA) API.

So I wrote a couple of tests using the 3 APIs that traverse a view from the well known ‘Fakenames’ database. It is based on the Domino Name & Address book template and contains 40,000 documents. The tests will loop through a view called ‘People’ containing all the documents and will read a value from one of the columns.

At this point I will be taking bets:

 Loading ...

Time to see for yourself! I created a simple application that allows you to test the different methods and view the results (latest 10 are shown only): take a look here.

The source code of the application and test code can be found here. The environment: CentOS 6 (64 bit), SSD, 1GB RAM, Domino 901FP5 (64 bit), ExtLib 17, ODA 3.2.1, JNA 0.9.5. Please let me know if I made any errors in the tests.

I don’t know what about you, but the results surprised me! I didn’t expect the overhead of the non-standard APIs to be that big. In every day use I guess this won’t affect your applications a lot, but it’s something you definitely have to be aware of if you needs to squeeze just a bit more performance out of your application.

UPDATE

– Upgraded the JNA project to 0.9.5
– Based on Andy Cunliffe’s comment I’ve added a new test using a ‘manual’ Java loop in the ODA:

ViewEntry ve = nav.getFirst();
while (null != ve) {

// code here

ve = nav.getNext();
}

That code seems to run about 30% faster that a standard Java loop in ODA:

for (ViewEntry ve : nav) {

// code here

}

I was doing some work with an Angular application talking to the IBM Connections API. More specifically: I wanted to show and create activities based on some user input.

The Connections API is pretty complex, so I normally run some ‘manual’ tests first based on the documentation, and then use those result to write the code to call the API. My preferred application for that is Postman.

While running the tests in Postman I ran into a big issue. All GET and PUT requests came through fine, but I wasn’t able to create anything using a POST request to the API: every request I made returned a 403 error:

<error xmlns="http://www.ibm.com/xmlns/prod/sn">
<code>403</code>
<message>You are not authorized to perform the requested action.</message>
<trace></trace>
</error>

The funny thing was that using the same credentials, I could create items using the web interface just fine. My first thought was that it must be some strange access control setting hidden away deep in a config file (“don’t allow users to create stuff using the API”), but then I found someone with a similar issue on StackOverflow. And I found this IBM Technote. So, apparently there’s something fishy going on with Postman.

So by enabling the Chrome Developer tools for Postman, I was able to look at the exact HTTP request that Postman sends. And I found the Origin header that was already mentioned in the StackOverflow post:

Origin: chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop

Turns out that IBM Connections doesn’t really like that. That’s by design and has something to do with cross-site request forgery.. Using a curl command I could easily bypass the header and confirm that it was indeed the cause of my problem.

So the only thing left (since I like testing with Postman), was to figure out how to change that Origin header. The header is one of the ‘restricted’ headers that is automatically added by the browser (Chrome) and can’t be easily changed. Luckily, the people at Postman also thought of that and created the “Postman Interceptor“: a Chrome extension that sits as a sort of proxy between your Postman requests. After enabling that in Postman I was able to change the Origin header of my requests. I set it to the hostname of the IBM Connections server and voila: happy times!

Over the past couple of weeks, Visual Studio Code has become my favorite editor. It has some great features, is fast and relatively easy to use. Of course I’m still learning how to use it most effectively. Here’s what I learned today.

Starting Visual Studio Code from the terminal

Visual Studio Code has a built-in function to add a shortcut to it in your OS X environment. That allows you to start it from a terminal. The procedure is described here and comes down to:

• Open Visual Studio Code
• Bring up the Command Palette ( Cmd – Ctrl – P ) and type “shell command”
• Execute the ‘Shell Command: Install ‘code’ command in PATH’ command

When you’ve done that, you can simply type ‘code‘ in your terminal to start Visual Studio Code. If you want to open a specific file with Code, type ‘code <filename>’.

Note: Windows users get this feature automatically when installing Code.

Open a file directly from a repo in SourceTree

SourceTree has a function that allows you to define ‘Custom Actions’ that can be started from files in any repo. If you’ve added the ‘code’ command from the previous step, you can create a Custom Action to open any selected file with Visual Studio Code:

• Bring up the SourceTree preferences (Cmd + comma)
• Go to the Custom Actions tab and configure it like this (OS X users):

Note that Windows users need to configure the custom action with the full path to the Code executable:

If you now select a file in your repo, you can right click > Custom Actions > Edit in Visual Studio Code to immediately open a file with Code.

Happy coding!