Scheduling XPages / OSGi Plugin Code – Part One

Before Christmas I blogged a teaser about running Xots tasks scheduled. Various demos have been constructed and a video will be coming, but I decided to blog about the components I would recommend to a customer. This is very much a modular approach with a lot of BYO components. But therein lies its strength, in my opinion:

• Microservice is the buzzword of the decade, for a very good reason
• It doesn’t reinvent the wheel
• It allows developers to use best of breed or familiar
• It avoids proprietary, to a large extent
• Data is passed around using standard formats which XPages / web application developers either should already be familiar with or should be wanting to become familiar with
• Security is considered from the outset
• If the scheduler can receive a REST service, it’s possible to chain jobs within the scheduler

The approach recommended comprises three elements:

1. A scheduler that can call a REST service with whatever authentication is required. If it can receive a REST service, this adds additional opportunities.
2. A REST service endpoint that can be stored in the same location as the rest of your application, can interact with Domino applications and can, with minimal effort, re-use common code used for the rest of your application.
3. Where asynchronous processing is required, a background task.

REST Service Endpoint

As with other components, there are a variety of options here. The starting point was the XPages REST Service component. But investigation showed it was not straightforward to access the component externally via a standard URL. The component is designed to load JSON data for use elsewhere on the same XPage. It’s not designed to load and return JSON data for use elsewhere.

There are a variety of options to achieve the required goal of running SSJS or Java code and returning JSON:

• SmartNSF with the NSF application. SmartNSF provides various coding options. Authentication is managed by the NSF’s ACL.
• An OSGi custom REST service, as I’ve blogged about before and for which I set up an Apache-licensed starter servlet on OpenNTF. This will typically be coded in Java. Authentication is managed however you choose and can use an NSF’s ACL.
• An XAgent. Actually, this was the approach I used in XPages Help Application over 6 years ago. There I manually generated the JSON as a string. Since then there are much better options. This can be coded in SSJS or Java. Again, authentication is managed by the NSF’s ACL. Remember to set viewState=”nostate” – there’s no point storing component trees for each session, because each session is unique and will never interact again with that same component tree.
• I haven’t dug into Darwino microservices, but they may also work.
• Agents. Not an approach I’d use, but one that could work. Again, authentication is managed by the NSF’s ACL.
• Whatever other REST service options for running custom code may be developed or may already be in use. If NodeJS access is available to the Domino server – there is an OpenNTF Slack channel where people are using NodeJS on Domino – coding could I guess be in JavaScript.

As I’ll show when I got through the demo code, there are ways to make this easier.

Asynchronous Processing

The limitation of the above options is that the REST services do not run asynchronously. That is, they are called, run code in real time and only after completion do they return a JSON response. For asynchronous processing you need a background threaded task. There is sample code in the Threads and Jobs project on OpenNTF. Or you can use Xots. As I mentioned on my previous blog post, I’ve had Xots running in production with no issues for several years. I’ve used it extensively in a recent project for multi-threaded background processing.

Scheduler

The choice I would recommend and use for the scheduler is Node-RED. Yes, it’s new for many. But:

• The editor can be locked down securely
• It’s got a web-based drag-and-drop GUI
• It may be new to many, but it’s standard amongst web development
• Flows can be exported as JSON
• It can be run in the cloud (including IBM Cloud), on premises or as a Docker image
• REST services can be triggered on schedule or on startup
• They can easily be coded to add header parameters or use Basic Authentication
• Scheduling is available out-of-the-box, but if you want something with greater flexibility you can use an additional module Big Timer

For those new to it, John Jardin has a great series about getting started with Node-RED and there is a course on Coursera by IBM which John pointed me in the direction of. Although I am sure there are advanced elements, the basics are very straigtforward. The documentation answered any questions I had about setup, configuration, and using the nodes.

I’m sure there are a number of alternative schedulers that could be used. All it needs to do is allow scheduling of a HTTP/S call to your Domino server, with Basic Authentication if you’re going to a secured NSF. The username and password to connect just need to be entered into the HTTP request. Note that if you re-open the node, like any decent password field on the web, the password is not displayed. It seems to be stored encrypted in a JSON file. So it’s no less secure than what I’ve used previously in LEI (Lotus Enterprise Integrator).

If the scheduler can also received HTTP requests, you have the potential to connect up the dots and chain jobs.

Connecting The Dots

So Node-RED can have an inject node that triggers an HTTP request to your Domino microservice. That microservice can run in real time or kick off a background task. On completion, the microservice or background task can trigger an HTTP request back to Node-RED, which triggers another HTTP request to a subsequent Domino microservice. By including success / failure / other data in the HTTP request to Node-RED, a subsequent Node-RED node could parse the JSON passed into the request to determine which of a variety of subsequent Domino microservices to trigger.

Summary

In subsequent parts and the video, I’ll step through some sample code.

But hopefully you can already see there is a lot of flexibility here that comes from splitting the stack and a lot of functionality that would take quite some time to build from scratch. That’s why I would not advocate a separate scheduler being developed for Domino. Maybe someone’s already using a similar approach, but I’ve not seen it blogged about. Maybe someone’s already built such a scheduler for Domino, but I’m not aware of one available for free or paid. Maybe someone will be able to construct a justification for building one themselves, in which case I’d recommend OpenNTF as a repository for such a scheduler. But there’s a wide breadth of functionality, and that’s before even considering other Node-RED nodes that could add to the functionality I’ve mentioned above, like the watch node, various Watson nodes, a project I accidentally came across for displaying the schedule of inject nodes etc. Personally I have no interest in myself or IBM reinventing this wheel. In my opinion, there are much better projects to spend time on.