Blog

Domino and Jar Files

  |   Blog   |   6 Comments

Those of us who have been working with Domino and Java for some time know that one of the big strengths over LotusScript is the host of standard Java libraries that can be utilised. If you want better string or number handling etc, you have Apache Commons libraries. If you want Excel export, you have Apache POI. If you want conversion from JSON to Java objects, there’s Jackson. And there are a host of others.

But how you use them is slightly less straightforward. In many Java application development environments a build management tool like Maven or Gradle is used. This has the benefit of downloading not only the relevant jar file but all relevant dependencies. That then requires two options for deployment and running. With something like Vert.x or Spring Boot, if the host for the application also has the relevant software, it’s possible to retrieve the dependencies at runtime. This is what is typically done when developing Java applications via an embedded web server. The other approach in Vert.x or Spring Boot is to package the application as a “fat jar” – the application plus all dependencies packaged at compile time into a single jar or war file. The application is a single discrete file, but could include dependencies that are also included in other applications, hence the term “fat jar”.

OSGi, the basis of XPages, works differently. The approach, popular with large enterprise systems, is intended to minimise the duplication by defining dependencies and relying on the administrator of the relevant server to ensure the plugins an application is dependent upon have been added. The downside is that dependencies could be nested at various levels and conflicts could occur if jar files are embedded in multiple plugins, particularly if they are exposed in the wrong manner. As you start adding more and more OSGi plugins – particularly custom ones – the risk of conflicts increases.

But it means dependencies should not occur more times than necessary.

This became more apparent recently when I happened to look at the temp folders on a server. Typically I’ve included Apache Commons Lang and others within an NSF. This can be done by using the Jar design element or manually by adding them to a folder (e.g. WebContent\WEB-INF\lib), right-clicking them and adding them to the build path. But this is what happens when you do either of those:

The four timed 12:58 are the jar files from a single NSF. The four timed 12:59 are the same jar files in a different NSF. The one timed 13:00 is a Jar design element from a third NSF. The code that’s running doesn’t necessarily use them. Indeed, for the Jar resource I just added it to the NSF and previewed a blank XPage in the application.

This may seem alarming, but when you think about it, there is no alternative. An NSF is a single file. The jar file is a file resource within that NSF. So obviously it needs to be detached to the file system in order for the server to use it. I’ve heard questions about why Domino doesn’t allow Maven to incorporate jar files and their dependencies in an NSF, but that would still require the jar files to be stored somewhere temporarily.

This highlights also the importance of admins ensuring temp drives are appropriately sized. Restricting them to the minimum could impact things, even if the server’s xsp properties are set to ensure XPages temporary file uploads, persistence files and component trees are stored on a separate drive.

Obviously, OSGi avoids the need for this completely. So it highlights further the benefits of packages jar files into an XPages extension library (a.k.a. OSGi plugin). That’s what I’ve done for a recent plugin but there are ways to leverage Maven as well, to minimise manual steps and avoid missing dependencies. And its standardisation is to be lauded, as I’ll show in an upcoming blog post.

One final comment on Java agents. It’s possible to add jar files to a Script Library or Java agent. But similarly those need detaching – both at design time and run time. Just to reinforce that, next time you open a Java agent, look at the Package Explorer in Domino Designer. You’ll see a separate Java project is created on-the-fly with the relevant hierarchy within it. Recommendations from developers tend to be to put any jar files on the server in the jvn/lib/ext folder. If you search, there are a couple of technotes for memory leaks when the jar files are detached at runtime and not fully cleaned up. Whether or not they’re fully resolved I don’t know, most Java developers avoid adding jar files to an agent so it’s probably not widely used in order to tell.

For all of this, the nirvana for Java developers is to have a scheduled process that can be coded in Java re-using code already developed for XPages without the need for reduplication. Whether you use agents or DOTS, there’s still a need for duplication (DOTS uses OSGi, but from a different folder, and if you code youor shared logic in a plugin then you will need to restart HTTP before seeing any change as well as importing the plugin into Design to use any new methods from Java in the NSF, so that’s not ideal either). There’s no easy answer, if you’re coding Java in an NSF, and I’ve wrestled with it for some time. We’ll see if cleverer minds than me can come up with a solution in the future.

AUTHOR - Paul Withers

Paul Withers has been an IBM Champion since 2011, has been an OpenNTF Board Member since 2013, has worked with XPages since 2009, co-authored XPages Extension Library and was technical editor for Mastering XPages 2nd Edition. He is one of the developers on OpenNTF Domino API as well as contributor to a variety of other OpenNTF projects.

6 Comments
  • Karsten Lehmann | Nov 28, 2017 at 3:16 pm

    “Obviously, OSGi avoids the need for this completely”
    Right, unfortunately, IBM loads the plugin jars from their Update Site databases and has to extract them to disk on each HTTP task restart, not deleting previously extracted temp files.

    We reported this in June 2017 via the Design Partner forum, the SPR is MKINAQ3T4C, our customer has added some weight by creating a PMR.
    Here is the error report that I sent to IBM:

    com.ibm.domino.internal.protocol.osginsf.OSGiNSFUrlConnection filling up disk space of Domino Server

    The class com.ibm.domino.internal.protocol.osginsf.OSGiNSFUrlConnection located in the archive /jvm/lib/ext/xsp.http.bootstrap.jar is the protocol handler for osginsf:// URLs that are used to load plugins from update site databases on Domino server startup.

    This class opens the update site database, finds the right plugin JAR document, opens the document’s file attachment as an EmbeddedObject and returns an InputStream to its data, calling EmbeddedObject.getInputStream().

    This EmbeddedObject.getInputStream() in the Notes.jar internally extracts the attachment to a temp directory (in our case c:\windows\temp) and returns a stream for it, which has always been a very bad implementation (using C API it would be no problem to directly stream the data and release open handles when the stream gets closed).

    The golden rule has always been to first close the InputStream and THEN recycle the EmbeddedObject, otherwise the temporary files with names like eo*tm stay on disk.

    The problem is that the class OSGiNSFUrlConnection does not do this. Instead it quickly recycles the EmbeddedObject in a try/finally statement while the InputStream is still open, not keeping track of the temp files that have been created.

    In the end (after several Domino restarts) all the disk space is gone and the HTTP OSGi system cannot start anymore.
    In our case, 22 GB of plugin garbage have been created this way, because there are some big plugins deployed on our customer’s server.

    I would suggest to change Notes.jar so that at least the file names of temp files are logged somewhere. On the next JVM startup, this list can then be used to clean up that mess.

  • Jesper Kiaer | Nov 28, 2017 at 5:31 pm

    It has always baffled me why Notes/Domino would create new versions of JAR files in agents in the file system every time it is run. A simple calculation of hash values for the files in the agent (saved in agent) and in the files system should clarify whether it is needed to detach any files

  • Lars Berntrop-Bos | Nov 28, 2017 at 11:24 pm

    Yuk, that means the recommended way of using an updatesite nsf for the OpenNTF libraries will generate tmp files.
    Furthermore, which directory issues that on IBM I?
    I now have some extra to-do items on my work list for tomorrow:
    1 check domino designer machine for tmp
    2 check servers
    3 check a couple of Java agents
    4 think about workarounds

    Lars

  • Karsten Lehmann | Nov 29, 2017 at 7:16 am

    @Lars:
    Don’t know about IBM I, but on Windows64 the files are stored in c:\Windows\Temp.

  • Lars Berntrop-Bos | Nov 29, 2017 at 7:39 am

    @Karsten: you are repeating yourself.

  • Lars Berntrop-Bos | Nov 29, 2017 at 2:22 pm

    Hmm. Opened a Domino Designer 9.0.1 and opened a big Java agent with several jar file contained within the agent. Searched my file system. *NO* temporary jar files found (with regex pattern jar,+\.tmp), aside from some installer jars in C:\Users\\AppData\Local\Temp)
    The jars contained in the agent are found (using regex .+\.jar) in the directory \workspace\.\archive, which suggests a temporary ODP for editing the Java agent. => the Domino Designer seems to efficiently manage this.
    When viewing an XPinC app or previewing an XPage: same. Logical, since my Notes client has the extensions installed, and is not loading them from the updatesite.nsf dynamically.

    Is it possible to make the Notes client load from an updatesite?

    Another less painful solution would be if you could fire an agent to update the Domino OSGI plugins from the updatesite, sort of like the Notes client OSGI update. Ideally, the code in xsp.http.bootstrap.jar would use the plugin on disk if present, and load dynamically if the updatesite is newer.

Post A Comment