I suspect I am far from alone in coming to XPages with no experience of Java development. I’d done lots of LotusScript, a reasonable amount of JavaScript and HTML, and even a bit of XML. But I knew virtually nothing about Java (apart from the basics of syntax and that Java classes have a “public static void main()” function). I knew nothing about JSF. As far as I was aware I didn’t need to. Domino took my controls, Simple Actions, @Formulas and Server Side JavaScript and converted them to Java for me.
Yes there were times when a little JSF knowledge was useful. Like understanding the JSF lifecycle, what I needed to partially refresh to get things working, why validation needed to happen before I could update things (prior to 8.5.2). And understanding the facesContext object and other JSF objects. But for Server Side JavaScript, the syntax is the same as JavaScript and the Domino objects are the same as LotusScript.
Or so I thought until last week. I had an 8.5.1 FP3 server crashing with out of memory issues. I had not changed the code on my XPage, but it had started crashing the server. The cause was a knowledge of LotusScript and a lack of knowledge of Java.
Let’s step back here. In LotusScript when your code or agent finishes, all your Domino objects get removed from memory. As a developer you do…nothing. As developers we just throw our objects away and Domino automatically removes the underlying C++ objects. In Java, however, I’ve learned that is not the case there. Instead developers need to be greener. We need to call the recycle() method once we’ve finished with an object to clear the memory.
My code was looping through a large collection of documents (over 18000 documents!). From LotusScript I assumed that when I called doc = dc.getNextDocument(doc) the code overwrote my doc variable with the new NotesDocument object. However, that’s not the case. It creates a new instance of doc, leaving the previous NotesDocument object still in memory. (I still haven’t got my head around this. Surely the first NotesDocument cannot be referenced by the variable doc, so if you reassign doc to a new NotesDocument, why is it still keeping the first NotesDocument in memory?)
To clear the Domino object from memory there is the recycle() method. So you need to construct your code like this:
-
var tmpDoc:NotesDocument;
-
var doc:NotesDocument = dc.getFirstDocument();
-
while (doc != null) {
-
//processing here
-
tmpDoc = dc.getNextDocument(doc);
-
doc.recycle();
-
doc = tmpDoc;
-
}
So line 5 stores the new NotesDocument object in a temporary variable, because we still need a handle on the first document to get the next one. Line 6 clears the first document from memory. Line 7, if I understand it correctly, moves tmpDoc into the doc variable. I’m assuming that because tmpDoc is moved to the doc variable in line 7, we don’t need to recycle it. I’m further assuming that if we were to recycle tmpDoc, doc would also become null.
Unfortunately this is a prime example of one of the main frustrations of new XPages developers. If you look in the Help for NotesDocument, you will find no reference to the recycle(). There is, however, a separate page on recycle() which says it is an inherited method for virtually all Domino objects. But there’s nothing to tell a non-Java developer how to use it or when to use it (and when not to use it). If any Java or more experienced XPages developers can clarify, it would be appreciated by me and probably other Domino developers. The technote I was referred was somewhat old and referred only to LotusScript and Java. This was my only reference for not calling recycle on tmpDoc after line 7.
It was my only reference until I searched the Lotus Notes and Domino Application Development wiki this evening and found an article documenting the properties and methods of the NotesDocument object posted on 17 December 2010. This is a perfect example of the kind of documentation developers are used to from LotusScript and have been asking for and shows how to use recycle() when looping through a NotesDocumentCollection. Unfortunately, it doesn’t reference recycle() as a method of NotesDocument, which will hopefully be remedied. But I am hopeful that this is the first of many articles which will be invaluable to developers and which will, I am sure, make their way into the Help. I for one will be looking to the wiki regularly to see pages about other Domino objects in XPages and maybe learning more that I didn’t know about the differences between Domino objects in XPages and Domino objects in LotusScript.
Bob Balaban had a great series of blogs on this topic.
{ Link }
Thanks, it’s a useful set of articles. Particularly interesting is this sentence: “What makes this unfortunate, from a product point of view, is that developers have to know to do this [recycle()], and have to know how to do it correctly, so that they don’t mess theselves up accidentally.”
I also went on to the second article, about multi-threading: { Link }
This throws up a question that comes back to how do you do it correctly. If you’re XPage refers to a NotesDocumentCollection or a NotesDatabase several times in SSJS, does each piece of SSJS run on its own thread or is there a single thread for the XPage? If a single thread for the XPage, where should you (best practice) recycle the objects? If they run in their own thread, which I suspect, does that cause problems with TLS (thread local storage)? And so again, where should the object be recycled?
If anyone knows, that would be a very appreciated blog post.
The wiki had a mishmash of examples, some with and some without recycle methods.
I took the liberty of adding recycle calls where appropiate.
And since I am not perfect, can you please look at that Wiki article and make sure I have not made matters worse?
Thanx!
@Lars I’d love to and I’ll have a look. But I’ve spent this morning adding recycle() to my XPages (4 in total), testing them, then trying to work out where the “NotesException: Object has been removed or recycled” errors were being triggered from, then fixing it because objects weren’t getting recycled in my error blocks which were being hit. Obviously I’m not the best person. It would be great for an expert to write a best practice document about when and where objects should be recycled, something akin to Bob Balaban’s posts. I intend to ask at the labs at Lotusphere. If there’s nothing out sooner, I’ll blog then.
To be clear, this problem exists with Lotusscript as well, it’s just harder to make it manifest itself. Instead of looping through a regular Collection, try putting the Documents in a List structure instead. You’ll see it show up there.
“If you’re XPage refers to a NotesDocumentCollection or a NotesDatabase several times in SSJS, does each piece of SSJS run on its own thread or is there a single thread for the XPage?”
All SSJS that must be evaluated to respond to a single request (whether full or partial) is run in one thread.
“If a single thread for the XPage, where should you (best practice) recycle the objects?”
The best practice, as you’ve encountered, is inside whatever loops you’re dealing with in SSJS. You are unlikely to have 1000s of different fields accessing 1000s of independent documents. Handle limits are 99.999% of the time reach by virtue of iterations and recursions.
“If they run in their own thread, which I suspect, does that cause problems with TLS (thread local storage)?”
When an HTTP request come in from the browser, the job of processing the request and populating the ResponseWriter is given to a single thread, which is why thread local storage never presents a problem in any single response.
“And so again, where should the object be recycled?”
If you aren’t doing batch document processing as in your examples, you don’t really need to focus on recycle(). The XPages servlet automatically .recycles() the Session after each request, so thread local handles are recycled on the thread that originally requested them.
This is why you can’t put a Document or Database handle into a SessionScope variable. It will be .recycled automatically when the page response is complete. It has to be, because the only way to release the C++ object handle is to do so on the same thread, and there’s no way to guarantee that some other process will run in the same thread after the request is complete. (At least that I’ve ever been able to discover.)
@Nathan: So I was correct to add recycle calls to each and every loop in the Wiki example. About half of the examples were missing recycle calls. They were a classic setup for “works in development, fails in production” scenario’s…
Re: product objects in Lists: a couple of 100 are enough to bring the average client to its knees, while you can have hundreds of thousands when using simple homebrew classes. Love the List structure!
Than you for your elaboration on SeSsionScope! I really like this kind of background info, makes it easier to remember.
Looking at the structure of the loop using a tmpDoc it strikes me that I use a variant of this construct in LotusScript quite often:
Set doc = dc.GetFirstDocument()
Do Until doc is Nothing
Set docNext = dc.GetNextDocument(doc)
..
.. do processing, might include romve or save
..
Set doc = docNext
Loop
Reason: I’ve had a couple of cases where the loop would skip docs or otherwise fail because getting the next doc on the basis of a changed or removed doc is not a good idea..
So getting the nextDoc/tmpDoc at the start of the loop in Java or SSJS might be good idea if processing involves changing the document. @Nate: Can you comment?
@8 – Yeah, that’s a pattern I’ve also used on a number of occasions. People tend not to pay attention to it, but there is an equivalent of .recycle for Lotusscript. It’s Delete. So you could explicitly include a Delete doc just before your Set doc = docNext and that would force the memory clean up instead of waiting for Lotusscript’s garbage collection. This works particularly well with custom classes, where you can create a Delete method that kicks off a bunch of other processes.
Unfortunately, Java doesn’t really have an equivalent concept. In the early days, people messed with the finalize() method for a while, but if you read the documentation, this is pretty much deprecated.
@6 Nathan, thanks for a very informative explanation. It’s always useful to understand how it works behind the scenes, which often clarifies best practice approaches.
@8 Lars, yes, I’ve had similar problems when removing documents. If my understanding is correct, the problem for changing documents comes when looping through a view and the field that is changed would affect the position of the document in the view.
@9 The main time I’ve used Delete is, as in Bob’s examples, when passing a document from LotusScript to an agent. So before I continue processing in my LotusScript I need to call Delete doc and assign doc again, in order to get the version updated by my agent.
Just had our fingers burnt here after applying recycle() to most of our loops having read the Xpages cheatsheet. In retrospect I think I would be more selective about where to use it e.g. not on small loops ( read Nathans comments above too ).
If you are using recycle() watch out for some of the side affects as listed in the Designer Help.
Recycling a parent recycles all the children.
If you create more than one object to represent the same Domino element, recycling one recycles all. For example:
View v1 = db.getView(“All”);
View v2 = db.getView(“All”);
v1.recycle(); // also recycles v2
Results are undefined if you attempt to use a recycled object. No error is guaranteed.
I am a little confuse, is this refering to actual Java (which I was aware of) or Server Side Java Script? (which kind of a mismasch of java, javascript, lotusscript, and formula language? The above syntax leads me to believe that you are refering to SSJS, but you say java, hence the confusion.
The problem and best practice recommendation actually refers to both. Developers who are used to writing Java agents are used to recycling already. I believe Java agents need recycling to a greater degree than is required for XPages, because XPages automatically recycles the Session object and all descendants at the end of each request.
Developers whose experience is LotusScript are not used to recycling. The only garbage collection I’ve forced in LotusScript is erasing lists.
In both SSJS and Java for XPages, though, whenever you’re looping through Notes objects you should recycle. Otherwise those handles are not recycled until the request finishes. If you’re looping through large collections that can result in server crashes, which is what I found in this scenario. At the time there were no tech notes covering XPages and recycling, though the documentation is better now.
You’re right that SSJS is a mismatch of Java, JavaScript, LotusScript and @Formula. But SSJS is effectively running Java code under the hood. The objects and functions are just made to look like JavaScript, LotusScript and @Formula to make it easier for developers familiar with those languages.
Why not recycle tmpDoc too? i cant find the answer anywhere. This object is also in the loop so it should create a new instance in each iteration. since it is not recycled why doesn’t it create a memory problem?
The answer to that comes with understanding recycling. You’re not recycling the variable – doc, tmpDoc, etc. You’re recycling the C++ handle to the underlying Domino object. So if you did:
Document tmpDoc = dc.getNextDocument(doc);doc = tmpDoc;
tmpDoc.recycle();
You would be recycling the C++ handle for the NotesDocument referenced by both tmpDoc and doc. So doc.getFirstItem(“myItem”) would then fail on the next loop.
Thankfully in the intervening years since I first hit the problem I’ve understood what happens much more, particularly through working on OpenNTF Domino API!