The DQL Approach

I’m loving the approach with DQL. Kudos to the team, there are some very good fundamental approaches. Will try to blog later to clarify

— Paul Withers (@PaulSWithers) 22 November 2018

Earlier today I posted this tweet. It followed another update from John Curtis teasing more enhancements coming to DQL. If you’ve been following closely presentations and announcements about DQL, there are a few things that have been said and seem clear to me, things that give insight into the approach with DQL that are interesting and demonstrate the team is thinking beyond history, beyond Domino, and building for the future. Let me elaborate.

Full Text Search

Full text searching is powerful within Domino. But it also has limitations and complications. This has been raised in sessions about DQL. Notes end users are not comfortable with building full text queries, which is why in the past I’ve deployed saved searches for users. Developers aren’t fully au fait with full text search syntax. I have the full text search syntax help page bookmarked and on more than one occasion I’ve had to fix a search in LotusScript because it was not coded correctly. So as a query language, it was never going to be the right approach for Node.js developers. Plus there are limitations around number of entries etc and performance enhancements were almost certainly required for heavy-duty interaction. A quick approach would have been a wrapper on top; a good approach was to build from the ground up.

Some work was done around that in Domino V10 – full text searches now update the index for up to 20 documents, if there are some that are unindexed when the search is performed. And although many applications will use full text search queries, and that may be a reasonable option for many years, the close relationship between DQL and Node.js mean that, once mature, DQL will likely be the preferable approach.

Languages

DQL was coded at the lowest language level, which means APIs can be added not only for Node.js, but also LotusScript and Java (and SSJS?). The other languages will come in 10.0.1. This key factor, along with the integration with Proton, are why it should become the preferable approach. But involving all those languages means better and more feedback, greater use and more reason to enhance.

The Design Catalog, DomQuery Tool

Building from the ground up allows for some improvements in fundamentals. The full text indexis stored in unreadable files and relies on the UNK table for datatypes, which can vary between environments and even replicas. I’m not sure if DQL also does, it’s possible that it does. But that can make it hard to work out why search syntax is not working as designed when called programmatically. I’ve answered a few questions on XPages full text searching by recommending that the developer opens the Notes Client and tests the search there. I’ve literally just found a technote with a notes.ini variable DEBUG_FTV_SEARCH=1 as well as others to give some debugging about full text searches. But I’ve never seen that used or suggested in a forum answer, and I suspect I’m not the only one who had never come across it.

With DQL we’ve got the DomQuery tool which gives a lot of information about how a search is built, where it’s looking, where the optimisations are. And the Design Catalog aggregates information about which views are and aren’t used. The logic behind all of this is documented, but I envisage a lot of databases not being optimised for DQL. But exposing the information allows some forensics to be developed by anyone who wishes, to help developers prepare a database better for DQL queries. It will be interesting to see if something is built into Domino itself to warn of sub-optimal query criteria that may be better optimised by creating a view, something along the lines of “no view available to optimise query criteria ‘XXXXXXX’, consider creating a view including ‘YYYYY'”.

Not Just “New FT Searches”

But the most significant aspect is that this is not just about making Domino easier to search for those who haven’t got a PhD from the School of Full Text Dark Arts”.

Already there’s been focus in conferences that it improves on MongoDB’s manual boolean tree construction. I can’t admit to being full conversant with quite what that means. But it means I don’t have to find out and code in a specific way. It’s improving the searching over other NoSQL search engines.

And the recent tweet talks about named and positional substitution variables to avoid even needing to understand about SQL injection and code around it. Again, improvements over known pain points from another platform.

Coming to DQL soon – named and positional substitution variables. To defeat the SQL Injection exposure before it even surfaces! API support available in all languages!#dominov10 #dominoforever #DQL

— John Curtis (@john_d_curtis_) 21 November 2018

It’s this approach which means DQL is being built fundamentally for the future, and not just a future for existing Domino developers or Node.js developers, but a bigger future.

Yes, there is other key functionality DQL will need – rich text and attachment handling, ORDER_BY equivalents, multi-database searching etc. That’s going to take some time and I would urge developers to understand that. And there are other areas of Domino that need work before it’s worthy of competing with other databases – flexibility in Docker deployments, external Java access, admin management, etc for example. But to build a mansion, you need to get the foundations right, or the pretty house comes tumbling down with the first heavy rains or earthquake. And it needs to appeal to a wide market, or you won’t get the investment needed to do it right. What I’m seeing with DQL shows some good foundational approaches and long-term planning.