As recalled by Dan Shearer.

The current version of LumoSQL is 0.8, released June 2026, 6 years after the project started. LumoSQL 0.8 is the first practically usable release, uniting any combination of 9 years of LMDB releases with 7 years of SQLite releases, two of the most-used software packages. With the addition of the not-quite-released LMDB v1.0, LumoSQL brings to SQLite the LMDB features of encryption and incremental backup. These are big changes for such highly stable software but we hope they have been done in a such a non-intrusive way that users of these APIs will barely notice -- but we shall see.

It was quite a journey to get here. Howard Chu's 2013 sqlightning proof-of-concept showed that swapping SQLite's btree.c for an LMDB shim appeared to be as fast or even faster than the native btree. In 2019 Dan Shearer decided there was potential in this because LMDB has features other than just speed, including excellent crash recovery and the propery that readers never block writers, and writers never block readers. Dan pitched it to NLNet, who awarded funding. From early 2020 (terrible timing, because of COVID-19) LumoSQL was under way, with the cool name Esther Payne thought of.

In late 2019 LumoSQL was founded by Dan Shearer, and Keith Maxwell and Dan had reproduced and updated Howard's results, noting that in the preceding 7 years SQLite's own btree had closed much of the performance gap. and established that, further to Howards clever hack, SQLite can in fact carry two storage backends through a btree-API abstraction layer. This involved taking Howard's quick hack and forward-porting it 7 years to current SQLite and tools. It was clear that something more sustainable was needed. In early 2020 Claudio Calvelli and Dan designed the Not-forking reproducibility tool, and Claudio implemented it. Not-forking is quite a unique piece of software, in an area several computer science papers have addressed since, and still one of very few tools in its class.

Claudio also contributed substantially to the LumoSQL design and architecture. The architectural payoff is that the SQL layer is native SQLite throughout and only the storage engine swaps. We chose the ancient, deprecated and now deleted-from-LumoSQL BDB backend to illustrate that the not-forking approach could swap any of LMDB, BDB and the original SQLite Btree under the SQLite database.

Part of Dan's pitch to NLnet was that SQLite needed open source cryptography that isn't just bolted on in a fork. Dan designed the concept of rowsums as an approach that could be extended to a new kind of row-based cryptography. Ruben De Smet introduced the idea of Attribute-based Encryption to implement Dan's conept and contributed to the cryptography and RFC work. Ruben also did the first cluster-based benchmarking runs. Gabriele Balachninaite contributed during the same period and included LumoSQL in her master's thesis at VUB, and Björn Johansson was essential to keeping the project organised and funded, amid the global chaos of COVID. Björn reviewed RFC drafts, and worked on team coordination and tooling during 2022. The authoritative current list is in AUTHORS.

Unfortunately by the end of 2022, the pandemic had really taken its toll on the project in various ways. A great deal of progress had been made, but LumoSQL remained stuck at version 0.4. It was a collection of good ideas working code as far as it went, and bitter-sweet memories.

In 2026, Dan Shearer had the time to reboot LumoSQL to get it to a roughly usable state with all of the functionality it was originally intended to have existing at least to a basic level. Happily, Howard Chu was preparing LMDBv1.0 ready for release which brings page-level encryption, incremental backup and checksums among other things with it, and LumoSQL inherits all of these features with LMDB thus saving an immense amount of work. It would probably have been much harder to reboot LumoSQL in say 2024! By re-focussing only on expert users, and consulting widely with the key upstream authors and users, Dan was able to address just the minimum necessary feature set. Attribute-based encryption is still a dream, but the rowsum mechanism now works and there is some interesting database theory to explore with that.