Dan Shearer
June 2026
As at LumoSQL v0.83
This history is unapologetically technical in nature. The software tree you have in front of you is a feat of engineering by the people named in this file. LumoSQL was very active 2019-2022, and then was dormant until 2026. Today's working, useful LumoSQL is due to the long history in this file, and doubtless more besides. Let us know if we have omitted a contribution.
LumoSQL 2019
Howard Chu's 2013 sqlightning proof-of-concept showed
that swapping SQLite's btree.c for an LMDB shim appeared to be as fast or even faster than the native btree. In 2019 Dan
Shearer decided there was potential in this because LMDB has features other than just speed, including
excellent crash recovery and the property that readers never block writers, and writers never block readers. Dan pitched it to
NLNet, who awarded funding and from December 2019 LumoSQL was under way, with the cool name Esther Payne thought of.
LumoSQL 2020-2022 and Global Pandemic
By early 2020, Keith Maxwell and Dan had reproduced and updated Howard's results, noting that in the preceding 7 years SQLite's own btree had closed much of the performance gap. We established that, further to Howards clever hack, SQLite can in fact be connected to either of two storage backends through a btree-API abstraction layer. This took Howard's quick hack and forward-ported it 7 years to then-current SQLite and tools, and Keith built a Docker workflow. It was clear that something more sustainable was needed. In early 2020 Claudio Calvelli and Dan worked out the requirements for what is now the Not-forking reproducibility tool, and Claudio did the design and first implementation. Not-forking is quite a unique piece of software, in an area several computer science papers have addressed since, and still one of very few tools in its class.
The work so far was presented at FOSDEM in Feb 2020, linux.conf.au in March 2020. Around about this time the world began to hear about a new virus called COVID-19.
The online-only lockdown-era Tcl/SQL conference in Nov 2020 involved a discussion with the SQLite developers about LumoSQL.
We chose the ancient, deprecated and now deleted-from-LumoSQL BDB backend to illustrate that the not-forking approach could swap any of LMDB, BDB and the original SQLite Btree under the SQLite database. LumoSQL would still crash very easily but the concept was exciting and quite new. However the LumoSQL plan called for a lot more than just swapping backends and seeing if Howard was right about speed.
Part of Dan's pitch to NLnet was that SQLite needed open source cryptography that isn't just bolted on by a fork, such as SQLCipher. Dan designed the concept of rowsums as an approach that could be extended to a new kind of row-based cryptography. Starting introducing the concept of an optional checksum for every row (not available in any of main databases), the idea is to extend that checksum to cryptographic control over that row. Ruben De Smet of the Vrije Universiteit Brussel introduced the idea of Attribute-based Encryption to implement Dan's concept and contributed to the cryptography and Lumion RFC work. Ruben also did the first cluster-based benchmarking runs. Gabriele Balachninaite contributed some R analysis and documentation during the same period. Martina Palmucci did some cryptographic analysis around this time, and included LumoSQL in her master's thesis at VUB. Björn Johansson was essential to keeping the project organised and funded, amid the global chaos of COVID. Björn reviewed RFC drafts, and worked on team coordination and tooling during
- The authoritative current list is in AUTHORS.
Unfortunately by the end of 2022, the pandemic had really taken its toll on the project in various ways. A great deal of progress had been made, but LumoSQL remained stuck at version 0.4. It was a collection of good ideas working code as far as it went, and bitter-sweet memories.
The 2026 reboot
In 2026, Dan Shearer rebooted LumoSQL to get it to a roughly usable state with all of the functionality it was originally intended to have existing at least to a basic level. Happily, this was just as Howard Chu was preparing LMDBv1.0 ready for release, which brings page-level encryption, incremental backup and checksums among other things with it, and LumoSQL inherits all of these features with LMDB thus saving an immense amount of work. It would probably have been much harder to reboot LumoSQL in say 2024! By re-focussing only on expert users, and consulting widely with the key upstream authors and users, Dan was able to address just the minimum necessary feature set. Attribute-based encryption is still a dream, but the rowsum mechanism now works and there is some interesting database theory to explore with that.
LumoSQL 0.8 was the first usable release ever, 6 years after the project started. LumoSQL 0.8 could compile any combination of ten years of software from two of the most-used software applications anywhere, SQLite and LMDB. With the addition of LMDB v1.0, LumoSQL brings to SQLite the LMDB features of encryption and incremental backup. These are big changes for such highly stable software but we hope they have been done in a such a non-intrusive way that users of these APIs will barely notice. This release was aimed at Fossil users because they are a very special use case of SQLite, and tend to give it a good workout just doing their daily activities.
LumoSQL 0.82 was the first release taken beyond the Fossil community to SQLite in general.
LumoSQL 0.83 added the rpm example, some significant performance improvements for the LMDB backend, more benchmarking results and doc fixes.
LumoSQL 0.84 added the Node.js example for the better-sqlite3 binding (8 million downloads weekly!), and intensively work on making LumoSQL more out-of-the box for users, especially improving makefiles and build.tcl. The Node.js example exposed the tricky corner case of options that need to be enabled when the SQLite parser is generated rather than when SQLite is compiled. Found a crashbug in LMDB btree caching code.
In 2026 Not-forking has also been taken further and connected with two very active fields of computer science research, but that is not the topic of this history.