<!-- Copyright 2020 The LumoSQL Authors, see LICENCES/MIT --> <!-- SPDX-License-Identifier: MIT --> <!-- SPDX-FileCopyrightText: 2020 The LumoSQL Authors --> <!-- SPDX-ArtifactOfProjectName: LumoSQL --> <!-- SPDX-FileType: Documentation --> <!-- SPDX-FileComment: Original by Dan Shearer, December 2019 --> <!-- toc -->

Welcome to LumoSQL v0.85

LumoSQL is a modification of the SQLite embedded data storage library, made using the not-forking software reproducibility tool. SQLite is among the most-deployed software which is why LumoSQL goes to great lengths not to fork it. LumoSQL adds features to SQLite by combining multiple source trees as they change over time, notably the LMDB library, besides other bespoke features if users select them. Not all features involve an incompatible file format like LMDB does. This preview release is for curious technical developers wondering why anyone would want to touch trillion-scale software?. The big-picture reasons are in the LumoSQL motivations document, while some pragmatic reasons include:

• You would like to run an SQLite application faster than it currently does. Speed and benchmarking are tricky subjects and we measure it extensively. LumoSQL may well run faster for your use case
• You would like to explore different approaches to robustness and crash recovery in SQLite applications
• You would like an easy, mainstream way to encrypt your SQLite database
• You would like additional features in SQLite stored compatibly with the sqlite3 file format, readable by every standard SQLite install

Production use is not yet recommended although we are reasonably confident LumoSQL will not eat your data. We welcome all reports of success and failure.

LumoSQL exists today because of the efforts of many people since late 2019, as the minimum glue between famously well-tested code found on devices and computers everywhere: SQLite and LMDB. LumoSQL is an unusual feat of engineering, but if you're interested in the computer science theory behind it (especially the idea of intention when making code changes) please do get in touch because we have very active work in that area.

LumoSQL was supported by the NLNet Foundation, and we thank them very much.

Things you can do with LumoSQL:

• Compile against applications that use SQLite. We maintain examples in the tree for Fossil and libfossil, rpm, and Node.js. We have also done some testing with Python webapps and a few other things. Do let us know how you get on.
• Replace the SQLite Btree storage engine with either of two LMDB storage engines, transparently to both SQLite and applications. Like SQLite, LMDB is a vastly deployed embedded library, and LumoSQL ties these two codebases together with minimum fuss. You can mix-and-match versions of each across the last decade or so. LMDB works on a different principle to SQLite btree, and one or the other may be better for particular use cases.
• The user functionality supports SQLite usage fully, except where a backend does not have an SQLite concept due to architectural differences. All SQL works unmodified, and nearly all pragmas.
• The third backend LumoSQL ships is the standard SQLite btree backend, but with the ability to add features to an ordinary sqlite3 database (the one specified as a US Library of Congress Standard, and we are compatible with that.) The first feature we added was storing a checksum for every row, and providing SQL functions to read and calculate the checksum. This is useful for detecting changes in rows without doing comparisons of BLOBs, and also for detecting corruption or unexpected modification on a per-row basis. Any SQLite binary that can read an sqlite3 database can read and write to a rowsum enabled database without noticing, or at least our testing shows that so far.
• Features that come with LMDBv1.0 include encryption and page-based checksums. Both of these can be compiled in to LumoSQL. Encryption is via libsodium, and we provide one cipher with basic key management via an environment variable or commandline. There are other ways of achieving both encryption and checksums in LumoSQL, as we document. But these come for free, often even in the sense of performance.
• Benchmark SQLite with results in an SQLite database. SQLite famously ships with 51,445 distinct TCL test cases, parameterised heavily enough that a single make quicktest runs close to a million individual tests and a full run performs millions. But these do not as far as we know compare different compilation options with each other on the same hardware, or across architectures, or across different users. LumoSQL benchmarking takes note of the hardware and software environment, and provides a basic statistical analysis tool. Benchmark databases can be merged, so people can send in their results. The benchmarking could easily be extended to other SQL databases but we haven't tried to do that.
• Run an SQL test suite that tries to break the storage backend across multiple binary configurations, including cross-binary interoperability tests. Like benchmarking, the results are queryable from an SQLite database via tool/test-sql-filter.tcl. The system supports role-based testing, corruption detection by corrupting a test database, and encryption round-trips. The Fossil test suite is separate and drives the LMDB-backed Fossil through correctness, speed, and stress checks. Similarly we run smoke.js as part of building the Node.js SQLite library.
• Mix and match LumoSQL features across a decade of SQLite and LMDB versions. If you build everything you have a combinatorial explosion: of the order of a thousand binaries and a good few hours even on a fast machine. Or you can pair an older LMDB with a newer SQLite if those versions matter for your embedded application, and choose to enable rowsums, encryption, checksummed pages or more.

LumoSQL source is at lumosql.org (Fossil) and Codeberg (git).

LumoSQL stays as close as possible to the SQLite way of software development, recognising that there is almost no room for change without becoming a fork. Therefore build tooling uses Tcl, we produce an SQLite-compatible amalgamation, and we use the MIT licence which is very close in spirit to the SQLite licensing situation, and our integration tool is literally called not-forking. It is also why Fossil was an early target to support, because Fossil and SQLite are symbiotic projects. If LumoSQL can work with Fossil, there's a good chance it can work with millions of other applications, as the other examples demonstrate.

Technical detail:

• the default SQLite Btree storage backend, optionally with rowsums, and the infrastructure to add other features over time.
• LMDB 0.9.x, very stable and probably the most widely-used key-value store (after SQLite's native btree key-value store). This is now in maintenance mode by the LMDB project.
• LMDBv1.0, which is a superset of the stable LMDB 0.9.x API with very different internals. LMDBv1.0 comes with optional page-based encryption, page-based checksums, and incremental backup. LMDB-backed SQLite inherits all these. LMDBv1.0 was released in June 2026.
• LumoSQL has been known to run on x86, ARM-32 and RISC-V architectures, and many Linux and BSD OSs. Our benchmark results are stored in an sqlite file with a tool to merge and compare, so do please try this at home and share.
• Other potential backends are discussed in our knowledgebase of relevant technologies document (there aren't very many.)
• LumoSQL reproducibly integrates 9 years of software infrastructure. The SQLite-side glue is about 2,000 lines under not-fork.d/sqlite3/. The LMDB and LMDBv1 backends are larger because each is a substantive re-implementation of SQLite's btree against an LMDB environment; the two backends share a great deal of code.

Dependencies

LumoSQL needs all of the following to build the default matrix of binaries, consisting of SQLite native + LMDB 0.9.x + LMDBv1.0 without encryption:

• a C compiler and GNU make
• Tcl 8.6 with Tclx
• Perl core plus the Text::Glob module
• git. This is the default source fetcher and the only one needed to build LumoSQL: every not-fork.d/*/upstream.conf sets vcs = git, and the examples fetch their own upstreams from git mirrors too. Fossil is optional, allowing fetching from the canonical repositories of LumoSQL, SQLite and not-forking.
• the not-forking tool 0.7 or later. No distribution packages it, so see below for how to build it.
• SQLite's own build-deps on Linux: zlib, readline, ncurses
• pkg-config or pkgconf, used to locate those libraries.

If you ever plan to enable encryption:

• libsodium — required for OPTION_LMDBV1_ENCRYPT=on (encrypted LMDBv1)

For the example applications in examples/:

• patch, which examples/fossil uses on Fossil's own source.

So, assuming you have the above dependencies:

On Debian/Ubuntu:

sudo apt install build-essential git patch pkgconf tcl tclx \
    libtext-glob-perl libsodium-dev \
    zlib1g-dev libreadline-dev libncurses-dev

Optional, only to fetch from the canonical repositories instead of the git mirrors. Note that the packaged Fossil is often too old to clone sqlite.org, which rejects it with 426 Upgrade Required; build Fossil from source if you hit that. make doctor reports this as a warning, not an error, because git covers it.

sudo apt install fossil

On Fedora/RHEL:

sudo dnf install make gcc git patch pkgconf tcl8 tclx \
    perl-Text-Glob perl-ExtUtils-MakeMaker libsodium-devel \
    zlib-devel readline-devel ncurses-devel

Optional, as above: sudo dnf install fossil

On FreeBSD:

sudo pkg install git patch pkgconf tcl86 libsodium readline

FreeBSD does not package Tclx, so build it from source:

git clone https://github.com/flightaware/tclx
cd tclx
./configure --prefix=/usr/local --with-tcl="$(dirname $(find / -name tclConfig.sh 2>/dev/null | head -1))"
make && sudo make install

NetBSD packages it, as pkgin install tclx.

No distribution packages not-forking, so build it from source. It needs Perl plus Text::Glob, installed by the lines above:

git clone https://codeberg.org/not-forking/not-forking   # or: fossil clone https://lumosql.org/src/not-forking
cd not-forking
perl Makefile.PL && make && sudo make install

Then get LumoSQL itself and check the dependencies:

git clone https://codeberg.org/lumosql/lumosql    # or, with Fossil: fossil clone https://lumosql.org/src/lumosql
cd lumosql
make doctor

make doctor checks every required dependency and tells you how to fix each one it can't find. Iterate until it is happy, then see Build and benchmark below.

Containers and CI without a /etc/passwd entry for the effective UID need export USER=XXSOMESTRING before make otherwise fossil complains.

NixOS has a unique link-based filesystem layout, so for example tools that expect /usr will fail. doc/lumo-build-nixos.md explains what we do about this.

If you are using a large shared machine (eg on a cluster) you may have outdated packages and no permissions to install software. Install locally, for example for Tcl or no Tclx, build them into $HOME/.local and prepend that prefix to PATH, LD_LIBRARY_PATH, LIBRARY_PATH, C_INCLUDE_PATH, PKG_CONFIG_PATH, and set TCLLIBPATH=$HOME/.local/lib. The benchmark script benchmark/lumosql-matrix-on-big-machine.sh shows one such environment.

About LMDB and mmap

LMDB and SQLite's native btree differ at the level of architecture. SQLite is a traditional userspace database program, with a pager, a page cache, a WAL (Write Ahead Log) to make transactions reliable and so on. LMDB takes an entirely different approach, by using the operating system's mmap system calls to do all memory management work for the database including managing buffers, COW (Copy on Write) and reliable persistence to disk. The part of the operating system that hands out memory, its virtual memory system, is tuned hard for the particular hardware, so LMDB is outsourcing one of the most difficult and performance-sensitive parts of a database. The benchmarking system does not yet measure CPU and RAM usage, but we can reasonably expect they are considerably lower under LMDB because the operating system VM has a global view whereas an ordinary userspace database application only knows about its own processes.

SQLite can also use mmap for databases, in three different contexts, but never as a page management mechanism for the BTree. There are many applications and some databases that use mmap, but only LMDB has tried to become the thinnest layer possible over the operating system's VM. This does come with some limitations, including that there is some hardware so limited that it doesn't really have an operating system at all, and certainly not a memory manager. LMDB cannot run on such systems.

mmap is unlikely ever to be supported by WASM because the memory models and system access requirements are incompatible. WASM is great at handling the memory requirements of ordinary userspace programs (see the handy SQLite Fiddle as an example). However anything that uses mmap including LMDB and the shm part of SQLite's WAL mode cannot work in WASM.

Because each LMDB environment reserves its address space up front, a process that opens several LumoSQL databases at once can exhaust the virtual address space even though it is nowhere near running out of RAM. The symptom is SQLITE_NOMEM and the message out of memory on opening the fourth or later database, or on concurrent sqlite3_backup operations, each of which opens a database of its own. This is a real ENOMEM from mmap, not a spurious one: the default reservation is 16TiB per environment, so roughly eight fit in the 128TiB user address space of x86-64.

Set LUMO_LMDB_MAPSIZE to the per-environment reservation in MiB to raise the number of databases that can be open at once. The value is a ceiling on how large each database may grow, so size it for your data, not for the default:

LUMO_LMDB_MAPSIZE=64 myapplication

The same variable is useful in containers and under cgroup memory limits, where the default reservation is harmless on bare metal but is accounted against a smaller limit. Values below 4MiB are ignored.

What doesn't work

There are plenty of details to fix. For example: the key handling for the encryption is rudimentary, and other items are listed in the LumoSQL TODO. LumoSQL builds and passes its tests on Debian, Ubuntu, Fedora, Arch, Alpine and NixOS, and on FreeBSD and NetBSD. We haven't tested on Windows although Windows is a first-class platform for both SQLite and LMDB so we do not expect much difficulty. We do not know of any crashbugs in LumoSQL, but as prerelease software there are doubtless some lurking.

Build and benchmark

make what lists the resolved option set. make targets resolves latest against upstream and lists the targets the matrix would build. make build builds them. make benchmark runs the speedtest1-derived tests against each built binary and appends to benchmarks.sqlite. make test-sql runs the LumoSQL-specific tests in test/sql/ and appends to test-sql.sqlite. See doc/lumo-build-benchmark.md for the detail.

A single-binary benchmark on a small machine takes a few minutes; the full matrix on a big machine takes hours, because by default the build system makes a number of binaries with different combinations. If you are just starting with LumoSQL, you probably want just two: unmodified SQLite of a given version (we will go with 3.53.2 for this example) and the same version backed by LMDBv1.0, with no encryption or other options to start with. Name them with TARGETS:

make build TARGETS="3.53.2 3.53.2+lmdbv1-1.0"

TARGETS accepts the target names make targets prints, and pins what gets built without modifying anything in the tree. Every make target that builds understands it, so you can test or benchmark the same pair:

make test-sql TARGETS="3.53.2+lmdbv1-1.0"
make benchmark TARGETS="3.53.2 3.53.2+lmdbv1-1.0"

The default matrix, which is what you get when you leave TARGETS off, is declared in the not-fork.d/*/benchmark/versions files, one per upstream source. Editing those changes the default for every later command, and leaves your checkout modified. They also take precedence over the version variables below. Use TARGETS for everyday work, and change the versions files only when you mean to move the project default.

Build options are passed as OPTION_X=value or as a member of the target tuple. Examples switching on encryption and row-level checksums:

make build SQLITE_VERSIONS=3.53.2 LMDB_VERSIONS=0.9.35 \
    SQLITE_FOR_LMDB=3.53.2
make build SQLITE_VERSIONS=3.53.2 USE_LMDB=no USE_LMDBV1=yes \
    LMDBV1_VERSIONS=1.0 SQLITE_FOR_LMDBV1=3.53.2 \
    ROWSUM=on OPTION_LMDBV1_ENCRYPT=on

Build encryption with OPTION_LMDBV1_ENCRYPT=on, then supply the passphrase via the URI parameter at open time:

sqlite3 'file:/path/to/db?lumo_key=passphrase'

The first open generates a <db>-salt sibling file; subsequent opens reuse it. Back up <db>-salt with the database and keep the two together. The salt is an input to the key derivation, so an encrypted database cannot be opened without it, even with the correct passphrase. Losing the salt loses the data. Opening a database whose salt is missing is refused giving you the chance to restore the salt from a backup. URI-key is the way SQLite-based products pass keys today (SQLCipher, SEE, and similar follow this convention) but isn't really ideal.

An application can hide the URI handling from its users as shown in examples/fossil. Running make LUMO_ENCRYPT=on in that directory builds a Fossil that reads the passphrase from LUMO_KEY in the environment and rewrites its own filenames, so fossil commands work unchanged on an encrypted repository. See also examples/fossil/README.md. The is more key-management work discussed in TODO-CRYPTO.md.

If you just want to use the SQLite-compatible LumoSQL library to link against your own application, stop here and read LUMOSQL-AMALGAMATION.md, which covers generating the sqlite3.c/sqlite3.h pair with KEEP_SOURCES=1, the helper sources that must travel with it, and the defines each variant needs. The three worked examples in examples/ each link a real application this way: Fossil and libfossil, rpm and Node.js better-sqlite3.

If you want to do benchmarking, replace 'build' with 'benchmark' in the make commands above to run benchmarking on the binaries you just built (if they aren't built, the benchmarking target will build them first anyway.)

Benchmarking results are in SQLite databases queryable via tool/benchmark-filter.tcl and tool/test-sql-filter.tcl. Here are some commands to try on the file benchmarks.sqlite at the top of the tree:

# Compare LMDB-backed SQLite against the same SQLite with the native btree:
tclsh tool/benchmark-filter.tcl -db benchmarks.sqlite \
    -compare -A -backend lmdb -B -no-backend

and

# List the 20 most recent benchmark runs with target, date and total duration:
tclsh tool/benchmark-filter.tcl -db benchmarks.sqlite \
    -list -fields TARGET,DATE,DURATION,BACKEND_NAME

There's a lot more benchmarking to play with, including the hardware that tests were run on, basic statistics, and exporting as TSV to statistical and graphing programs.

Other details

Target names encode the option dimensions: 3.53.2+lmdbv1-1.0+lmdbv1_encrypt-on+rowsum-on is one binary at build/3.53.2+lmdbv1-1.0+lmdbv1_encrypt-on+rowsum-on/sqlite3. Built binaries are shell wrappers that set LD_LIBRARY_PATH to find the backend .so they were linked against.

The not-fork cache lives under $CACHE_DIR (default ~/.cache/LumoSQL/not-fork) and is content-addressed, so re-running make build after editing a .mod fragment only rebuilds affected targets.

3.43.0 is the effective lower bound for LMDBv1 or rowsum builds, and 3.48.0 is where SQLite switched from Autotools to Autosetup. sqlite-version-map.md maps each version range to the .mod recipes that apply and the upstream anchor each one depends on. Read it first when a build fails on an unusual SQLite version.

Where to go next

• LUMOSQL-AMALGAMATION.md: embedding LumoSQL in your own application
• doc/lumo-build-benchmark.md: the long-form build and benchmark reference
• benchmark/README.md: what we measure and why
• test/README.md: the SQL test suite and its cross-binary directives
• sqlite-version-map.md: which patches apply to which SQLite versions
• TODO.md and TODO-CRYPTO.md: what is known to be missing
• HISTORY.md: how LumoSQL got here, and who built it
• doc/README.md: index of everything in doc/, grouping about-* on purpose and design, context-* on background research, and design-* on proposals

Getting involved

Bug reports, build failures on unusual platforms and benchmark result databases are all welcome, and you do not need to be a C programmer to help. CONTRIBUTING.md explains how we work and what we need; everyone taking part is asked to follow the code of conduct. Development happens in Fossil at lumosql.org, mirrored to Codeberg.