Importing massive trees into Fossil on LumoSQL

FOSSIL backed by LumoSQL with LMDB handles substantial repositories well, because the operations required for import are where LumoSQL is the strongest. For example, INSERT-from-SELECT benchmarks at 10–14× faster than native SQLite as of June 2026. While the SQLite repo is the canonical test target, with tens of thousands of check-ins, check-in comments up to ~12 kB (which are used as a key), files with thousands of revisions, larger repos seem to work. The GNU Compiler Collection is much larger tha SQLite, withs ~7 million lines of code over 30 years from a bit under a thousand contributors, and it converts to fossil from git relatively smoothly (note! anything larger than the SQLite repo is not regularly tested.) All of this document assumes LMDBv1.0.

So first of all, give it a go. You might be surprised. If you experience a hang or a crash on a giant Git repo, LumoSQL has known pressure points and there is a lot of room for optimisation. This documents the practical steps available now and the directions worth taking for extreme SCM operations.

From time to time people in the Fossil community like to test against trees vastly bigger than Fossil was ever designed for and which fail every time, such as the entire NetBSD sources. There are good reasons why this is too big, but also it's a good stress test. Do please try and tell us the results if you are this kind of person, we'd love to know what breaks.

These are just a few thoughts and lots of testing and input is needed.

Temporary storage (ephemeral tables)

Many Fossil operations compile to SQL that needs per-statement temporary storage, such as DISTINCT, ORDER BY, IN (...). SQLite opens these as BTREE_SINGLE ("ephemeral") b-trees. See about ephemeral tables in the LumoSQL docs for relevant background.

LumoSQL backs those ephemeral b-trees with an in-RAM AVL tree (see not-fork.d/{lmdb,lmdbv1}/files/ephemeral.c) rather than an LMDB temporary environment. This is close to what native SQLite itself does for transient tables. An LMDB temp environmenti, despite the name "temp", is implemented usingl reliable persistent mmap storage which is overhead that temporary tables don't need. But for huge tree imports, LMDB COW reliability slows everything down enormously.

At present LumoSQL's in-memory AVL treatment of ephemeral tables grows without bound in malloced RAM. For reasonable trees that works, but it won't for giant trees (please do test it! LumoSQL is pre-release software and we need people to be doing these things.) SQLite keeps a transient table in its page cache and spills it to a temporary file once it grows past SQLITE_DEFAULT_TEMP_CACHE_SIZE. This appears to work in all the testing done to date, but equally, when dealing with codebases as widely tested and deployed as SQLite and LMDB, an AVL tree written over the last month barely registers.

The backend logs the LMDB maximum key size at open (mdb_env_get_maxkeysize, via LUMO_LOG). Capture it and compare against the largest key your import actually produces. That tells you whether the key limit is in play at all for your particular data (for NetBSD it almost certainly will.)

The LUMO_NOEPH=1 hack

LUMO_NOEPH is intended for developers only.

Setting the environment variable LUMO_NOEPH=1 makes BTREE_SINGLE b-trees bypass the in-RAM AVL and use the original on-disk LMDB temporary-environment path instead. Because that path is disk-backed you won't run out of RAM. It might be way tooo slow, but give it a go. With LUMO_NOEPH=1, there is no proper cursor-moved/rowid/range-seek handling as fixed in AVL code, and LMDB is the wrong place to do this. But perhaps a giant import would work better if it was implemented, so if you have a useful test case then talk to us, may it is worth duplicating this logic in a way that is inefficient for general use.

Raising LMDB v1.0's max key size

If your ingest fails with SQLITE_TOOBIG from a large key the limit is a property of the LMDB build. LumoSQL already checks the limit dynamically with mdb_env_get_maxkeysize(), 511 is indeed a non-1.0 LMDB limit, but that's detected at runtime.

LMDB v1.0 derives the maximum key from the page size: roughly half a page minus node overhead (LMDB requires at least two keys per page). With the default 4 kB page that is on the order of ~1.9 kB. Try something like a page on the order of 128 kB or larger which is more than LMDB (or most software) can handle. Bigger pages mean more bytes touched and zeroed per copy-on-write page write, which is what ephemerals were designed to avoid. A large page seems reasonable for a dedicated bulk-ingest build. Try ingesting with a build with a very large pagesize, and then consider if you still need the large pagesize to interact with the new fossil repo. A large pagesize can cause you to run low on memory, but this is something you need to experiment with.

Proper architectural improvements for giant imports

Spill the ephemeral backing to disk

We need to copy SQLite and spill to disk, first spending the time to learn exactly what method SQLite uses. But how would we do this with LMDB anyway?

• Building a bespoke pager underneath the AVL would reinvent machinery that LMDB (and SQLite's own temp b-tree) already provide. An AVL is a poor structure to page to disk, where you want packed, page-aligned nodes. So nope, this is not a good option.
• Spilling back to an lmdbv1 temporary environment could work. This is essentially making LUMO_NOEPH automatic rather than the current all-or-nothing, and it requires the 3 fixes mentioned above. But first we need a strogly motivating case, and someone outside the LumoSQL project who will put in the effort. If, say, NetBSD-in-Fossil is your passion, let us know.

Larger keys as build target

If long keys rather than RAM are the dominant problem for particular trees, we could add a build target called "large-page lmdbv1" which would make the trade-off reproducible.