Fix lower bound of SQLite version in not-fork.d/sqlite3/benchmark/versions and not-fork.d/lmdb/benchmark/versions. It was 3.36.0 but there has been a little drift in vdbe.c . Temporarily bumped it up to 3.43.0 so that eg make
test-sqlworks first time. Needs some testing and simple fixes.We don't produce a good version number for LumoSQL except in the VCS and documentation, plus we inconsistently append to the version hash, eg: 3.53.2++lmdbv1_debug-on has version "3.53.2 2026-06-03 19:12:13 d6e03d8c777cfa2d35e3b60d8ec3e0187f3e9f99d8e2ee9cac695fd6fcdfalt1" While 3.53.2+lmdb-0.9.35 has version "3.53.2 2026-06-03 19:12:13 d6e03d8c777cfa2d35e3b60d8ec3e0187f3e9f99d8e2ee9cac695fd6fcdfalt1lmdb". We should probably #define LUMOSQL_VERSION, and add a --lumosql-version parameter at least. What else? This is important including for: benchmark database (what LumoSQL version gave this result?); not-forking; and very likely for putting in the header of the database in a similar way to SQLite, assuming we can reconcile what to do on different backends including standard SQLite. This will become a problem when we are consolidating benchmark data from many people. We have done some research on SQLite Meta Slots which could potentially hold some detailed version information, and in the test suite we do some probing for capabilities in a LumoSQL binary, and that is potentially how some version reporting could be done. We don't define
LUMOSQL_VERSIONor have--lumosql-versionyet, see also benchmark/README.md. One of the open questions for sqlite.org.Introduce a solution for AVL ephemeral tree spilling, as discussed in examples/fossil/importing-massive-trees-to-fossil-lumosql.md and the comment for bd8a225546 . This would mean LUMO_NOEPH would no longer be a developer-only test tool. We don't need this for ordinary workloads, so it seems.
Stress test the AVL ephemeral tree code. We do currently test it but not to extremes. It's small but complicated and brand-new^H^H^H^H a bit new, and critical.
We expose rowsums via the
lumo_rowsumfunction. This may be sufficient for doing column comparisons, but since we don't have users yet that isn't clear. Rowsums have a cost, but for some workloads they are much faster where column comparisons are done. Perhaps rowsums should be exposed in higher-level SQL, so we can say "store the rowsums for this table" and then later "list only the rows that have changed" which will involve integer comparisons no matter how large and complicated the row contents might be.We vendor the portable BLAKE3 algorithm. The non-portable SIMD version of BLAKE3 is much faster, taking advantage of CPU features on x86_64, ARM and WASM. On the other hand, we have profiled the rowsum slowdown and it is not due to hashing at all, at least on hardware that isn't tiny and embedded.
not-fork upstream.conf version_filter cannot cleanly express "exclude the 1.x line" against the LMDB recipe. The current
< 1.0-alphain upstream.conf is a hack exploiting the convention that pre-release suffixes sort below the bare version:1.0-alphais the lowest known pre-release token, so comparing against it excludes everything else including any 1.x releases. This is a not-fork bug but not urgent for LumoSQL now we have this hack in place.It's time to carefully review the versions in the benchmarking files under not-fork/ . These are the defaults and override what is specified on the make commandline.
SQLITE_VERSIONS=on the command line is honoured, but it is added to the default matrix rather than replacing it.make what SQLITE_VERSIONS=3.53.1builds 3.53.1 and every version in theversionsfiles (3.43.0, 3.45.0, 3.50.0, 3.53.2, 3.54.0, each with its backend pairings). Someone asking for one version gets eleven. This is the same root cause as the item above about reviewing theversionsfiles, andTARGETS=is the documented way to pin an exact set. Worth deciding whether a command-line version should narrow the matrix instead of extending it.when OPTION_LMDBV1_ENCRYPT=on is set, the encryption option is applied to every target in the matrix, including those with no lmdbv1 backend. The target-name logic acts on the option whether lmdbv1 is present or not, so
make what OPTION_LMDBV1_ENCRYPT=oncurrently emits ten targets of which one (3.54.0+lmdbv1-1.0+lmdbv1_encrypt-on) is the one asked for. The rest are spurious: native builds with an empty backend field, eg 3.43.0++lmdbv1_encrypt-on (note the double '+'), and LMDB 0.9 builds carrying an lmdbv1 option that means nothing to them, eg 3.43.0+lmdb-0.9.35+lmdbv1_encrypt-on. These are correctly working binaries we don't want and didn't ask for, with dud names.LMDB-backed Fossil can't clone from a local file:// path (is this still true, post single-file LMDB in Lumo?) This is not a LumoSQL bug. The Fossil patches now live as real patch files in
examples/fossil/patches/, applied byexamples/fossil/tools/patch-source.sh, so a fix for this belongs there rather than in prose.Need to do build/bench on ARM. This may involve a small amount of porting. Every run currently in
benchmarks.sqliteis x86_64 on one machine (an Intel i5-8350U), so while the README says LumoSQL has been known to run on ARM-32 and RISC-V, we have no benchmark data from anything except that one laptop. Cross-architecture comparison is one of the things the benchmark schema was designed for, and we are not yet exercising it.The global config clash between native Fossil and LMDB-backed Fossil is fixed for the
examples/fossilbuild. The patchexamples/fossil/patches/db-configdb-name-lmdb.patchmakesdb_configdb_name()pick.fossil.lmdb/fossil.db.lmdb/_fossil.lmdbunderLUMO_LMDB, covering all three of the dot, XDG and Windows forms, so the two binaries co-exist and neither finds a config it can't read. What remains: anyone building Fossil against LumoSQL by hand, without going throughexamples/fossil/, still hits the original problem, where the new binary inherits a config it can't read and fails withSQLITE_CANTOPENor sometimes just "unknown error". The workaround there is still to clear it withrm -rf ~/.config/fossil.db ~/.config/fossil.db/ ~/.fossil ~/.fossil/before switching binaries, or to isolate the config per test directory. Worth deciding whether the rename belongs upstream in Fossil rather than in a LumoSQL patch.As part of profiling, focus on the LMDB property that readers never block writers, and writers never block readers. Concurrency testing of LumoSQL suggests this property could be exposed more, by considering the design of the LMDB backend shim. We need more concurrency testing anyway. The test-sql target makes a good start.
We vendor the reference implementation of SipHash with a fixed, public MAC key, 0xfeedface. Need to support private MAC keys.
Replace the LMDBv1 backup code which currently has v0.9 stuff with a stub and appropriate error. LMDBv1 does incremental backup and that's what should be used.
not-fork.d/lmdbv1/files/backup.cis still the v0.9 single-shotmdb_env_copy()implementation, to the point that its own header comment names its path asnot-fork.d/lmdb/files/backup.c. It makes the whole copy on the firstsqlite3_backup_step()and reports a one-page job so the shell's progress loop terminates, which is the right answer for 0.9 and the wrong one for v1.0.examples/fossil/tools/prepare-source.sh copies and checks LumoSQL build outputs which are specific to LMDB. We have a BACKEND_SPEC variable intended for a backend-agnostic build but the script is not agnostic yet. It could make sense even at this point for rowsum with native SQLite, and certainly for future backends with other btree replacements than LMDB.
Reimplement and benchmark safe temporary LMDB environment reuse in both backends. Each temporary or
:memory:database currently creates, maps, closes and removes a private environment. The previous pool improved this path but could segfault whilelumoTempEnvBlank()reset an environment and was removed. Reproduce that failure, then design a pool that fully resets each environment and keeps incompatible LMDBv1 configurations separate.LMDB error codes are mapped as SQLite generic strings.
error_mapin bothbtree.cfiles maps an LMDB code to a SQLite one, and the text comes fromsqlite3ErrStr(), so a failed decryption and a refused nested transaction read alike.mdb_strerror()has the specific text andstruct Btreeholdssqlite3 *db. Fis is to modifyerror_mapand present withsqlite3ErrorWithMsg(). It is possible that a wrong passphrase for the encrypted case won't fit this design, so think about that. open-path case. There's still going to beunknown errorreturned, but less often.Switch on -Wall in Lumo compiles. Currently only on in the examples/ . SQLite's amalgamation has warnings with -Wall, so it likely needs -Wno-unused-function and friends like Fossil does. A place to start would be to add -Wall to lumo.mk (all of them).
Decide how the LMDB backends will support
sqlite3BtreeSavepoint(p, SAVEPOINT_ROLLBACK, -1). SQLite defines this as rolling back all contents while keeping the write transaction and its locks open. The LMDB backends instead abort the root transaction, enterSQLITE_TXN_NONEand release the process writer lock. SQLite 3.53.2 and trunk have no caller for this path. Exact support probably requires a clean outer LMDB write transaction whose replaceable child holds the transaction's changes; aborting and reopening the root would briefly release the cross-process writer lock.LUMO_NOEPHgives wrong window-function results in both LMDB backends.sum(a) OVER (ORDER BY a ROWS 1 PRECEDING)returns 1,3,2 on lmdb and 1,3,7 on lmdbv1, where stock SQLite returns 1,3,5. A plainORDER BYwindow fails with "malformed".Test for that class of error: add a
test/sqlfile of ephemeral-table queries, run with and withoutLUMO_NOEPH, and compare the output with stock SQLite. It should cover several cursors on one ephemeral table, DISTINCT, IN, ORDER BY with LIMIT, automatic indexes, RETURNING, UPDATE FROM, zeroblob rows, and backward scans after changes.tool/test-runner.tclfirst needs a way to setLUMO_NOEPHfor a test file.sqlite3BtreeRollbackin both LMDB backends trips every cursor when a savepoint or statement transaction is open, even when SQLite passeswriteOnly, becausecloseAllSavepointsinvalidates cursors withwriteOnly0. Upstream trips only write cursors. PasswriteOnlythroughcloseAllSavepoints.The libdb changes to VACUUM,
sqlite3BtreeRowCountEstand savepoint rollback have not been built or tested. Build a libdb target, runtest/sql, and add a test for each: a commit after VACUUM changesPRAGMA data_versionin a second connection;SAVEPOINT s; DROP TABLE t; ROLLBACK TO s; COMMITkeeps t; andOP_IfSizeBetweenon a populated table before any full stat.In lmdb and lmdbv1,
invalidateCursorscan leave a cursor with neither an LMDB cursor nor a saved key, andrestoreCursornever reopens it, where libdb'srestoreCursordoes. A later move on such a cursor would get EINVAL from LMDB. Write a test to find out whether SQLite reaches this state.Wrap the comment lines over 80 columns in the three
btree.cfiles, including the libdblumoDeleteAllsignature and flags parameter comment, the twocached_limitcomments inpager.c, andlumo-crypto.cline 93.