Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
50 most recent check-ins that include changes to files matching 'not-fork.d/*'
|
2026-07-21
| ||
| 19:00 | Fix fts comment check-in: 80f3d53d7a user: danshearer tags: trunk | |
| 18:58 | Make LumoSQL build with FTS5 off by default, matching SQLite upstream. check-in: 4a8fc37423 user: danshearer tags: trunk | |
| 09:40 | Add example for the better-sqlite3 bindings in Node.js . check-in: 43f78d5a72 user: danshearer tags: trunk | |
| 08:56 | 1: Add a linked list to track how many writers in a connection, because LMDB uses Unix per-process advisory write locks while SQLite does per-connection. This will, for example, return SQLITE_BUSY for a second BEGIN EXCLUSIVE attempt within an existing BEGIN EXCLUSIVE, within a single process. 2. Switch off temp LMDB environment reuse, because is can lead to a segfault around lumoTempEnvBlank(). This can be a useful performance optimisation which is now switched off globally. Revisit later. Accidentally included these two unrelated changes in this one btree.c diff. Very annoying. check-in: 73ca09c25a user: danshearer tags: trunk | |
| 08:47 | Generate UDL-capable (Upper Default Limit) SQLite parser for downstream update-limit builds. Compiling with -D SQLITE_ENABLE_UPDATE_DELETE_LIMIT only works if the parser has been generated with code to support UPDATE/DELETE ... ORDER BY ... LIMIT grammar. We now generate SQLite with SQLITE_UDL_CAPABLE_PARSER so downstream consumers can enable the feature at compile time without forcing update-limit behavior on any LumoSQL build. Discovered when creating the Node.js better-sqlite3 example. check-in: b419c44cf9 user: danshearer tags: trunk | |
|
2026-07-15
| ||
| 16:35 | Fix LMDB ephemeral cursor crash/corruption in window queries. Showed up in npm's sqlite3-better test cases. check-in: ce8a10f989 user: danshearer tags: trunk | |
|
2026-07-13
| ||
| 16:21 | Make git the default. Too often Fossil fails due to lack of HTTP/1.0, and applying the patch is painful check-in: af460fc8a0 user: danshearer tags: trunk | |
|
2026-07-08
| ||
| 20:30 | Optimise LMDB CursorHasMoved with cached row, yet another cursor saving. Use BtCursor::curValid to answer sqlite3BtreeCursorHasMoved() without probing MDB_GET_CURRENT when the cursor already has a cached current row. Applies identically to both lmdb backends. check-in: c0c15bd380 user: danshearer tags: trunk | |
| 20:02 | Optimise LMDB cached payload reads, identical fix just applied to lmdbv1 reuse the current-row cursor cache in sqlite3BtreePayloadChecked(), and avoid the no-op helper in sqlite3BtreePayloadFetch() for normal btrees. check-in: ef035cc316 user: danshearer tags: trunk | |
| 19:51 | Optimise LMDBv1 cached payload reads in btree.c for lmdbv1 Reuse the cached current row in sqlite3BtreePayloadChecked(), avoiding restoreCursor() and MDB_GET_CURRENT on the incremental-blob read path when the cursor is already positioned, and also, skip the no-op stable-row helper in sqlite3BtreePayloadFetch() for normal Btrees. check-in: 7c1a594ad9 user: danshearer tags: trunk | |
|
2026-07-06
| ||
| 11:20 | lmdb/lmdbv1: use SQLite's specialised record comparator in index_compare Make index_compare() call sqlite3VdbeFindCompare() instead of sqlite3VdbeRecordCompare() (which is way more generalised) for every B-tree descent step. There, the more specific vdbeRecordCompareInt/String reduces the write path and gives ~7% speedup on both LMDB backends. This is me getting practice with callgrind. check-in: 69e75d485e user: danshearer tags: trunk, v0.83 | |
|
2026-07-04
| ||
| 22:37 | Implement a cache on rowsum traversals check-in: 25cc7800c4 user: danshearer tags: trunk | |
| 21:25 | Final part of the multi-open speedup patch check-in: 944439a4b9 user: danshearer tags: trunk | |
| 20:52 | Opening an LMDB environment read/write allocated (and zeroed) a ~2 MiB per-env dirty-page scratch in mdb_env_open, even for connections that only ever read. Helps intensive read workloads that keep re-connecting. Now with a benchmark test. check-in: dbe91c91cf user: danshearer tags: trunk | |
|
2026-06-22
| ||
| 09:47 | Improve docs Summarise the transaction option benchmarking (basically just use default). check-in: 53e828831b user: danshearer tags: trunk | |
|
2026-06-21
| ||
| 06:39 | fix chroot jail when running server as root with LMDB check-in: 689867c5e9 user: danshearer tags: trunk | |
| 06:10 | benchmark operations with temporary tables check-in: 8f4564bd2e user: danshearer tags: trunk | |
|
2026-06-20
| ||
| 23:14 | lmdb,lmdbv1: pool temp environments and keep them unsynced Every temp or :memory: database opened a private LMDB env at a fresh mkstemp path and unlinked it on close, often hundreds opens and therefore fsyncs as compared to ~3 for native sqlite for the same Fossil operation. We fix this by keeping MDB_NOSYNC on temp and :memory: envs, and have a small pool of envs ready to go. Major speed improvement. check-in: f020f61185 user: danshearer tags: trunk | |
| 15:18 | LMDB cursor cache: invalidate only the writing cursor cursorCacheInvalidateSiblings clears the cached current row on the writing cursor alone, due to the way VDBE works in SQLite. This means some cache code was unusued so we have deleted it in both LMDB backends. check-in: 68f6623c3d user: danshearer tags: trunk | |
|
2026-06-19
| ||
| 21:39 | Implement the single-file LMDB in v1 backend, just copying lmdb check-in: 5741abfea7 user: danshearer tags: trunk | |
| 20:41 | Move from the default LMDB files-in-directory to using the LMDB flag for just using a plain file plus a lockfile with the same name, which matches existing SQLite. Thanks to Howard Chu for pointing out the MDB_NOSUBDIR exists. check-in: eda9a500dc user: danshearer tags: trunk | |
|
2026-06-18
| ||
| 10:53 | Update TODO and enforce the temporary lower bound of SQLite versions at 3.43 check-in: a378c46408 user: danshearer tags: trunk | |
| 08:53 | Add siphash message authentication code by Jean-Philippe Aumasson and Daniel J. Bernstein. Use it as fast row-based checksum by hardcoding a public default key. Other uses will come. check-in: 3284ed8b78 user: danshearer tags: trunk | |
| 07:31 | Update versions so that by default, benchmarking does a reasonable number of LMDB backends check-in: af6cf5920f user: danshearer tags: trunk | |
|
2026-06-17
| ||
| 14:00 | another mod invalidated by the new not-fork patch strictily non-fuzzy check-in: ed069c58f4 user: danshearer tags: trunk | |
| 13:13 | Make mod more robust by anchoring to text unaffected by a newline that appeared for a while from SQLite version 3.35 check-in: 17429c57dd user: danshearer tags: trunk | |
| 12:32 | Move anchor point so that the mod doesn't break on an extra linespace that appeared for a few SQLite versions following 3.34. check-in: 0187f3ea34 user: danshearer tags: trunk | |
| 12:31 | Doc tidyups check-in: 9211e21e5b user: danshearer tags: trunk | |
|
2026-06-14
| ||
| 09:02 | Default LumoSQL to sync=normal in all circumstances and ensure the amalgamation sees it. MDB_NOMETASYNC is Lumo's sync=normal, which is one fsync per LMDB commit and maximum loss of one transaction. This is what is now persisted throughout as normal if nothing is specified. check-in: 63c02c8703 user: danshearer tags: trunk | |
| 02:23 | lmdbv1: optional CRC-32C page checksums (OPTION_LMDBV1_CHECKSUM) Add per-page corruption detection to the lmdbv1 backend. The purpose of this is to detect corruption in pages at rest on disk. (Already added docs in previous commits.) When built with lmdbv1_checksum=on, the backend registers a CRC-32C checksum function via mdb_env_set_checksum(). LMDB does the rest. The CRC is Mark Adler's crc32c.c v1.5, vendored verbatim (zlib-style licence) as lumo-crc32c.c . Temp/memdb scratch is not checksummed. Adds test/sql/page-checksum-corruption.test flips a bit on-disk to see if it is detected. 6 new files created, 3 modified. check-in: 225196918f user: danshearer tags: trunk | |
| 01:05 | Add sync build option to lmdbv1 (exactly the same as for lmdb 0.9.x, although the trees are starting to diverge now.) check-in: 8add7f5256 user: danshearer tags: trunk | |
| 00:14 | Implement sync with NORMAL as default (same functionality as SQLite WAL mode default "normal") check-in: 01acfce9f3 user: danshearer tags: trunk | |
| 00:12 | Adding the sync patches bit by bit check-in: 131bff077d user: danshearer tags: trunk | |
|
2026-06-13
| ||
| 14:29 | VACUUM now works with encrypted databases, which means fossil import and fossil rebuild now work. The problem was partly in the LMDB shim (copying btrees from temp to permanent in the wrong way) and partly in LMDB (overwriting metadata at the end of pages under some circumstances.) Thanks to Howard Chu for the LMDB fix https://github.com/openldap/openldap/commit/b4a1d0fbe606fa655ad4569720286e492bb51844 check-in: bf97a4b6ea user: danshearer tags: trunk | |
| 10:44 | Check crypto paths to determine if there are bugs in LMDB or LumoSQL check-in: b9c301ca4a user: danshearer tags: trunk | |
|
2026-06-09
| ||
| 18:35 | Fix bug that prevented most rowsums working properly: check_rowsum() returns tri-state (-1, 0, 1) but the caller tested as a boolean. check-in: 3f5ed208fc user: danshearer tags: trunk | |
|
2026-06-08
| ||
| 11:20 | New files for the big ephemeral commit (see commit message for bd8a225546, doc/context-about-ephemeral-tables.md , and fossil/importing-massive-trees-to-fossil-lumosql.md) check-in: 4a16ba04aa user: danshearer tags: trunk | |
| 11:13 | Temporary raise the lower bound for SQLite versions to 3.43.0. As mentioned in TODO there has been some drift in vdbe.c which stops things like 'make test-sql' work out of the box. check-in: bf9d17aa90 user: danshearer tags: trunk | |
| 10:25 | Fix hang and SQLITE_INTERNAL in ephemeral (BTREE_SINGLE) btrees This is a large change, fixing 3 showstopper bugs found by Fossil but also addressing much bigger architectural problem. The bugs were: - Infinite hang: sqlite3BtreeCursorHasMoved/Restore had no ephemeral case, so an ephemeral cursor (no LMDB handle) was reported "moved" while restore said "unchanged", looping forever. Seen in fossil finfo/timeline/branch/leaves. - SQLITE_INTERNAL on rowid-keyed ephemeral tables only handled index btrees. We now support (ephInsertIntKey/SeekInt/DeleteInt/IntKey). - Range seeks: sqlite3BtreeIndexMoveto only worked for exact positions. Replaced with a positioned ephSeek that supports SeekGE/GT/LE/LT and IdxLE/GT. See doc/context-about-ephemeral-tables.md for SQLite background and how it is relevant to LumoSQL. For this transient use case, LMDB's write amplification into reliable storage doesn't make sense. LMDB 0.9.x also has the 511-byte limit for long keys (eg Fossil check-in comments are used as keys and they can be up to 12k!) and even LMDB v1.0's much larger default key size is insufficient. So we have two architectural improvements as well as 3 bugfixes. We now fully implement ephemeral tables, hopefully, and we definitely implement it in a better way than using LMDB COW. The implementation is in the sqlite3 wrapper and depends on LUMO_HAVE_EPHEMERAL_BACKEND (settable via LUMO_NOEPH=1) and is identical for lmdb and lmdbv1. We do more-or-less what native SQLite already does for transient tables, which is: keep them in RAM, no transactions, no locking, discard at statement end. We grow in malloced memory without bound, so this is a great way to crash-test LumoSQL. SQLite does not grow without bound, spilling out to a temporary file if it exceeds SQLITE_DEFAULT_TEMP_CACHE_SIZE. As the LumoSQL TODO.md says, in due course LumoSQL needs to detect memory pressure and move back to LMDB, and the use case implies enormous transient tables so its best to do with LMDBv1 which has other advantages. But this patch leaves it as an unsolved problem, and in practical terms LMDB-backed Fossil handles the SQLite tree so that's a pretty good start. The in-memory implementation was on the one hand straightforward once I had decided it should be an AVL tree (because its more lookups than inserts/deletes) rather than any kind of btree such as red-black. But on the other hand it was not straightforward, its a long time since I have implemented a tree and this is very hot code. Kimi was massively helpful here in testing prototypes and is actually quite a good C generator if carefully instructed. A thing to remember is that SQLite trees (native, or LMDB, whatever) know nothing about indexes, they just use a comparator function to determine if node X data is greater/less than node Y data, and the AVL trees uses the same comparator function. This AVL implementation does seem to work, but it isn't trustable until it has had a lot more testing. LUMO_NOEPH=1 switches off all of this in-RAM backing and puts BTREE_SINGLE btrees back through the original on-disk LMDB temp environment. In this case the 3 bugs which lead to this whole patch series are still present, so they need to be fixed (also in TODO.md). For now LUMO_NOEPH=1 is for developer testing only, it doesn't make sense for users. Previous commits fixed backend-generate to add an "include-only" feature so ephemeral.c gets a backend shim but no standalone compile unit. check-in: bd8a225546 user: danshearer tags: trunk | |
| 09:04 | Explain the new include-only keyword, and just generally tell people what is going on with this file and why simply editing it is not sufficient to get the build to notice. check-in: 961ecb66ee user: danshearer tags: trunk | |
| 08:37 | not-fork.d/sqlite3/Makefile - delete unused FILE var check-in: 7a1eb120e7 user: danshearer tags: trunk | |
| 08:26 | not-fork/sqlite3: generate rename rules with printf, not echo dash is the default /bin/sh shell on several operating systems including Debian/Ubuntu. POSIX leaves echo's handling of backslashes implementation-defined. Under dash echo collapses "\\b" to "\b", so regenerating backend-rename.mod produced patterns like '\bbtree\.c\b' instead of the correct '\\bbtree\\.c\\b'. This was quite subtle because the btree.c -> lumo_btree.c rename silently didn't happen so the native SQLite btree is compiled instead of the backend, and libsqlite3.so / tclsqlite3 fail to link. Even more confusing, the existing files were not generated on dash and work. And finally, backend-generate is only run rarely, and manually, when a backend replaces an entire SQLite file (or stops doing so.) Switch to using printf because that is guaranteed on all shells. check-in: b0d7bf98c1 user: danshearer tags: trunk | |
|
2026-06-07
| ||
| 08:33 | Implement sqlite3BtreeIsEmpty() stub via mdb_stat instead of always returning "empty". Found by doing anything in Fossil that does a join with 3 or more tables, because it returns zero rows. Examples: fossil timeline, fossil leaves, fossil branch list. This function was introduced in SQLite 3.51. The stub was returning *pRes=1 unconditionally, oops. All we do is copy the behaviour of sqlite3BtreeRowCountEst and it works. check-in: 03db288c2b user: danshearer tags: trunk | |
|
2026-06-04
| ||
| 23:34 | Add missing version floor. This mod was being applied to versions of SQLite that can't use it and didn't have a corresponding bit of code to patch, so it aborted. Fix comments. check-in: e521819b7a user: danshearer tags: trunk | |
| 12:33 | Big comment refactor. Clean docs for relaunch release. check-in: 9fe0ea0614 user: dan tags: trunk | |
|
2026-06-03
| ||
| 13:19 | Clean up comments check-in: 1ccdb681e7 user: dan tags: trunk | |
| 13:17 | LMDB backend fixes uncovered by fuzzing for queries returning wrong rowcounts and in one case a spurious SQLITE_CORRUPT. The fuzz harnesses are full of SQL that no sane person would use. It might make sense for LumoSQL to have its own fuzzer in test/ . There's a structural choice here with ephemeral btrees now storing the whole record as the LMDB key. Hope it doesn't have a performance impact, but we'll see. lumoCursorSpills needed some fixing which isn't surprising. LEFT JOIN counts were off and that was really tricky cursor stuff. Thanks to anonymous for doing the hard work on that one. check-in: 92c8aa280c user: dan tags: trunk | |
|
2026-06-02
| ||
| 17:34 | One line fix to btree.c to fix ORDER BY ... DESC with a constraint. Both backends. Add SQL test repro_order-by.sql , and added tests to order.test. Ported the txt read->write upgrade fix from 0.9 to v1.0. We're going to have to be very careful to keep these in synch. check-in: 2dbf12f452 user: dan tags: trunk | |
| 14:18 | Improved documentation a lot: * test all the amalgamation steps * dropped old distros and ancient docker instructions * moved benchmarking stuff out of loft/ into the top level, because its status is clear and good * lots of updates to how-to-install, remove bitrot * update design-corruption-detection-and-magic.md for the fact that most of this is now implemented * improve benchmarking docs Also added: * make doctor to test dependencies * a one-line bug in btree.c that gave 250 'ERR1' rows in benchmarks.sqlite check-in: 08d3eafb8c user: dan tags: trunk | |
|
2026-05-30
| ||
| 12:43 | Total of the last month of piecemeal changes... this is getting close to LumoSQL v0.8 with LMDBv1.0 now committed with one example of encrytion, Fossil known to work, and much more. The docs now focus on expert users because there is enough now to potentially interest embedded and related developers, and we can take it from there if this actually happens. check-in: fed7b90293 user: dan tags: trunk | |