Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
50 most recent check-ins
|
2026-06-08
| ||
| 18:09 | Add testing section Leaf check-in: da1bf3ae7e user: danshearer tags: trunk | |
| 17:49 | Explain that this is all temporary, pending expert review. check-in: 699e029d0d user: danshearer tags: trunk | |
| 17:47 | Add instructions for encrypted Lumo+LMDB backend to Fossil. check-in: 5726770902 user: danshearer tags: trunk | |
| 14:06 | Make sure the two fossil patches are in the recipe, one to let LMDB databases be opened (because they are directories not files), and another implementing HTTP 1.1 so Fossil isn't blocked by application firewalls. check-in: 4281356acd user: danshearer tags: trunk | |
| 11:20 | New files for the big ephemeral commit (see commit message for bd8a225546, doc/context-about-ephemeral-tables.md , and fossil/importing-massive-trees-to-fossil-lumosql.md) check-in: 4a16ba04aa user: danshearer tags: trunk | |
| 11:13 | Temporary raise the lower bound for SQLite versions to 3.43.0. As mentioned in TODO there has been some drift in vdbe.c which stops things like 'make test-sql' work out of the box. check-in: bf9d17aa90 user: danshearer tags: trunk | |
| 10:53 | For the people who keep wanting to put NetBSD or Linux into Fossil... check-in: 1bca31e090 user: danshearer tags: trunk | |
| 10:25 | Fix hang and SQLITE_INTERNAL in ephemeral (BTREE_SINGLE) btrees This is a large change, fixing 3 showstopper bugs found by Fossil but also addressing much bigger architectural problem. The bugs were: - Infinite hang: sqlite3BtreeCursorHasMoved/Restore had no ephemeral case, so an ephemeral cursor (no LMDB handle) was reported "moved" while restore said "unchanged", looping forever. Seen in fossil finfo/timeline/branch/leaves. - SQLITE_INTERNAL on rowid-keyed ephemeral tables only handled index btrees. We now support (ephInsertIntKey/SeekInt/DeleteInt/IntKey). - Range seeks: sqlite3BtreeIndexMoveto only worked for exact positions. Replaced with a positioned ephSeek that supports SeekGE/GT/LE/LT and IdxLE/GT. See doc/context-about-ephemeral-tables.md for SQLite background and how it is relevant to LumoSQL. For this transient use case, LMDB's write amplification into reliable storage doesn't make sense. LMDB 0.9.x also has the 511-byte limit for long keys (eg Fossil check-in comments are used as keys and they can be up to 12k!) and even LMDB v1.0's much larger default key size is insufficient. So we have two architectural improvements as well as 3 bugfixes. We now fully implement ephemeral tables, hopefully, and we definitely implement it in a better way than using LMDB COW. The implementation is in the sqlite3 wrapper and depends on LUMO_HAVE_EPHEMERAL_BACKEND (settable via LUMO_NOEPH=1) and is identical for lmdb and lmdbv1. We do more-or-less what native SQLite already does for transient tables, which is: keep them in RAM, no transactions, no locking, discard at statement end. We grow in malloced memory without bound, so this is a great way to crash-test LumoSQL. SQLite does not grow without bound, spilling out to a temporary file if it exceeds SQLITE_DEFAULT_TEMP_CACHE_SIZE. As the LumoSQL TODO.md says, in due course LumoSQL needs to detect memory pressure and move back to LMDB, and the use case implies enormous transient tables so its best to do with LMDBv1 which has other advantages. But this patch leaves it as an unsolved problem, and in practical terms LMDB-backed Fossil handles the SQLite tree so that's a pretty good start. The in-memory implementation was on the one hand straightforward once I had decided it should be an AVL tree (because its more lookups than inserts/deletes) rather than any kind of btree such as red-black. But on the other hand it was not straightforward, its a long time since I have implemented a tree and this is very hot code. Kimi was massively helpful here in testing prototypes and is actually quite a good C generator if carefully instructed. A thing to remember is that SQLite trees (native, or LMDB, whatever) know nothing about indexes, they just use a comparator function to determine if node X data is greater/less than node Y data, and the AVL trees uses the same comparator function. This AVL implementation does seem to work, but it isn't trustable until it has had a lot more testing. LUMO_NOEPH=1 switches off all of this in-RAM backing and puts BTREE_SINGLE btrees back through the original on-disk LMDB temp environment. In this case the 3 bugs which lead to this whole patch series are still present, so they need to be fixed (also in TODO.md). For now LUMO_NOEPH=1 is for developer testing only, it doesn't make sense for users. Previous commits fixed backend-generate to add an "include-only" feature so ephemeral.c gets a backend shim but no standalone compile unit. check-in: bd8a225546 user: danshearer tags: trunk | |
| 09:34 | Delete 511-byte index key limit item. Yay! check-in: b33a3dbedc user: danshearer tags: trunk | |
| 09:09 | It looks like we just fixed a whole category of seeming-reads hitting COW instead, by implementing ephemeral tables in RAM rather than opening an LMDB scratch environment. Very satisfying to delete a sentence that says "40% speed hit" :-) check-in: 2caaa72821 user: danshearer tags: trunk | |
| 09:04 | Explain the new include-only keyword, and just generally tell people what is going on with this file and why simply editing it is not sufficient to get the build to notice. check-in: 961ecb66ee user: danshearer tags: trunk | |
| 08:58 | Document the existing backend-generator design. When it came to adding ephemeral files, I had no idea how this works. check-in: 939e20eef8 user: danshearer tags: trunk | |
| 08:44 | Update for Lumo relaunch. This file may no longer be needed; review sometime. check-in: 5f9387c36b user: danshearer tags: trunk | |
| 08:37 | not-fork.d/sqlite3/Makefile - delete unused FILE var check-in: 7a1eb120e7 user: danshearer tags: trunk | |
| 08:26 | not-fork/sqlite3: generate rename rules with printf, not echo dash is the default /bin/sh shell on several operating systems including Debian/Ubuntu. POSIX leaves echo's handling of backslashes implementation-defined. Under dash echo collapses "\\b" to "\b", so regenerating backend-rename.mod produced patterns like '\bbtree\.c\b' instead of the correct '\\bbtree\\.c\\b'. This was quite subtle because the btree.c -> lumo_btree.c rename silently didn't happen so the native SQLite btree is compiled instead of the backend, and libsqlite3.so / tclsqlite3 fail to link. Even more confusing, the existing files were not generated on dash and work. And finally, backend-generate is only run rarely, and manually, when a backend replaces an entire SQLite file (or stops doing so.) Switch to using printf because that is guaranteed on all shells. check-in: b0d7bf98c1 user: danshearer tags: trunk | |
| 02:58 | Understanding SQLite ephemeral tables and why they are a real problem for LumoSQL, and how we solve them. This is really complicated stuff and requires review from experts in SQLite internals. check-in: 31923d9f02 user: danshearer tags: trunk | |
|
2026-06-07
| ||
| 14:38 | Zap the amalgamation manually before rebuilding it. Will be fixed when we have an amalgamation build target. check-in: f51bdf9751 user: danshearer tags: trunk | |
| 11:18 | Wee explanatory note at the top check-in: 48b187283e user: danshearer tags: trunk | |
| 08:33 | Implement sqlite3BtreeIsEmpty() stub via mdb_stat instead of always returning "empty". Found by doing anything in Fossil that does a join with 3 or more tables, because it returns zero rows. Examples: fossil timeline, fossil leaves, fossil branch list. This function was introduced in SQLite 3.51. The stub was returning *pRes=1 unconditionally, oops. All we do is copy the behaviour of sqlite3BtreeRowCountEst and it works. check-in: 03db288c2b user: danshearer tags: trunk | |
| 02:41 | Lint check-in: 9e7d4e1ce0 user: danshearer tags: trunk | |
| 02:20 | Get the names and dates right check-in: a6a696242f user: danshearer tags: trunk | |
| 01:57 | Lint check-in: a8fcd5adc8 user: danshearer tags: trunk | |
| 01:52 | Many fixes check-in: d0267c5b95 user: danshearer tags: trunk | |
| 01:39 | Ready for release check-in: 6a2889b6e7 user: danshearer tags: trunk | |
| 01:20 | Ready for release check-in: f1f5a9b4df user: danshearer tags: trunk | |
| 00:55 | Getting ready for release 0.8 check-in: 7ca4bf2c10 user: danshearer tags: trunk | |
| 00:42 | Now to consult the actual people named in this... history is hard! check-in: 7063e7a2f4 user: danshearer tags: trunk | |
|
2026-06-05
| ||
| 19:17 | linting check-in: e2c42d0d38 user: danshearer tags: trunk | |
| 18:20 | Linting recipe check-in: 6d2ddf75e6 user: danshearer tags: trunk | |
| 18:15 | Document mutex bug in libfossil compilation. Make it clear this is a temporary hacky file. check-in: 04467dd60a user: danshearer tags: trunk | |
| 12:43 | Handbuilt testing recipe for building libfossil/fossil on NixOS. This is a fragile hacky test and should not survive in its current form. check-in: 94d4ebd0f0 user: danshearer tags: trunk | |
|
2026-06-04
| ||
| 23:34 | Add missing version floor. This mod was being applied to versions of SQLite that can't use it and didn't have a corresponding bit of code to patch, so it aborted. Fix comments. check-in: e521819b7a user: danshearer tags: trunk | |
| 22:41 | Moved the R/shiny basic analysis script to loft/ . It is likely this will be completely replaced with a much better analysis and plotting system, but its worth having the original I wrote around until we have decided what we are going to do. check-in: 93ad97a0a1 user: dan tags: trunk | |
| 22:38 | The new-doc tree was intended to absorb the collection of markdown files in doc/ into an MDBook project. We may well still do that, but for the 2026 relaunch I'm just putting it into the unspecified bits-and-pieces loft/ directory. check-in: b0bd137d2a user: dan tags: trunk | |
| 22:37 | This is a history of significant changes in the SQLite code such that we are forced to have a different .mod file after the change. There are surprisingly few such changes given the size of SQLite and how fast it changes internally. check-in: c52f0b1de4 user: dan tags: trunk | |
| 12:48 | Getting the history right check-in: 7325f8916b user: dan tags: trunk | |
| 12:33 | Big comment refactor. Clean docs for relaunch release. check-in: 9fe0ea0614 user: dan tags: trunk | |
| 12:32 | Update docs for 2026 and compsci facts check-in: e4c1ee5546 user: dan tags: trunk | |
| 05:18 | More thinking about --version check-in: aa47586d77 user: dan tags: trunk | |
| 04:58 | Remembering how we got to the first usable release in 2026... check-in: 6333949733 user: dan tags: trunk | |
|
2026-06-03
| ||
| 13:19 | Clean up comments check-in: 1ccdb681e7 user: dan tags: trunk | |
| 13:17 | LMDB backend fixes uncovered by fuzzing for queries returning wrong rowcounts and in one case a spurious SQLITE_CORRUPT. The fuzz harnesses are full of SQL that no sane person would use. It might make sense for LumoSQL to have its own fuzzer in test/ . There's a structural choice here with ephemeral btrees now storing the whole record as the LMDB key. Hope it doesn't have a performance impact, but we'll see. lumoCursorSpills needed some fixing which isn't surprising. LEFT JOIN counts were off and that was really tricky cursor stuff. Thanks to anonymous for doing the hard work on that one. check-in: 92c8aa280c user: dan tags: trunk | |
|
2026-06-02
| ||
| 22:46 | More build testing and doc improvements. Finally add 'make doctor' script. check-in: 2c4b0f189b user: dan tags: trunk | |
| 17:52 | Better examples in the Fossil/libfossil doc check-in: 47b6b6ac2f user: dan tags: trunk | |
| 17:34 | One line fix to btree.c to fix ORDER BY ... DESC with a constraint. Both backends. Add SQL test repro_order-by.sql , and added tests to order.test. Ported the txt read->write upgrade fix from 0.9 to v1.0. We're going to have to be very careful to keep these in synch. check-in: 2dbf12f452 user: dan tags: trunk | |
| 15:34 | Lots of more testing, more doc fixes... check-in: 74720f9c46 user: dan tags: trunk | |
| 14:24 | Remove rowsum reference in Fossil docs, it's a pointless exercise. Fossil is already a Merkle DAG of checksums. check-in: ef342cde1c user: dan tags: trunk | |
| 14:18 | Improved documentation a lot: * test all the amalgamation steps * dropped old distros and ancient docker instructions * moved benchmarking stuff out of loft/ into the top level, because its status is clear and good * lots of updates to how-to-install, remove bitrot * update design-corruption-detection-and-magic.md for the fact that most of this is now implemented * improve benchmarking docs Also added: * make doctor to test dependencies * a one-line bug in btree.c that gave 250 'ERR1' rows in benchmarks.sqlite check-in: 08d3eafb8c user: dan tags: trunk | |
|
2026-06-01
| ||
| 11:06 | Improve accuracy check-in: 81afeb4d0c user: dan tags: trunk | |
| 10:01 | Clarify check-in: 86a9844b05 user: dan tags: trunk | |