Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
183 check-ins using file doc/project-announce-phase2.md version 41278ab86f
|
2026-06-11
| ||
| 05:46 | Adjust for the fossil focus of v0.8 Leaf check-in: fcbca05c27 user: danshearer tags: trunk | |
| 03:59 | Tidy up for release. check-in: d7ad558140 user: danshearer tags: trunk | |
| 02:55 | Finally tagged version 0.8, so its not on the TODO any more... check-in: b7479f7ca4 user: danshearer tags: trunk | |
| 02:52 | Final linting before v0.8 release check-in: 20a33e263a user: danshearer tags: trunk, 0.8, v0.8 | |
|
2026-06-10
| ||
| 23:33 | Fix typo check-in: 4a5ee31955 user: danshearer tags: trunk | |
| 23:22 | Linting the recipe instructions check-in: 7681db978d user: danshearer tags: trunk | |
| 23:01 | More Fossil build recipe bugfixes check-in: e41d43e597 user: danshearer tags: trunk | |
| 22:03 | More checking the Fossil-on-LumoSQL recipe, and fixing bugs the recipe reveals. check-in: 5693d9ae96 user: danshearer tags: trunk | |
| 20:44 | Lots of changes to the "Building Fossil with LumoSQL for experts" type docs. check-in: b0e5d08f98 user: danshearer tags: trunk | |
|
2026-06-09
| ||
| 19:30 | More detail on the build system making the union of all specified and default targets, with example of how to get just one single target if that's what you actually want. check-in: 4b0bf0b38c user: danshearer tags: trunk | |
| 19:29 | Fix corruption test check-in: f5701a776c user: danshearer tags: trunk | |
| 18:35 | Fix bug that prevented most rowsums working properly: check_rowsum() returns tri-state (-1, 0, 1) but the caller tested as a boolean. check-in: 3f5ed208fc user: danshearer tags: trunk | |
| 13:30 | Shout out to the Test Anything Protocol fmt check-in: cc55c9d173 user: danshearer tags: trunk | |
| 13:26 | Add fossil stress test script. Lint the Fossil on Lumo recipe. check-in: 9727301cbe user: danshearer tags: trunk | |
|
2026-06-08
| ||
| 21:13 | Fix text formatting errors check-in: 8b91102f93 user: danshearer tags: trunk | |
| 18:09 | Add testing section check-in: da1bf3ae7e user: danshearer tags: trunk | |
| 17:49 | Explain that this is all temporary, pending expert review. check-in: 699e029d0d user: danshearer tags: trunk | |
| 17:47 | Add instructions for encrypted Lumo+LMDB backend to Fossil. check-in: 5726770902 user: danshearer tags: trunk | |
| 14:06 | Make sure the two fossil patches are in the recipe, one to let LMDB databases be opened (because they are directories not files), and another implementing HTTP 1.1 so Fossil isn't blocked by application firewalls. check-in: 4281356acd user: danshearer tags: trunk | |
| 11:20 | New files for the big ephemeral commit (see commit message for bd8a225546, doc/context-about-ephemeral-tables.md , and fossil/importing-massive-trees-to-fossil-lumosql.md) check-in: 4a16ba04aa user: danshearer tags: trunk | |
| 11:13 | Temporary raise the lower bound for SQLite versions to 3.43.0. As mentioned in TODO there has been some drift in vdbe.c which stops things like 'make test-sql' work out of the box. check-in: bf9d17aa90 user: danshearer tags: trunk | |
| 10:53 | For the people who keep wanting to put NetBSD or Linux into Fossil... check-in: 1bca31e090 user: danshearer tags: trunk | |
| 10:25 | Fix hang and SQLITE_INTERNAL in ephemeral (BTREE_SINGLE) btrees This is a large change, fixing 3 showstopper bugs found by Fossil but also addressing much bigger architectural problem. The bugs were: - Infinite hang: sqlite3BtreeCursorHasMoved/Restore had no ephemeral case, so an ephemeral cursor (no LMDB handle) was reported "moved" while restore said "unchanged", looping forever. Seen in fossil finfo/timeline/branch/leaves. - SQLITE_INTERNAL on rowid-keyed ephemeral tables only handled index btrees. We now support (ephInsertIntKey/SeekInt/DeleteInt/IntKey). - Range seeks: sqlite3BtreeIndexMoveto only worked for exact positions. Replaced with a positioned ephSeek that supports SeekGE/GT/LE/LT and IdxLE/GT. See doc/context-about-ephemeral-tables.md for SQLite background and how it is relevant to LumoSQL. For this transient use case, LMDB's write amplification into reliable storage doesn't make sense. LMDB 0.9.x also has the 511-byte limit for long keys (eg Fossil check-in comments are used as keys and they can be up to 12k!) and even LMDB v1.0's much larger default key size is insufficient. So we have two architectural improvements as well as 3 bugfixes. We now fully implement ephemeral tables, hopefully, and we definitely implement it in a better way than using LMDB COW. The implementation is in the sqlite3 wrapper and depends on LUMO_HAVE_EPHEMERAL_BACKEND (settable via LUMO_NOEPH=1) and is identical for lmdb and lmdbv1. We do more-or-less what native SQLite already does for transient tables, which is: keep them in RAM, no transactions, no locking, discard at statement end. We grow in malloced memory without bound, so this is a great way to crash-test LumoSQL. SQLite does not grow without bound, spilling out to a temporary file if it exceeds SQLITE_DEFAULT_TEMP_CACHE_SIZE. As the LumoSQL TODO.md says, in due course LumoSQL needs to detect memory pressure and move back to LMDB, and the use case implies enormous transient tables so its best to do with LMDBv1 which has other advantages. But this patch leaves it as an unsolved problem, and in practical terms LMDB-backed Fossil handles the SQLite tree so that's a pretty good start. The in-memory implementation was on the one hand straightforward once I had decided it should be an AVL tree (because its more lookups than inserts/deletes) rather than any kind of btree such as red-black. But on the other hand it was not straightforward, its a long time since I have implemented a tree and this is very hot code. Kimi was massively helpful here in testing prototypes and is actually quite a good C generator if carefully instructed. A thing to remember is that SQLite trees (native, or LMDB, whatever) know nothing about indexes, they just use a comparator function to determine if node X data is greater/less than node Y data, and the AVL trees uses the same comparator function. This AVL implementation does seem to work, but it isn't trustable until it has had a lot more testing. LUMO_NOEPH=1 switches off all of this in-RAM backing and puts BTREE_SINGLE btrees back through the original on-disk LMDB temp environment. In this case the 3 bugs which lead to this whole patch series are still present, so they need to be fixed (also in TODO.md). For now LUMO_NOEPH=1 is for developer testing only, it doesn't make sense for users. Previous commits fixed backend-generate to add an "include-only" feature so ephemeral.c gets a backend shim but no standalone compile unit. check-in: bd8a225546 user: danshearer tags: trunk | |
| 09:34 | Delete 511-byte index key limit item. Yay! check-in: b33a3dbedc user: danshearer tags: trunk | |
| 09:09 | It looks like we just fixed a whole category of seeming-reads hitting COW instead, by implementing ephemeral tables in RAM rather than opening an LMDB scratch environment. Very satisfying to delete a sentence that says "40% speed hit" :-) check-in: 2caaa72821 user: danshearer tags: trunk | |
| 09:04 | Explain the new include-only keyword, and just generally tell people what is going on with this file and why simply editing it is not sufficient to get the build to notice. check-in: 961ecb66ee user: danshearer tags: trunk | |
| 08:58 | Document the existing backend-generator design. When it came to adding ephemeral files, I had no idea how this works. check-in: 939e20eef8 user: danshearer tags: trunk | |
| 08:44 | Update for Lumo relaunch. This file may no longer be needed; review sometime. check-in: 5f9387c36b user: danshearer tags: trunk | |
| 08:37 | not-fork.d/sqlite3/Makefile - delete unused FILE var check-in: 7a1eb120e7 user: danshearer tags: trunk | |
| 08:26 | not-fork/sqlite3: generate rename rules with printf, not echo dash is the default /bin/sh shell on several operating systems including Debian/Ubuntu. POSIX leaves echo's handling of backslashes implementation-defined. Under dash echo collapses "\\b" to "\b", so regenerating backend-rename.mod produced patterns like '\bbtree\.c\b' instead of the correct '\\bbtree\\.c\\b'. This was quite subtle because the btree.c -> lumo_btree.c rename silently didn't happen so the native SQLite btree is compiled instead of the backend, and libsqlite3.so / tclsqlite3 fail to link. Even more confusing, the existing files were not generated on dash and work. And finally, backend-generate is only run rarely, and manually, when a backend replaces an entire SQLite file (or stops doing so.) Switch to using printf because that is guaranteed on all shells. check-in: b0d7bf98c1 user: danshearer tags: trunk | |
| 02:58 | Understanding SQLite ephemeral tables and why they are a real problem for LumoSQL, and how we solve them. This is really complicated stuff and requires review from experts in SQLite internals. check-in: 31923d9f02 user: danshearer tags: trunk | |
|
2026-06-07
| ||
| 14:38 | Zap the amalgamation manually before rebuilding it. Will be fixed when we have an amalgamation build target. check-in: f51bdf9751 user: danshearer tags: trunk | |
| 11:18 | Wee explanatory note at the top check-in: 48b187283e user: danshearer tags: trunk | |
| 08:33 | Implement sqlite3BtreeIsEmpty() stub via mdb_stat instead of always returning "empty". Found by doing anything in Fossil that does a join with 3 or more tables, because it returns zero rows. Examples: fossil timeline, fossil leaves, fossil branch list. This function was introduced in SQLite 3.51. The stub was returning *pRes=1 unconditionally, oops. All we do is copy the behaviour of sqlite3BtreeRowCountEst and it works. check-in: 03db288c2b user: danshearer tags: trunk | |
| 02:41 | Lint check-in: 9e7d4e1ce0 user: danshearer tags: trunk | |
| 02:20 | Get the names and dates right check-in: a6a696242f user: danshearer tags: trunk | |
| 01:57 | Lint check-in: a8fcd5adc8 user: danshearer tags: trunk | |
| 01:52 | Many fixes check-in: d0267c5b95 user: danshearer tags: trunk | |
| 01:39 | Ready for release check-in: 6a2889b6e7 user: danshearer tags: trunk | |
| 01:20 | Ready for release check-in: f1f5a9b4df user: danshearer tags: trunk | |
| 00:55 | Getting ready for release 0.8 check-in: 7ca4bf2c10 user: danshearer tags: trunk | |
| 00:42 | Now to consult the actual people named in this... history is hard! check-in: 7063e7a2f4 user: danshearer tags: trunk | |
|
2026-06-05
| ||
| 19:17 | linting check-in: e2c42d0d38 user: danshearer tags: trunk | |
| 18:20 | Linting recipe check-in: 6d2ddf75e6 user: danshearer tags: trunk | |
| 18:15 | Document mutex bug in libfossil compilation. Make it clear this is a temporary hacky file. check-in: 04467dd60a user: danshearer tags: trunk | |
| 12:43 | Handbuilt testing recipe for building libfossil/fossil on NixOS. This is a fragile hacky test and should not survive in its current form. check-in: 94d4ebd0f0 user: danshearer tags: trunk | |
|
2026-06-04
| ||
| 23:34 | Add missing version floor. This mod was being applied to versions of SQLite that can't use it and didn't have a corresponding bit of code to patch, so it aborted. Fix comments. check-in: e521819b7a user: danshearer tags: trunk | |
| 22:41 | Moved the R/shiny basic analysis script to loft/ . It is likely this will be completely replaced with a much better analysis and plotting system, but its worth having the original I wrote around until we have decided what we are going to do. check-in: 93ad97a0a1 user: dan tags: trunk | |
| 22:38 | The new-doc tree was intended to absorb the collection of markdown files in doc/ into an MDBook project. We may well still do that, but for the 2026 relaunch I'm just putting it into the unspecified bits-and-pieces loft/ directory. check-in: b0bd137d2a user: dan tags: trunk | |
| 22:37 | This is a history of significant changes in the SQLite code such that we are forced to have a different .mod file after the change. There are surprisingly few such changes given the size of SQLite and how fast it changes internally. check-in: c52f0b1de4 user: dan tags: trunk | |
| 12:48 | Getting the history right check-in: 7325f8916b user: dan tags: trunk | |
| 12:33 | Big comment refactor. Clean docs for relaunch release. check-in: 9fe0ea0614 user: dan tags: trunk | |
| 12:32 | Update docs for 2026 and compsci facts check-in: e4c1ee5546 user: dan tags: trunk | |
| 05:18 | More thinking about --version check-in: aa47586d77 user: dan tags: trunk | |
| 04:58 | Remembering how we got to the first usable release in 2026... check-in: 6333949733 user: dan tags: trunk | |
|
2026-06-03
| ||
| 13:19 | Clean up comments check-in: 1ccdb681e7 user: dan tags: trunk | |
| 13:17 | LMDB backend fixes uncovered by fuzzing for queries returning wrong rowcounts and in one case a spurious SQLITE_CORRUPT. The fuzz harnesses are full of SQL that no sane person would use. It might make sense for LumoSQL to have its own fuzzer in test/ . There's a structural choice here with ephemeral btrees now storing the whole record as the LMDB key. Hope it doesn't have a performance impact, but we'll see. lumoCursorSpills needed some fixing which isn't surprising. LEFT JOIN counts were off and that was really tricky cursor stuff. Thanks to anonymous for doing the hard work on that one. check-in: 92c8aa280c user: dan tags: trunk | |
|
2026-06-02
| ||
| 22:46 | More build testing and doc improvements. Finally add 'make doctor' script. check-in: 2c4b0f189b user: dan tags: trunk | |
| 17:52 | Better examples in the Fossil/libfossil doc check-in: 47b6b6ac2f user: dan tags: trunk | |
| 17:34 | One line fix to btree.c to fix ORDER BY ... DESC with a constraint. Both backends. Add SQL test repro_order-by.sql , and added tests to order.test. Ported the txt read->write upgrade fix from 0.9 to v1.0. We're going to have to be very careful to keep these in synch. check-in: 2dbf12f452 user: dan tags: trunk | |
| 15:34 | Lots of more testing, more doc fixes... check-in: 74720f9c46 user: dan tags: trunk | |
| 14:24 | Remove rowsum reference in Fossil docs, it's a pointless exercise. Fossil is already a Merkle DAG of checksums. check-in: ef342cde1c user: dan tags: trunk | |
| 14:18 | Improved documentation a lot: * test all the amalgamation steps * dropped old distros and ancient docker instructions * moved benchmarking stuff out of loft/ into the top level, because its status is clear and good * lots of updates to how-to-install, remove bitrot * update design-corruption-detection-and-magic.md for the fact that most of this is now implemented * improve benchmarking docs Also added: * make doctor to test dependencies * a one-line bug in btree.c that gave 250 'ERR1' rows in benchmarks.sqlite check-in: 08d3eafb8c user: dan tags: trunk | |
|
2026-06-01
| ||
| 11:06 | Improve accuracy check-in: 81afeb4d0c user: dan tags: trunk | |
| 10:01 | Clarify check-in: 86a9844b05 user: dan tags: trunk | |
|
2026-05-31
| ||
| 22:04 | Small doc fixes check-in: d97d55256a user: dan tags: trunk | |
|
2026-05-30
| ||
| 12:43 | Total of the last month of piecemeal changes... this is getting close to LumoSQL v0.8 with LMDBv1.0 now committed with one example of encrytion, Fossil known to work, and much more. The docs now focus on expert users because there is enough now to potentially interest embedded and related developers, and we can take it from there if this actually happens. check-in: fed7b90293 user: dan tags: trunk | |
|
2026-05-29
| ||
| 21:13 | Getting ready for v0.8, which will include LMDBv1.0... check-in: 2811bfc059 user: dan tags: trunk | |
| 20:18 | Tiny change: we now reflect design intention in the test suite. We don't require all rows produced by a rowsum-enabled binary to have a trailing checksum. If its there we verify it, if it isn't we say nothing. This is because we let users choose when they do and don't use rowsums. check-in: 8126e3e817 user: dan tags: trunk | |
| 20:14 | vacuum.test caused a segfault with rowsum=on on some versions of SQLite. gdb showed this was lumoExtensionAdd() not checking parameters. SQLite 3.35's VACUUM bulk-loader path on indexes invokes our OP_IdxInsert hook with an empty key for the destination database, triggering the crash. Later SQLite versions (3.36+) don't implement VACUUM in the same way, so the bug remained but wasn't triggered on later versions. check-in: 58799ba7b9 user: dan tags: trunk | |
| 17:46 | Update the version pinning. These lock files only apply when --use-upstream-lock is passed to build.tcl, and it uses the maximum versions we have said we know. The default is not to set this flag, and the build will opportunistically fetch the latest using not-forking. For most software this is going to eventually fail, walking up the versions like that, but is often surprising how far it is possible to get. With this version lock change, LumoSQL is saying it can now recommend users safely leap forward 4 years of software development. check-in: 64fc9e790b user: dan tags: trunk | |
| 17:18 | Tightened up the .mod files so they are more precise about where the patches should go. check-in: 099c0be77d user: dan tags: trunk | |
| 15:33 | 1000+ benchmark runs plus some callgrind gives pretty reliable comparisons of native vs LMDB backends in various circumstances. check-in: 123cfaddfd user: dan tags: trunk | |
| 15:32 | Lots of work on Fossil/libfossil, including test. check-in: 6e4dacddef user: dan tags: trunk | |
| 11:04 | Spelling check in docs, formatting etc. check-in: 1ba8fb059b user: dan tags: trunk | |
|
2026-05-28
| ||
| 12:53 | Add .timeout 30000 in two places because of intra-process contention. Previously we gave up on inter-process contention (ie multiple build/bench scripts running at once) but there are still cases where multiple workers from one benchmark script carefully calculated not to trip over one another still contend on database access (which is standard SQLite in WAL mode.) check-in: bf7d39da79 user: danshearer tags: trunk | |
|
2026-05-27
| ||
| 17:33 | Oops - need to re-create vcs directory check-in: 91587b208d user: danshearer tags: trunk | |
| 17:24 | Fix concurrency bug by just serialising everything. Not great, but reliable for now. The real problem is contention in the not-fork cache, which needs fixed in not-fork. There was a difference between build.tcl's cache directory and the one set in this script. Script now follows build.tcl. Also add --clean-cache to build matrix which will scrub all vcs directories (but not the repos, the expensive bit to create). This handles the case of broken Fossil checkouts. check-in: 75467453cb user: danshearer tags: trunk | |
| 17:07 | Fixed two bugs: - make benchmark looked like it did a build first, but didn't really. Now it does. - make benchmark got very confused if a build failed part-way through. check-in: b0b7249542 user: danshearer tags: trunk | |
| 13:53 | Latest build / bench scripts for running on a big machine, plus sample benchmark results database check-in: 5c6c0ad15f user: danshearer tags: trunk | |
| 13:40 | Latest comparison analysis check-in: b05471622d user: danshearer tags: trunk | |
| 13:10 | Results of weeks of comparing LMDB to native SQLite btree, getting ready to show the experts who know these codebases down to the instruction count level. TODO is nice and short, and there's some handy hints in loft/basic-change-verification.md but hopefully the various make build and make test/bench targets are good enough most of the time. check-in: ef826a0112 user: danshearer tags: trunk | |
| 12:22 | Fixed two bugs in make test-sql Found another place to change the default rowsum alg to blake3_256 check-in: dae75b830f user: danshearer tags: trunk | |
| 11:16 | Fixed rowsum test for default blake3_256 check-in: d3a6738ace user: danshearer tags: trunk | |
|
2026-05-26
| ||
| 23:08 | Test to go with the change in rowsum default. check-in: ade40e15e4 user: danshearer tags: trunk | |
| 23:07 | Make BLAKE3 the default hashing algorithm, which is ~70% faster than SHA256. It will be still faster again when we enable SIMD, but that is some build system modification requiring a lot of thought. check-in: ecf7f42ab1 user: danshearer tags: trunk | |
| 23:00 | Remove migration of old schemas in test database. If it becomes a problem we can sort it out then. Meanwhile keep it simple. check-in: d0d5aff9f4 user: danshearer tags: trunk | |
| 22:59 | Change default for rowsum on: no longer errors if a rowsum-enabled binary reads a non-rowsum database. The behaviour wasn't wrong, just unexpected and not very useful. check-in: 00ef120c0c user: danshearer tags: trunk | |
| 22:18 | Fix bug with opening an lmdb txn with an invalid dbi handle in some circumstances (such as every `fossil commit`.) check-in: 02a2ca70ff user: danshearer tags: trunk | |
| 12:43 | Updated test system, can now compare on-disk database against multiple binary types Fixed VACUUM and VACUUM INTO on all database types including :memory: Rowsum project complete and covered with tests check-in: 9007ccb4f9 user: danshearer tags: trunk | |
| 11:27 | Doc how to create an SQLite amalgamation with Lumo including LMDB-backed check-in: 37ebef4e32 user: danshearer tags: trunk | |
| 11:25 | Fix rowsum and add the test query tool I've been using for a bit. The query tool closely follows the benchmark query tool. check-in: d2c216d322 user: danshearer tags: trunk | |
| 10:42 | Preparing for new test system. check-in: 412d389db1 user: danshearer tags: trunk | |
| 10:41 | Final version of ROWSUM before next release. This exposes a checksum at the end of every row via an SQLite scalar function, adds test cases. The core functionality was already working but some bugfixes were needed. Proven to be readable still by stock SQLite, and added coverage tests. I'm not happy with the test harness because there is currently no way of trying the same thing with two different binaries (rowsum-enabled and not rowsum-enabled). I have a draft replacement test system to push sometime soon. check-in: 62026ca784 user: danshearer tags: trunk | |
| 08:04 | Proper plan for v1.0, which is now very tight and simple check-in: 94ac4b8286 user: danshearer tags: trunk | |
|
2026-05-25
| ||
| 14:14 | Put benchmark DB in WAL mode. Note that if we used LMDB under LumoSQL for the benchmarking/testing SQLite databases we wouldn't have (or need) WAL made, because LMDB doesn't block readers, but we're wise to use the widely-known standard sqlite3 format for data we wish to share with the world. check-in: 954bb0d25e user: danshearer tags: trunk | |
| 14:02 | Changed strategy for 'make test'. Originally wanted to be as close to SQLite as possible, but this doesn't make sense because (at this stage) LumoSQL is only changing storage engines and all of the SQLite storage engine probes are highly specific to the native SQLite backend. All of the other probes are about parsing SQL and the like, which we are not touching. So until we need it, not importing the SQLite test suite. This also means we don't need to have any copies of SQLite around linked to TCL, which the test suite relies on. This works with LumoSQL, but it's another complication we don't need. Instead we use the sqlite3 CLI only, and test using Lumo-specific commands we know stress the storage layer. Also commit much-modified btree.c/lumo.build from infinite testing against Fossil. This is pretty stable now. Must be getting close to forking everything for LMDBv1.0! check-in: 68789abbb3 user: danshearer tags: trunk | |
| 08:44 | Copied the idea of benchmarking and repurposed it for test. Test results are stored in an sqlite file, and there is a tool to query it. Implemented `make test-sql' Renamed `make test` to `make test-benchmarking-works` Created tool/test-sql-filter.tcl Imported SQL tests from SQLite check-in: 8d21cb9742 user: danshearer tags: trunk | |
|
2026-05-24
| ||
| 18:59 | Fix VACUUM INTO bug, which wasn't obvious check-in: 63f51f1157 user: danshearer tags: trunk | |
| 18:23 | Add version knowledge, so that probes can skip tests that will fail because particular features weren't available in that version. It only does backwards in time not ranges, but since SQLite is pretty keen on backwards compatibility that won't be a problem we hope. check-in: 01552793a3 user: danshearer tags: trunk | |
| 18:22 | Update copyright headers check-in: f23992e0d5 user: danshearer tags: trunk | |
| 18:06 | Fill in 3 more btree stubs, and the first backup stub. This is getting btree.c in shape for a hard fork to the LMDBv1.0 btree.c. check-in: 7f9c958472 user: danshearer tags: trunk | |
| 18:05 | Allow colon ranges (eg 3.52.0:3.54.0) which is what not-fork supports anyway. Sometimes this feels nicer that the + and - suffixes, which work like this: 3.30.1+ means --version-range "3.30.1:" ie 3.30.1 and everything newer 3.30.1- means --version-range ":3.30.1" ie 3.30.1 and everything older check-in: 82888986e2 user: danshearer tags: trunk | |
| 17:44 | Add new tests for 4 new btree.c stub fillings-out Verified every test by hand and updated all comments check-in: 778ef273d9 user: danshearer tags: trunk | |
| 05:30 | Fix FTS5, rtree, and WITHOUT-ROWID under OPTION_ROWSUM=on. Replace the trailing-serial-type heuristic in lumoExtensionAdd / lumoExtensionPresent . This made 6 SQL test probes suddenly pass. check-in: eea66261eb user: danshearer tags: trunk | |
| 05:27 | Long key support. After trying various different schemes, the simplest one works. We compile LMDB with MDB_MAXKEYSIZE=0, which tells LMDB to compute its key size from the page size at initialisation time. this is is approximately 1980 bytes when page sizes are 4k, which solves the problem of LMDB's default being 411 bytes. We do not have a goal of being compatible with other LMDB databases so that is fine. Anything above this returns SQLITE_TOOBIG . check-in: 89325afa90 user: danshearer tags: trunk | |
| 00:49 | One possible error case I forgot in long-keys plan check-in: fef7beea33 user: danshearer tags: trunk | |
|
2026-05-23
| ||
| 23:33 | Remove .NOTPARALLEL , but added a comment explaining what parallelism we do and don't support. `make -jN` is safe but doesn't really do anything. Read the comment to see how we achieve that effect. check-in: acc8bb60e7 user: danshearer tags: trunk | |
| 23:21 | Delete ancient container target in Makefile, and a couple of dead files. check-in: 03137915f5 user: danshearer tags: trunk | |
| 23:11 | LumoSQL was designed for safe concurrency, the build system locks anything to do with a target before working on it. So for example two build/bench processes concurrently will work, one wins and the other either skips this target or waits. What was not concurrent was global shared access that applies to all targets, things like Makefile.options or benchmarks.sqlite which any target may want access to. Testing with concurrent runs easily showed up all of the cases fixed here, and hopefully enough careful checking has been done so that we don't have any more lurking. check-in: 3b5d6f10c7 user: danshearer tags: trunk | |
| 19:09 | Comment out race condition in build.tcl, discovered when running lots of benchmarks and a build on a big shared storage machine. This isn't the best answer but it will solve the problem while I work out if its hiding other problems. check-in: 1add4f00de user: danshearer tags: trunk | |
| 16:34 | I *think* this empty commit is the way to remove an empty directory in fossil following `fossil mv --hard FROM-X TO-Y`. check-in: 0f5a130129 user: danshearer tags: trunk | |
| 16:32 | Moved more historical stuff to loft. some of this is likely to be useful, but not just yet and its important to be able to see LumoSQL at a glance. I suspect now I'm using slurm for benchmarking I should look at Ruben's kbench files again. check-in: 8c51cf0db4 user: danshearer tags: trunk | |
| 16:24 | Update for 2026 reboot check-in: f9795168bf user: danshearer tags: trunk | |
| 16:20 | Move alternative-tests to loft/ , and note in the relevant section of TODO.md. check-in: c6acf2ec66 user: danshearer tags: trunk | |
| 16:18 | Start directory for old/cruft/unknown/interesting files that don't yet have a known purpose or obvious place to live. check-in: 44265fbf3b user: danshearer tags: trunk | |
| 16:09 | Explain alternative-tests, which will be very useful when we finish off rowsum work but is not currently a particularly useful 'make test' target. check-in: 33b5cb9958 user: danshearer tags: trunk | |
| 15:53 | Fix misleading comment about empty files being dead weight. Empty files override the SQLite amalgamation mechanism as explained in the last commit. check-in: 5ef6351e85 user: danshearer tags: trunk | |
| 15:47 | This reverts "Remove dead code" (44507d22), because the SQLite amalgamation is a beast. Claudio had these there by careful design, because these are how the LMDB backend suppresses upstream SQLite's WAL/mutex code from the amalgamation. The not-fork sed pass rewrites every reference to wal.c (etc.) in src/main.mk, Makefile.in, mksqlite3c.tcl etc to lumo_wal.c, which is a 4-line shim that #includes the *empty* backend file. This makes the amalgamation tool read zero bytes where it would otherwise pull in src/wal.c, and it works fine. Oops. check-in: 77b224d81a user: danshearer tags: trunk | |
| 15:33 | Clean up the stub implementation of SQLite's backup API. Write up in TODO so that we don't forget to implement this using LMDB's backup facilities. check-in: 6f9be6e788 user: danshearer tags: trunk | |
| 15:19 | Remove dead code. These are stub files and their handling code from when we thought we were going to fill them out. Some of them were zero bytes. It's a smaller cleanup than it looks but definitely makes life easier and safer. check-in: 44507d2285 user: danshearer tags: trunk | |
| 15:02 | Finished the thinking of long keys implementation, in the end it looks like a few hundred lines in btree.c , and a lot of care. check-in: 998604be37 user: danshearer tags: trunk | |
| 14:22 | Lots more thinking about long keys and rowsums. check-in: 4314ba1a08 user: danshearer tags: trunk | |
| 09:14 | Removed obsolete distro references, fixed formatting, updated for the new rebooted LumoSQL check-in: c5a9bcdabe user: danshearer tags: trunk | |
| 09:10 | More bringing README up to date for the relaunch check-in: 0c4819cffe user: danshearer tags: trunk | |
| 08:26 | Warn when warming up an empty cache, because otherwise it can seem like nothing is happening as many Mb are downloaded the first time check-in: 5108303e28 user: danshearer tags: trunk | |
| 08:25 | Add instructions for the non-root case, typically on a shared compute cluster of the sort ideal for monster LumoSQL build runs. check-in: e4815bd084 user: danshearer tags: trunk | |
|
2026-05-15
| ||
| 11:10 | Remove BDB from everywhere including documentation except for historical factual references. BDB was a useful part of the original proof of concept, showing that multiple backends were possible. The port was done by Oracle, long-since abandoned, as is BDB. Also started to update the documentation generally. Many things have changed since 2022, and we now have a feeling for which of the many original goals are likely to be most useful. check-in: 83e970b26e user: danshearer tags: trunk | |
|
2026-05-06
| ||
| 08:33 | Updated for the many changes to 2026 check-in: a2ac713d02 user: danshearer tags: trunk | |
|
2026-05-05
| ||
| 00:24 | Update status of the checksum VFS, which is maintained and still has lessons for us check-in: 09245f16bb user: danshearer tags: trunk | |
| 00:19 | Refine the rowsum argument with respect to what exists already in 2026 check-in: 2e53111960 user: danshearer tags: trunk | |
| 00:15 | Update database support claims for 2026 check-in: 2b6a0c9245 user: danshearer tags: trunk | |
|
2026-05-04
| ||
| 23:46 | Tighten up crypto thoughts check-in: 0651665169 user: danshearer tags: trunk | |
| 23:40 | Tighten up LMDB v1.0 draft plan for comment check-in: 9894a41503 user: danshearer tags: trunk | |
|
2026-05-02
| ||
| 09:59 | Update mirror comment check-in: 24cca33fed user: danshearer tags: trunk | |
|
2026-04-30
| ||
| 17:14 | Fix strangeness with LMDB v1.0 numbering in preparation for 1.0 port Document the benchmark SQLite database schema and give examples of useful queries check-in: 5420617f59 user: danshearer tags: trunk | |
| 17:05 | Plan for and comments about LMDB 1.0 backend, which will be another not-fork recipe and involve page level crypto at the later stages. check-in: dd5e63598d user: danshearer tags: trunk | |
|
2026-04-29
| ||
| 16:34 | Notes on current status check-in: 1d0c11edf0 user: danshearer tags: trunk | |
| 12:33 | Summarise work remaining in Lumo as-is in TODO.md Add TODO-CRYPTO.md, which is only about page-level crypto via LMDB which cut a v1.0 branch yesterday Row-level crypto is only addressed in the docs, there is no code or prototype work check-in: fb7b27fd17 user: danshearer tags: trunk | |
|
2026-04-27
| ||
| 06:42 | Corrected comments in sql coverage for current supported status. There's a lot less broken now, which gives us nice regression tests for the future. check-in: 12288843ad user: danshearer tags: trunk | |
| 04:12 | Reduced the lower bound for LumoSQL to be SQLite 3.30.0 (August 2020). This needed five small #if SQLITE_VERSION_NUMBER additions not-fork.d/lmdb/files/btree.c and pager.c. Things like the pager filename, and a couple of the Btree transaction API function calls were renamed around 3.34, and a minor change to BtreeMaxPageCount's rootpage type which changed in 3.33 etc. This is probably where the lower bound will stay unless there a user who is stuck on an older version really really wants another backend. check-in: cad2ad9692 user: danshearer tags: trunk | |
|
2026-04-26
| ||
| 20:03 | Document the limitations of rowsum, which now works for all normal schemas. Later we can make it work for all cases. The fix isn't hard, just requires a care not to break vdbe. check-in: 42b1a0b6ee user: danshearer tags: trunk | |
| 19:01 | Made rowsum work: * In vdbe: fix OP_Column and OP_Insert_Idx * In vdbeAdd.c: stop returning corruption on non-rowid last column * In sql-coverage, add all the failure conditions found for rowsum (loads - 7 in total, although most were due to a few errors.) check-in: bb02b3969a user: danshearer tags: trunk | |
| 12:08 | Add rowsum tests check-in: 37835880b0 user: danshearer tags: trunk | |
|
2026-04-25
| ||
| 20:18 | Fix FTS3 cursor mis-positioning bug: sqlite3BtreeMovetoUnpacked (via either of its two helpers) lied to the caller whenever the search key compared larger than every existing row in the table. Every SQLite optimiser path issuing a "seek just past this key" trusted the cursor's current row, but was actually reading the wrong row whenever the seek key happened to exceed the table's maximum. Added one probe to the coverage suite. LMDB coverage went from 60 pass / 4 expected_fail / 0 unexpected_pass to 64 pass / 1 expected_fail / 0 unexpected_pass (with one more probe in the suite, so 65 → 65 total); no-backend went from 64 pass / 0 expected_fail to 65 pass / 0 expected_fail. check-in: c3a3bac37d user: danshearer tags: trunk | |
| 16:47 | Added FTS5 to build flags. FTS5 only works for the no-backend builds, but it stops errors on other builds that test for FTS5 existence even though FTS5 errors when exercised. Not quite as obvious as it looked: upstream enables FTS4 by default. sql-coverage now probes for FTS5. check-in: 4d7b976df6 user: danshearer tags: trunk | |
| 16:18 | Deleting cruft, dead files, an #IF 0, etc. check-in: 0965e5ccff user: danshearer tags: trunk | |
| 15:46 | LMDB doesn't have a pager so everything here is a no-op. There are 14 functions previously returning unintelligent reponses, all now return the best response I can think of. some of these changes have visible effects: - PRAGMA journal_size_limit=N now round-trips: setting a number then querying reports N back, instead of always 0. - PRAGMA journal_mode=delete|truncate|persist|memory now round-trips and behaves like SQLite. - PRAGMA journal_mode=wal is now silently suppressed (the prior journal mode is preserved). - PRAGMA locking_mode now reports normal check-in: b124ab6be3 user: danshearer tags: trunk | |
| 15:09 | Tests for SQL we're supporting. The goal is to watch the failures go down to zero. check-in: df9264f7f9 user: danshearer tags: trunk | |
| 15:08 | ANALYZE on LMDB now writes correct row counts to sqlite_stat1 (was 0 1, now 200 1 for a 200-row table) sqlite_offset() returns distinct values per row instead of always 0. Did this by replacing four LMDB-backend stubs returning bogus values of SQLITE_INTERNAL or 0 where the upstream callers expected real numbers. Specifically: sqlite3BtreeRowCountEst now returns mdb_stat().ms_entries sqlite3BtreeCursorIsValidNN now reports mdb cursor state sqlite3BtreeMaxRecordSize returns 2147483647 upper bound, which I took from the comment in Btree.c: "Everything will still work if this routine always returns 2147483647 (which is the largest record that SQLite can handle)". sqlite3BtreeOffset returns the LMDB data pointer, which I think is probably correct check-in: efc81884ad user: danshearer tags: trunk | |
| 14:22 | These are SQLite settings that have no LMDB equivalent (LMDB doesn't have a page cache, autovacuum, or a configurable page size in the SQLite sense). Stubs returning SQLITE_INTERNAL is wrong because the call sites in pragma.c, vacuum.c, etc. expect a sensible return. The fix is to return a safe and sometimes meaningful default. This converts 16 stubs to intelligent NO-OPs: SetSpillSize SetMmapLimit SetPagerFlags SetPageSize GetPageSize MaxPageCount SecureDelete GetRequestedReserve SetAutoVacuum GetAutoVacuum Checkpoint IncrVacuum SetVersion ConnectionCount GetReserveNoMutex HeaderSizeBtree check-in: 092d4d9ac5 user: danshearer tags: trunk | |
| 14:05 | Made SAVEPOINT work on LMDB backend. SQLite's native btree backend rolls back at the page level via the rollback journal. LMDB's rollback unit is a transaction, so we open MDB_txn at SAVEPOINT BEGIN, abort it at ROLLBACK TO, commit it into its parent at RELEASE. SQLite WAL is lower level, so savepoint behaviour is the same whether journalling or WAL. check-in: 1f1c4b73e9 user: danshearer tags: trunk | |
| 13:41 | Update for new benchmark commands (-average, -compare etc) check-in: 04cd997a03 user: danshearer tags: trunk | |
| 13:40 | Add -average , -compare , -group-by and the TSV (tab-separated values) format. TSV is native to Unix and generally easier and safer. check-in: 958dfddfa0 user: danshearer tags: trunk | |
| 13:37 | A partial or stale lock directory can really confuse subsequent runs. Now it removes the lock dir regardless of the nature of the exit. check-in: 7a8332b984 user: danshearer tags: trunk | |
| 13:35 | CREATE TABLE dropped CHECK clauses with LMDB backend, because the stub for sqlite3BtreeIsReadonly always returned read-only. Invisible except with EXPLAIN INSERT. check-in: 81d5014182 user: danshearer tags: trunk | |
| 06:23 | if versions_list returns nothing for latest sqlite3, exit with error suggesting not-fork cache is corrupt and SQLITE_FOR_DB=. reject an empty SQLITE_FOR_DB. in the bulld loop, if any build name is empty, abort with an internal-error message including the offending build_list, instead of silently trying to open the lock directory. (Which won't work) check-in: 6529085477 user: danshearer tags: trunk | |
| 02:05 | Bringing lumosql up to date for 2026 check-in: 13466cbb10 user: danshearer tags: trunk | |
|
2024-03-07
| ||
| 17:31 | Make "among the most-deployed software", not "the most deployed" check-in: 024abcebc7 user: danshearer tags: trunk | |
|
2023-01-11
| ||
| 11:46 | Add paper on definitions of Functional Encryption. Predicate Encryption seems to be a subclass of Functional Encryption. check-in: be5579f383 user: danshearer tags: trunk | |
|
2023-01-09
| ||
| 14:46 | Add a DOI etc check-in: 39a2897050 user: danshearer tags: trunk | |
| 13:50 | Add paper "Attribute‐based encryption implies identity‐based encryption" check-in: 177c8a57ba user: danshearer tags: trunk | |
|
2022-12-22
| ||
| 12:52 | Rename README to README.md and start to update for current documentation conventions check-in: fc70d1e0e8 user: danshearer tags: trunk | |
| 12:48 | Improve description of source tools. Note: this file was an experiment in Fossil-flavoured Markdown, which should be replaced with Github-flavoured Markdown for use with the new documentation toolchain. check-in: 88245ad2a7 user: danshearer tags: trunk | |
| 12:41 | Start to document R presentation instructions. check-in: 03229234aa user: danshearer tags: trunk | |
| 12:36 | Make the licensing situation even clearer in response to an enquiry. check-in: ae5938dcb1 user: danshearer tags: trunk | |
| 11:50 | Add Ruben and Gabby check-in: 9cc1f1ed2c user: danshearer tags: trunk | |
| 10:27 | Fix typo in README check-in: 025c324cde user: abdur-rahmaanj tags: trunk | |
| 10:17 | Fix typo in CONTRIBUTING check-in: 6ae23b3ac1 user: abdur-rahmaanj tags: trunk | |
|
2022-10-26
| ||
| 12:16 | Add Martina Palmucci's thesis: Securing databases using Attribute Based Encryption and Shamir’s Secret Sharing. check-in: cf77c681fe user: danshearer tags: trunk | |
|
2022-10-24
| ||
| 19:37 | Add reference to supported hardware architectures and operating systems. check-in: b3c54d382c user: danshearer tags: trunk | |
| 15:16 | Adding a fallback to "uname -m" if nothing else works for CPU detection. Also added a special case to recognise Virtio block devices on an emulated riscv running Linux check-in: f02941290c user: Uilebheist tags: trunk | |
| 14:37 | Refactor the first part of the README to add two images and remove verbose words. check-in: 374ba9190d user: danshearer tags: trunk | |
|
2022-09-06
| ||
| 18:17 | Make error message for missing not-fork more useful by pointing at master URL. check-in: 6a0b3966ce user: danshearer tags: trunk | |
| 17:52 | Reinstate comment that GitHub is a read-only mirror check-in: 1530cc70b7 user: danshearer tags: trunk | |
|
2022-08-05
| ||
| 17:09 | CHanging the context size in the fragment patch (vdbe-changes.mod) to cope with a larger range of sqlite versions, including the current latest (3.39.2) check-in: f4a08a00ce user: Uilebheist tags: trunk | |
|
2022-08-04
| ||
| 11:22 | Removing some warnings produced when testing with clang check-in: f80048502d user: Uilebheist tags: trunk | |
| 07:25 | Small update to hardware-detect for FreeBSD on non-x86 check-in: a8d095c05b user: Uilebheist tags: trunk | |
|
2022-08-03
| ||
| 08:27 | Adding an option to specify a version of sqlite3 to build and use to update the results databases (default latest, which was the version used before). Also adding an option to pass a mirror directory to not-fork. Updating documentation. check-in: aacad5c1a9 user: Uilebheist tags: trunk | |
|
2022-08-02
| ||
| 07:19 | Adding recent versions of sqlite3 to the "upstream.lock" file check-in: 881386f1dd user: Uilebheist tags: trunk | |
|
2022-07-10
| ||
| 14:28 | Minor typo check-in: c8427d4252 user: Uilebheist tags: trunk | |
|
2022-07-07
| ||
| 13:48 | adding mdbook settings file check-in: f2dc4ac441 user: moonshine tags: trunk | |
| 13:46 | adding toc preprocessor tag to documentation files check-in: 6cfa464328 user: moonshine tags: trunk | |