Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
240 check-ins using file doc/meetings/lumo-meeting-notes-2022-01-14.md version 6b23f20b2b
|
2026-06-04
| ||
| 12:32 | Update docs for 2026 and compsci facts check-in: e4c1ee5546 user: dan tags: trunk | |
| 05:18 | More thinking about --version check-in: aa47586d77 user: dan tags: trunk | |
| 04:58 | Remembering how we got to the first usable release in 2026... check-in: 6333949733 user: dan tags: trunk | |
|
2026-06-03
| ||
| 13:19 | Clean up comments check-in: 1ccdb681e7 user: dan tags: trunk | |
| 13:17 | LMDB backend fixes uncovered by fuzzing for queries returning wrong rowcounts and in one case a spurious SQLITE_CORRUPT. The fuzz harnesses are full of SQL that no sane person would use. It might make sense for LumoSQL to have its own fuzzer in test/ . There's a structural choice here with ephemeral btrees now storing the whole record as the LMDB key. Hope it doesn't have a performance impact, but we'll see. lumoCursorSpills needed some fixing which isn't surprising. LEFT JOIN counts were off and that was really tricky cursor stuff. Thanks to anonymous for doing the hard work on that one. check-in: 92c8aa280c user: dan tags: trunk | |
|
2026-06-02
| ||
| 22:46 | More build testing and doc improvements. Finally add 'make doctor' script. check-in: 2c4b0f189b user: dan tags: trunk | |
| 17:52 | Better examples in the Fossil/libfossil doc check-in: 47b6b6ac2f user: dan tags: trunk | |
| 17:34 | One line fix to btree.c to fix ORDER BY ... DESC with a constraint. Both backends. Add SQL test repro_order-by.sql , and added tests to order.test. Ported the txt read->write upgrade fix from 0.9 to v1.0. We're going to have to be very careful to keep these in synch. check-in: 2dbf12f452 user: dan tags: trunk | |
| 15:34 | Lots of more testing, more doc fixes... check-in: 74720f9c46 user: dan tags: trunk | |
| 14:24 | Remove rowsum reference in Fossil docs, it's a pointless exercise. Fossil is already a Merkle DAG of checksums. check-in: ef342cde1c user: dan tags: trunk | |
| 14:18 | Improved documentation a lot: * test all the amalgamation steps * dropped old distros and ancient docker instructions * moved benchmarking stuff out of loft/ into the top level, because its status is clear and good * lots of updates to how-to-install, remove bitrot * update design-corruption-detection-and-magic.md for the fact that most of this is now implemented * improve benchmarking docs Also added: * make doctor to test dependencies * a one-line bug in btree.c that gave 250 'ERR1' rows in benchmarks.sqlite check-in: 08d3eafb8c user: dan tags: trunk | |
|
2026-06-01
| ||
| 11:06 | Improve accuracy check-in: 81afeb4d0c user: dan tags: trunk | |
| 10:01 | Clarify check-in: 86a9844b05 user: dan tags: trunk | |
|
2026-05-31
| ||
| 22:04 | Small doc fixes check-in: d97d55256a user: dan tags: trunk | |
|
2026-05-30
| ||
| 12:43 | Total of the last month of piecemeal changes... this is getting close to LumoSQL v0.8 with LMDBv1.0 now committed with one example of encrytion, Fossil known to work, and much more. The docs now focus on expert users because there is enough now to potentially interest embedded and related developers, and we can take it from there if this actually happens. check-in: fed7b90293 user: dan tags: trunk | |
|
2026-05-29
| ||
| 21:13 | Getting ready for v0.8, which will include LMDBv1.0... check-in: 2811bfc059 user: dan tags: trunk | |
| 20:18 | Tiny change: we now reflect design intention in the test suite. We don't require all rows produced by a rowsum-enabled binary to have a trailing checksum. If its there we verify it, if it isn't we say nothing. This is because we let users choose when they do and don't use rowsums. check-in: 8126e3e817 user: dan tags: trunk | |
| 20:14 | vacuum.test caused a segfault with rowsum=on on some versions of SQLite. gdb showed this was lumoExtensionAdd() not checking parameters. SQLite 3.35's VACUUM bulk-loader path on indexes invokes our OP_IdxInsert hook with an empty key for the destination database, triggering the crash. Later SQLite versions (3.36+) don't implement VACUUM in the same way, so the bug remained but wasn't triggered on later versions. check-in: 58799ba7b9 user: dan tags: trunk | |
| 17:46 | Update the version pinning. These lock files only apply when --use-upstream-lock is passed to build.tcl, and it uses the maximum versions we have said we know. The default is not to set this flag, and the build will opportunistically fetch the latest using not-forking. For most software this is going to eventually fail, walking up the versions like that, but is often surprising how far it is possible to get. With this version lock change, LumoSQL is saying it can now recommend users safely leap forward 4 years of software development. check-in: 64fc9e790b user: dan tags: trunk | |
| 17:18 | Tightened up the .mod files so they are more precise about where the patches should go. check-in: 099c0be77d user: dan tags: trunk | |
| 15:33 | 1000+ benchmark runs plus some callgrind gives pretty reliable comparisons of native vs LMDB backends in various circumstances. check-in: 123cfaddfd user: dan tags: trunk | |
| 15:32 | Lots of work on Fossil/libfossil, including test. check-in: 6e4dacddef user: dan tags: trunk | |
| 11:04 | Spelling check in docs, formatting etc. check-in: 1ba8fb059b user: dan tags: trunk | |
|
2026-05-28
| ||
| 12:53 | Add .timeout 30000 in two places because of intra-process contention. Previously we gave up on inter-process contention (ie multiple build/bench scripts running at once) but there are still cases where multiple workers from one benchmark script carefully calculated not to trip over one another still contend on database access (which is standard SQLite in WAL mode.) check-in: bf7d39da79 user: danshearer tags: trunk | |
|
2026-05-27
| ||
| 17:33 | Oops - need to re-create vcs directory check-in: 91587b208d user: danshearer tags: trunk | |
| 17:24 | Fix concurrency bug by just serialising everything. Not great, but reliable for now. The real problem is contention in the not-fork cache, which needs fixed in not-fork. There was a difference between build.tcl's cache directory and the one set in this script. Script now follows build.tcl. Also add --clean-cache to build matrix which will scrub all vcs directories (but not the repos, the expensive bit to create). This handles the case of broken Fossil checkouts. check-in: 75467453cb user: danshearer tags: trunk | |
| 17:07 | Fixed two bugs: - make benchmark looked like it did a build first, but didn't really. Now it does. - make benchmark got very confused if a build failed part-way through. check-in: b0b7249542 user: danshearer tags: trunk | |
| 13:53 | Latest build / bench scripts for running on a big machine, plus sample benchmark results database check-in: 5c6c0ad15f user: danshearer tags: trunk | |
| 13:40 | Latest comparison analysis check-in: b05471622d user: danshearer tags: trunk | |
| 13:10 | Results of weeks of comparing LMDB to native SQLite btree, getting ready to show the experts who know these codebases down to the instruction count level. TODO is nice and short, and there's some handy hints in loft/basic-change-verification.md but hopefully the various make build and make test/bench targets are good enough most of the time. check-in: ef826a0112 user: danshearer tags: trunk | |
| 12:22 | Fixed two bugs in make test-sql Found another place to change the default rowsum alg to blake3_256 check-in: dae75b830f user: danshearer tags: trunk | |
| 11:16 | Fixed rowsum test for default blake3_256 check-in: d3a6738ace user: danshearer tags: trunk | |
|
2026-05-26
| ||
| 23:08 | Test to go with the change in rowsum default. check-in: ade40e15e4 user: danshearer tags: trunk | |
| 23:07 | Make BLAKE3 the default hashing algorithm, which is ~70% faster than SHA256. It will be still faster again when we enable SIMD, but that is some build system modification requiring a lot of thought. check-in: ecf7f42ab1 user: danshearer tags: trunk | |
| 23:00 | Remove migration of old schemas in test database. If it becomes a problem we can sort it out then. Meanwhile keep it simple. check-in: d0d5aff9f4 user: danshearer tags: trunk | |
| 22:59 | Change default for rowsum on: no longer errors if a rowsum-enabled binary reads a non-rowsum database. The behaviour wasn't wrong, just unexpected and not very useful. check-in: 00ef120c0c user: danshearer tags: trunk | |
| 22:18 | Fix bug with opening an lmdb txn with an invalid dbi handle in some circumstances (such as every `fossil commit`.) check-in: 02a2ca70ff user: danshearer tags: trunk | |
| 12:43 | Updated test system, can now compare on-disk database against multiple binary types Fixed VACUUM and VACUUM INTO on all database types including :memory: Rowsum project complete and covered with tests check-in: 9007ccb4f9 user: danshearer tags: trunk | |
| 11:27 | Doc how to create an SQLite amalgamation with Lumo including LMDB-backed check-in: 37ebef4e32 user: danshearer tags: trunk | |
| 11:25 | Fix rowsum and add the test query tool I've been using for a bit. The query tool closely follows the benchmark query tool. check-in: d2c216d322 user: danshearer tags: trunk | |
| 10:42 | Preparing for new test system. check-in: 412d389db1 user: danshearer tags: trunk | |
| 10:41 | Final version of ROWSUM before next release. This exposes a checksum at the end of every row via an SQLite scalar function, adds test cases. The core functionality was already working but some bugfixes were needed. Proven to be readable still by stock SQLite, and added coverage tests. I'm not happy with the test harness because there is currently no way of trying the same thing with two different binaries (rowsum-enabled and not rowsum-enabled). I have a draft replacement test system to push sometime soon. check-in: 62026ca784 user: danshearer tags: trunk | |
| 08:04 | Proper plan for v1.0, which is now very tight and simple check-in: 94ac4b8286 user: danshearer tags: trunk | |
|
2026-05-25
| ||
| 14:14 | Put benchmark DB in WAL mode. Note that if we used LMDB under LumoSQL for the benchmarking/testing SQLite databases we wouldn't have (or need) WAL made, because LMDB doesn't block readers, but we're wise to use the widely-known standard sqlite3 format for data we wish to share with the world. check-in: 954bb0d25e user: danshearer tags: trunk | |
| 14:02 | Changed strategy for 'make test'. Originally wanted to be as close to SQLite as possible, but this doesn't make sense because (at this stage) LumoSQL is only changing storage engines and all of the SQLite storage engine probes are highly specific to the native SQLite backend. All of the other probes are about parsing SQL and the like, which we are not touching. So until we need it, not importing the SQLite test suite. This also means we don't need to have any copies of SQLite around linked to TCL, which the test suite relies on. This works with LumoSQL, but it's another complication we don't need. Instead we use the sqlite3 CLI only, and test using Lumo-specific commands we know stress the storage layer. Also commit much-modified btree.c/lumo.build from infinite testing against Fossil. This is pretty stable now. Must be getting close to forking everything for LMDBv1.0! check-in: 68789abbb3 user: danshearer tags: trunk | |
| 08:44 | Copied the idea of benchmarking and repurposed it for test. Test results are stored in an sqlite file, and there is a tool to query it. Implemented `make test-sql' Renamed `make test` to `make test-benchmarking-works` Created tool/test-sql-filter.tcl Imported SQL tests from SQLite check-in: 8d21cb9742 user: danshearer tags: trunk | |
|
2026-05-24
| ||
| 18:59 | Fix VACUUM INTO bug, which wasn't obvious check-in: 63f51f1157 user: danshearer tags: trunk | |
| 18:23 | Add version knowledge, so that probes can skip tests that will fail because particular features weren't available in that version. It only does backwards in time not ranges, but since SQLite is pretty keen on backwards compatibility that won't be a problem we hope. check-in: 01552793a3 user: danshearer tags: trunk | |
| 18:22 | Update copyright headers check-in: f23992e0d5 user: danshearer tags: trunk | |
| 18:06 | Fill in 3 more btree stubs, and the first backup stub. This is getting btree.c in shape for a hard fork to the LMDBv1.0 btree.c. check-in: 7f9c958472 user: danshearer tags: trunk | |
| 18:05 | Allow colon ranges (eg 3.52.0:3.54.0) which is what not-fork supports anyway. Sometimes this feels nicer that the + and - suffixes, which work like this: 3.30.1+ means --version-range "3.30.1:" ie 3.30.1 and everything newer 3.30.1- means --version-range ":3.30.1" ie 3.30.1 and everything older check-in: 82888986e2 user: danshearer tags: trunk | |
| 17:44 | Add new tests for 4 new btree.c stub fillings-out Verified every test by hand and updated all comments check-in: 778ef273d9 user: danshearer tags: trunk | |
| 05:30 | Fix FTS5, rtree, and WITHOUT-ROWID under OPTION_ROWSUM=on. Replace the trailing-serial-type heuristic in lumoExtensionAdd / lumoExtensionPresent . This made 6 SQL test probes suddenly pass. check-in: eea66261eb user: danshearer tags: trunk | |
| 05:27 | Long key support. After trying various different schemes, the simplest one works. We compile LMDB with MDB_MAXKEYSIZE=0, which tells LMDB to compute its key size from the page size at initialisation time. this is is approximately 1980 bytes when page sizes are 4k, which solves the problem of LMDB's default being 411 bytes. We do not have a goal of being compatible with other LMDB databases so that is fine. Anything above this returns SQLITE_TOOBIG . check-in: 89325afa90 user: danshearer tags: trunk | |
| 00:49 | One possible error case I forgot in long-keys plan check-in: fef7beea33 user: danshearer tags: trunk | |
|
2026-05-23
| ||
| 23:33 | Remove .NOTPARALLEL , but added a comment explaining what parallelism we do and don't support. `make -jN` is safe but doesn't really do anything. Read the comment to see how we achieve that effect. check-in: acc8bb60e7 user: danshearer tags: trunk | |
| 23:21 | Delete ancient container target in Makefile, and a couple of dead files. check-in: 03137915f5 user: danshearer tags: trunk | |
| 23:11 | LumoSQL was designed for safe concurrency, the build system locks anything to do with a target before working on it. So for example two build/bench processes concurrently will work, one wins and the other either skips this target or waits. What was not concurrent was global shared access that applies to all targets, things like Makefile.options or benchmarks.sqlite which any target may want access to. Testing with concurrent runs easily showed up all of the cases fixed here, and hopefully enough careful checking has been done so that we don't have any more lurking. check-in: 3b5d6f10c7 user: danshearer tags: trunk | |
| 19:09 | Comment out race condition in build.tcl, discovered when running lots of benchmarks and a build on a big shared storage machine. This isn't the best answer but it will solve the problem while I work out if its hiding other problems. check-in: 1add4f00de user: danshearer tags: trunk | |
| 16:34 | I *think* this empty commit is the way to remove an empty directory in fossil following `fossil mv --hard FROM-X TO-Y`. check-in: 0f5a130129 user: danshearer tags: trunk | |
| 16:32 | Moved more historical stuff to loft. some of this is likely to be useful, but not just yet and its important to be able to see LumoSQL at a glance. I suspect now I'm using slurm for benchmarking I should look at Ruben's kbench files again. check-in: 8c51cf0db4 user: danshearer tags: trunk | |
| 16:24 | Update for 2026 reboot check-in: f9795168bf user: danshearer tags: trunk | |
| 16:20 | Move alternative-tests to loft/ , and note in the relevant section of TODO.md. check-in: c6acf2ec66 user: danshearer tags: trunk | |
| 16:18 | Start directory for old/cruft/unknown/interesting files that don't yet have a known purpose or obvious place to live. check-in: 44265fbf3b user: danshearer tags: trunk | |
| 16:09 | Explain alternative-tests, which will be very useful when we finish off rowsum work but is not currently a particularly useful 'make test' target. check-in: 33b5cb9958 user: danshearer tags: trunk | |
| 15:53 | Fix misleading comment about empty files being dead weight. Empty files override the SQLite amalgamation mechanism as explained in the last commit. check-in: 5ef6351e85 user: danshearer tags: trunk | |
| 15:47 | This reverts "Remove dead code" (44507d22), because the SQLite amalgamation is a beast. Claudio had these there by careful design, because these are how the LMDB backend suppresses upstream SQLite's WAL/mutex code from the amalgamation. The not-fork sed pass rewrites every reference to wal.c (etc.) in src/main.mk, Makefile.in, mksqlite3c.tcl etc to lumo_wal.c, which is a 4-line shim that #includes the *empty* backend file. This makes the amalgamation tool read zero bytes where it would otherwise pull in src/wal.c, and it works fine. Oops. check-in: 77b224d81a user: danshearer tags: trunk | |
| 15:33 | Clean up the stub implementation of SQLite's backup API. Write up in TODO so that we don't forget to implement this using LMDB's backup facilities. check-in: 6f9be6e788 user: danshearer tags: trunk | |
| 15:19 | Remove dead code. These are stub files and their handling code from when we thought we were going to fill them out. Some of them were zero bytes. It's a smaller cleanup than it looks but definitely makes life easier and safer. check-in: 44507d2285 user: danshearer tags: trunk | |
| 15:02 | Finished the thinking of long keys implementation, in the end it looks like a few hundred lines in btree.c , and a lot of care. check-in: 998604be37 user: danshearer tags: trunk | |
| 14:22 | Lots more thinking about long keys and rowsums. check-in: 4314ba1a08 user: danshearer tags: trunk | |
| 09:14 | Removed obsolete distro references, fixed formatting, updated for the new rebooted LumoSQL check-in: c5a9bcdabe user: danshearer tags: trunk | |
| 09:10 | More bringing README up to date for the relaunch check-in: 0c4819cffe user: danshearer tags: trunk | |
| 08:26 | Warn when warming up an empty cache, because otherwise it can seem like nothing is happening as many Mb are downloaded the first time check-in: 5108303e28 user: danshearer tags: trunk | |
| 08:25 | Add instructions for the non-root case, typically on a shared compute cluster of the sort ideal for monster LumoSQL build runs. check-in: e4815bd084 user: danshearer tags: trunk | |
|
2026-05-15
| ||
| 11:10 | Remove BDB from everywhere including documentation except for historical factual references. BDB was a useful part of the original proof of concept, showing that multiple backends were possible. The port was done by Oracle, long-since abandoned, as is BDB. Also started to update the documentation generally. Many things have changed since 2022, and we now have a feeling for which of the many original goals are likely to be most useful. check-in: 83e970b26e user: danshearer tags: trunk | |
|
2026-05-06
| ||
| 08:33 | Updated for the many changes to 2026 check-in: a2ac713d02 user: danshearer tags: trunk | |
|
2026-05-05
| ||
| 00:24 | Update status of the checksum VFS, which is maintained and still has lessons for us check-in: 09245f16bb user: danshearer tags: trunk | |
| 00:19 | Refine the rowsum argument with respect to what exists already in 2026 check-in: 2e53111960 user: danshearer tags: trunk | |
| 00:15 | Update database support claims for 2026 check-in: 2b6a0c9245 user: danshearer tags: trunk | |
|
2026-05-04
| ||
| 23:46 | Tighten up crypto thoughts check-in: 0651665169 user: danshearer tags: trunk | |
| 23:40 | Tighten up LMDB v1.0 draft plan for comment check-in: 9894a41503 user: danshearer tags: trunk | |
|
2026-05-02
| ||
| 09:59 | Update mirror comment check-in: 24cca33fed user: danshearer tags: trunk | |
|
2026-04-30
| ||
| 17:14 | Fix strangeness with LMDB v1.0 numbering in preparation for 1.0 port Document the benchmark SQLite database schema and give examples of useful queries check-in: 5420617f59 user: danshearer tags: trunk | |
| 17:05 | Plan for and comments about LMDB 1.0 backend, which will be another not-fork recipe and involve page level crypto at the later stages. check-in: dd5e63598d user: danshearer tags: trunk | |
|
2026-04-29
| ||
| 16:34 | Notes on current status check-in: 1d0c11edf0 user: danshearer tags: trunk | |
| 12:33 | Summarise work remaining in Lumo as-is in TODO.md Add TODO-CRYPTO.md, which is only about page-level crypto via LMDB which cut a v1.0 branch yesterday Row-level crypto is only addressed in the docs, there is no code or prototype work check-in: fb7b27fd17 user: danshearer tags: trunk | |
|
2026-04-27
| ||
| 06:42 | Corrected comments in sql coverage for current supported status. There's a lot less broken now, which gives us nice regression tests for the future. check-in: 12288843ad user: danshearer tags: trunk | |
| 04:12 | Reduced the lower bound for LumoSQL to be SQLite 3.30.0 (August 2020). This needed five small #if SQLITE_VERSION_NUMBER additions not-fork.d/lmdb/files/btree.c and pager.c. Things like the pager filename, and a couple of the Btree transaction API function calls were renamed around 3.34, and a minor change to BtreeMaxPageCount's rootpage type which changed in 3.33 etc. This is probably where the lower bound will stay unless there a user who is stuck on an older version really really wants another backend. check-in: cad2ad9692 user: danshearer tags: trunk | |
|
2026-04-26
| ||
| 20:03 | Document the limitations of rowsum, which now works for all normal schemas. Later we can make it work for all cases. The fix isn't hard, just requires a care not to break vdbe. check-in: 42b1a0b6ee user: danshearer tags: trunk | |
| 19:01 | Made rowsum work: * In vdbe: fix OP_Column and OP_Insert_Idx * In vdbeAdd.c: stop returning corruption on non-rowid last column * In sql-coverage, add all the failure conditions found for rowsum (loads - 7 in total, although most were due to a few errors.) check-in: bb02b3969a user: danshearer tags: trunk | |
| 12:08 | Add rowsum tests check-in: 37835880b0 user: danshearer tags: trunk | |
|
2026-04-25
| ||
| 20:18 | Fix FTS3 cursor mis-positioning bug: sqlite3BtreeMovetoUnpacked (via either of its two helpers) lied to the caller whenever the search key compared larger than every existing row in the table. Every SQLite optimiser path issuing a "seek just past this key" trusted the cursor's current row, but was actually reading the wrong row whenever the seek key happened to exceed the table's maximum. Added one probe to the coverage suite. LMDB coverage went from 60 pass / 4 expected_fail / 0 unexpected_pass to 64 pass / 1 expected_fail / 0 unexpected_pass (with one more probe in the suite, so 65 → 65 total); no-backend went from 64 pass / 0 expected_fail to 65 pass / 0 expected_fail. check-in: c3a3bac37d user: danshearer tags: trunk | |
| 16:47 | Added FTS5 to build flags. FTS5 only works for the no-backend builds, but it stops errors on other builds that test for FTS5 existence even though FTS5 errors when exercised. Not quite as obvious as it looked: upstream enables FTS4 by default. sql-coverage now probes for FTS5. check-in: 4d7b976df6 user: danshearer tags: trunk | |
| 16:18 | Deleting cruft, dead files, an #IF 0, etc. check-in: 0965e5ccff user: danshearer tags: trunk | |
| 15:46 | LMDB doesn't have a pager so everything here is a no-op. There are 14 functions previously returning unintelligent reponses, all now return the best response I can think of. some of these changes have visible effects: - PRAGMA journal_size_limit=N now round-trips: setting a number then querying reports N back, instead of always 0. - PRAGMA journal_mode=delete|truncate|persist|memory now round-trips and behaves like SQLite. - PRAGMA journal_mode=wal is now silently suppressed (the prior journal mode is preserved). - PRAGMA locking_mode now reports normal check-in: b124ab6be3 user: danshearer tags: trunk | |
| 15:09 | Tests for SQL we're supporting. The goal is to watch the failures go down to zero. check-in: df9264f7f9 user: danshearer tags: trunk | |
| 15:08 | ANALYZE on LMDB now writes correct row counts to sqlite_stat1 (was 0 1, now 200 1 for a 200-row table) sqlite_offset() returns distinct values per row instead of always 0. Did this by replacing four LMDB-backend stubs returning bogus values of SQLITE_INTERNAL or 0 where the upstream callers expected real numbers. Specifically: sqlite3BtreeRowCountEst now returns mdb_stat().ms_entries sqlite3BtreeCursorIsValidNN now reports mdb cursor state sqlite3BtreeMaxRecordSize returns 2147483647 upper bound, which I took from the comment in Btree.c: "Everything will still work if this routine always returns 2147483647 (which is the largest record that SQLite can handle)". sqlite3BtreeOffset returns the LMDB data pointer, which I think is probably correct check-in: efc81884ad user: danshearer tags: trunk | |
| 14:22 | These are SQLite settings that have no LMDB equivalent (LMDB doesn't have a page cache, autovacuum, or a configurable page size in the SQLite sense). Stubs returning SQLITE_INTERNAL is wrong because the call sites in pragma.c, vacuum.c, etc. expect a sensible return. The fix is to return a safe and sometimes meaningful default. This converts 16 stubs to intelligent NO-OPs: SetSpillSize SetMmapLimit SetPagerFlags SetPageSize GetPageSize MaxPageCount SecureDelete GetRequestedReserve SetAutoVacuum GetAutoVacuum Checkpoint IncrVacuum SetVersion ConnectionCount GetReserveNoMutex HeaderSizeBtree check-in: 092d4d9ac5 user: danshearer tags: trunk | |
| 14:05 | Made SAVEPOINT work on LMDB backend. SQLite's native btree backend rolls back at the page level via the rollback journal. LMDB's rollback unit is a transaction, so we open MDB_txn at SAVEPOINT BEGIN, abort it at ROLLBACK TO, commit it into its parent at RELEASE. SQLite WAL is lower level, so savepoint behaviour is the same whether journalling or WAL. check-in: 1f1c4b73e9 user: danshearer tags: trunk | |
| 13:41 | Update for new benchmark commands (-average, -compare etc) check-in: 04cd997a03 user: danshearer tags: trunk | |
| 13:40 | Add -average , -compare , -group-by and the TSV (tab-separated values) format. TSV is native to Unix and generally easier and safer. check-in: 958dfddfa0 user: danshearer tags: trunk | |
| 13:37 | A partial or stale lock directory can really confuse subsequent runs. Now it removes the lock dir regardless of the nature of the exit. check-in: 7a8332b984 user: danshearer tags: trunk | |
| 13:35 | CREATE TABLE dropped CHECK clauses with LMDB backend, because the stub for sqlite3BtreeIsReadonly always returned read-only. Invisible except with EXPLAIN INSERT. check-in: 81d5014182 user: danshearer tags: trunk | |
| 06:23 | if versions_list returns nothing for latest sqlite3, exit with error suggesting not-fork cache is corrupt and SQLITE_FOR_DB=. reject an empty SQLITE_FOR_DB. in the bulld loop, if any build name is empty, abort with an internal-error message including the offending build_list, instead of silently trying to open the lock directory. (Which won't work) check-in: 6529085477 user: danshearer tags: trunk | |
| 02:05 | Bringing lumosql up to date for 2026 check-in: 13466cbb10 user: danshearer tags: trunk | |
|
2024-03-07
| ||
| 17:31 | Make "among the most-deployed software", not "the most deployed" check-in: 024abcebc7 user: danshearer tags: trunk | |
|
2023-01-11
| ||
| 11:46 | Add paper on definitions of Functional Encryption. Predicate Encryption seems to be a subclass of Functional Encryption. check-in: be5579f383 user: danshearer tags: trunk | |
|
2023-01-09
| ||
| 14:46 | Add a DOI etc check-in: 39a2897050 user: danshearer tags: trunk | |
| 13:50 | Add paper "Attribute‐based encryption implies identity‐based encryption" check-in: 177c8a57ba user: danshearer tags: trunk | |
|
2022-12-22
| ||
| 12:52 | Rename README to README.md and start to update for current documentation conventions check-in: fc70d1e0e8 user: danshearer tags: trunk | |
| 12:48 | Improve description of source tools. Note: this file was an experiment in Fossil-flavoured Markdown, which should be replaced with Github-flavoured Markdown for use with the new documentation toolchain. check-in: 88245ad2a7 user: danshearer tags: trunk | |
| 12:41 | Start to document R presentation instructions. check-in: 03229234aa user: danshearer tags: trunk | |
| 12:36 | Make the licensing situation even clearer in response to an enquiry. check-in: ae5938dcb1 user: danshearer tags: trunk | |
| 11:50 | Add Ruben and Gabby check-in: 9cc1f1ed2c user: danshearer tags: trunk | |
| 10:27 | Fix typo in README check-in: 025c324cde user: abdur-rahmaanj tags: trunk | |
| 10:17 | Fix typo in CONTRIBUTING check-in: 6ae23b3ac1 user: abdur-rahmaanj tags: trunk | |
|
2022-10-26
| ||
| 12:16 | Add Martina Palmucci's thesis: Securing databases using Attribute Based Encryption and Shamir’s Secret Sharing. check-in: cf77c681fe user: danshearer tags: trunk | |
|
2022-10-24
| ||
| 19:37 | Add reference to supported hardware architectures and operating systems. check-in: b3c54d382c user: danshearer tags: trunk | |
| 15:16 | Adding a fallback to "uname -m" if nothing else works for CPU detection. Also added a special case to recognise Virtio block devices on an emulated riscv running Linux check-in: f02941290c user: Uilebheist tags: trunk | |
| 14:37 | Refactor the first part of the README to add two images and remove verbose words. check-in: 374ba9190d user: danshearer tags: trunk | |
|
2022-09-06
| ||
| 18:17 | Make error message for missing not-fork more useful by pointing at master URL. check-in: 6a0b3966ce user: danshearer tags: trunk | |
| 17:52 | Reinstate comment that GitHub is a read-only mirror check-in: 1530cc70b7 user: danshearer tags: trunk | |
|
2022-08-05
| ||
| 17:09 | CHanging the context size in the fragment patch (vdbe-changes.mod) to cope with a larger range of sqlite versions, including the current latest (3.39.2) check-in: f4a08a00ce user: Uilebheist tags: trunk | |
|
2022-08-04
| ||
| 11:22 | Removing some warnings produced when testing with clang check-in: f80048502d user: Uilebheist tags: trunk | |
| 07:25 | Small update to hardware-detect for FreeBSD on non-x86 check-in: a8d095c05b user: Uilebheist tags: trunk | |
|
2022-08-03
| ||
| 08:27 | Adding an option to specify a version of sqlite3 to build and use to update the results databases (default latest, which was the version used before). Also adding an option to pass a mirror directory to not-fork. Updating documentation. check-in: aacad5c1a9 user: Uilebheist tags: trunk | |
|
2022-08-02
| ||
| 07:19 | Adding recent versions of sqlite3 to the "upstream.lock" file check-in: 881386f1dd user: Uilebheist tags: trunk | |
|
2022-07-10
| ||
| 14:28 | Minor typo check-in: c8427d4252 user: Uilebheist tags: trunk | |
|
2022-07-07
| ||
| 13:48 | adding mdbook settings file check-in: f2dc4ac441 user: moonshine tags: trunk | |
| 13:46 | adding toc preprocessor tag to documentation files check-in: 6cfa464328 user: moonshine tags: trunk | |
|
2022-06-24
| ||
| 08:28 | Add MongoDB blog post as reference check-in: 85f162cbd5 user: rubdos tags: trunk | |
| 07:58 | Fix two incorrect paper dates check-in: dbf69b7e07 user: danshearer tags: trunk | |
| 07:57 | Database/data store internals references check-in: 552050da7f user: danshearer tags: trunk | |
|
2022-06-16
| ||
| 14:13 | fixed internal links to account for new filenames check-in: 0a7a702eaa user: moonshine tags: trunk | |
|
2022-06-14
| ||
| 17:14 | renaming the documentation files and moving it all to doc/ check-in: 17993985fb user: moonshine tags: trunk | |
|
2022-06-11
| ||
| 23:41 | merged the meetbot documentation into CONTRIBUTING.md check-in: 724559ed0a user: moonshine tags: trunk | |
|
2022-06-10
| ||
| 16:35 | Deleted the generated docs from the current merkle tree. In addition, on the server, shunned half a dozen image files so they are physically removed from the repo. check-in: 5d6c6bb96a user: danshearer tags: trunk | |
|
2022-06-03
| ||
| 09:24 | added a chapter about the data structure design motivation check-in: ee4bb010f7 user: moonshine tags: trunk | |
|
2022-05-16
| ||
| 08:44 | Define the LumoSQL timezone as being Brussels local time check-in: 4f5e4447d6 user: danshearer tags: trunk | |
|
2022-05-13
| ||
| 12:35 | Add dudle for open source and accessible meeting scheduling and polling check-in: 5845074ff5 user: danshearer tags: trunk | |
|
2022-05-04
| ||
| 16:48 | Regenerated upstream.lock with full commit ID rather than truncated check-in: 0559d60a5c user: Uilebheist tags: trunk | |
|
2022-04-19
| ||
| 02:36 | This is the first version of the documentation using mdbook, html output is in the book directory, read how.sh for how it's made, I'm ignoring fossil warnings, sorry if I make a mess check-in: 185cc271c1 user: moonshine tags: trunk | |
| 01:46 | added SUMMARY.md, which determines the layout of mdbook check-in: f84e49f558 user: moonshine tags: trunk | |
|
2022-04-18
| ||
| 12:10 | Typo fix check-in: dcddde5bd0 user: Uilebheist tags: trunk | |
|
2022-04-14
| ||
| 20:50 | editing the documentation check-in: a7e4e48d56 user: moonshine tags: trunk | |
|
2022-04-13
| ||
| 14:31 | Adding some extra information so that not-fork can find locally mirrored sources. Adding "upstream.lock" files containing the list of revisions known at the time of committing. build.tcl will not use them automatically because they require the latest not-fork, but they are there in preparation. check-in: 24305c2754 user: Uilebheist tags: trunk | |
|
2022-04-11
| ||
| 08:02 | Adding sha-256 checksum for BDB download check-in: 19b27de50a user: Uilebheist tags: trunk | |
|
2022-04-09
| ||
| 17:08 | benchmark-filter.tcl now recognised "-datasize x,x" as equal to "-datasize x" for compatibility; also "-details" shows some more information check-in: fc56cbc572 user: Uilebheist tags: trunk | |
|
2022-04-06
| ||
| 11:58 | Add comments by gabby_bch check-in: 8447484636 user: luisdamiano tags: trunk | |
|
2022-04-05
| ||
| 20:17 | Filled in Bayes' question sheet, improving the questions check-in: f639f979b9 user: danshearer tags: trunk | |
|
2022-04-03
| ||
| 09:21 | Missing "?" in regexp caused wrong parsing of -datasize in benchmark-filter check-in: 8e57ba6dd6 user: Uilebheist tags: trunk | |
|
2022-03-29
| ||
| 07:46 | Make -fields and -tests option behave as documented, and clarified the note about specifying multiple files for -import check-in: 19b267dcc2 user: Uilebheist tags: trunk | |
|
2022-03-25
| ||
| 20:39 | Adding two dream questions for a future time check-in: 1da188fb48 user: Uilebheist tags: trunk | |
| 19:45 | Add notes on model questions check-in: 0e4991bfa8 user: luisdamiano tags: trunk | |
| 17:11 | Add missing '@' in curl command. check-in: 335b6b5f99 user: danshearer tags: trunk | |
| 15:23 | Add SPDX attribution and copyright block to top of cluster scripts check-in: 1d5cd5ad02 user: danshearer tags: trunk | |
| 12:26 | Double the number of jobs for a given DATASIZE range. All read sizes are now tried against all write sizes. Also improve dry run output. check-in: 697c400942 user: danshearer tags: trunk | |
| 12:01 | Script to submit jobs to work-loop.sh. Initial version does lots of sanity checks and will loop over a simple DATASIZE progression. check-in: c8dcb7d0e8 user: danshearer tags: trunk | |
|
2022-03-24
| ||
| 16:35 | Simplify. check-in: 621c25fec0 user: danshearer tags: trunk | |
|
2022-03-23
| ||
| 22:06 | Update analysis check-in: 43c0fce7f0 user: luisdamiano tags: trunk | |
| 19:05 | deleting benchmark result databse check-in: 5a008b6181 user: moonshine tags: trunk | |
| 18:58 | Add mockup page for "Releases and Downloads". This will take some thought because it involves explaining accessibly how the 700+ work products of the LumoSQL project relate to the needs of different groups of people. It is likely this will change a lot. check-in: de6e4c6f4f user: danshearer tags: trunk | |
| 18:54 | Adding some new columns in the benchmark-filter output; updating docs check-in: 3af40942ee user: Uilebheist tags: trunk | |
| 18:13 | shiny app check-in: 419b855e11 user: moonshine tags: trunk | |
|
2022-03-22
| ||
| 15:56 | Remove references to the Lumodoc project. Vita brevis. check-in: 9b4643c430 user: danshearer tags: trunk | |
| 15:15 | First step to using Pandoc to create a landing page. Updated index.md text. Added Makefile. Changed CSS to be more accessible. check-in: 496c17d370 user: danshearer tags: trunk | |
| 13:42 | Update for 2022 state of LumoSQL. Remove all references to the documentation project, which is now merged into the main repo. check-in: a36257b64e user: danshearer tags: trunk | |
| 13:25 | Updating code to make it easier to use alternative test lists (tests only for now, rather than benchmarks). Updated documentation. Added a new set of tests to check the interaction between Lumo metadata and unmodified sqlite check-in: d8a12beeb4 user: Uilebheist tags: trunk | |
| 12:24 | Initial commit of existing top-level website check-in: df140f7aab user: danshearer tags: trunk | |
| 12:23 | Add link to license discussion, bump copyright year check-in: 5cfd71c3ab user: danshearer tags: trunk | |
| 12:22 | Initial commit of existing license website check-in: 5f11692b4e user: danshearer tags: trunk | |
|
2022-03-19
| ||
| 19:32 | Moving Gabby's refactored docs from the lumodoc project to the lumosql project. BibLaTex references also moved. The lumodoc project will be closed down. check-in: e25fb647f6 user: danshearer tags: trunk | |
| 12:07 | Adding a note "rotational" or "non-rotational" for Linux devices when the kernel has this information. This helps distinguishing solid state disks when no other information will tell us. check-in: b3fb0cac73 user: Uilebheist tags: trunk | |
|
2022-03-18
| ||
| 08:56 | Adding timestamps to some of the messages likely to be followed by longer processing check-in: 079d9e4b7e user: Uilebheist tags: trunk | |
|
2022-03-17
| ||
| 10:38 | Fix link check-in: d387a8ea84 user: danshearer tags: trunk | |
|
2022-03-16
| ||
| 20:44 | Update notes, add analysis script check-in: 491ccc7260 user: luisdamiano tags: trunk | |
| 19:19 | Add analysis notes check-in: 72f5e50d73 user: luisdamiano tags: trunk | |
| 19:12 | Create statistical analysis directory check-in: b0a01567b4 user: danshearer tags: trunk | |
| 14:06 | Updating docs for the options added in the last 2 commits check-in: 24d30eb75f user: Uilebheist tags: trunk | |
| 14:01 | Implementing option to delete selected runs ("-delete"), and also adding option to delete selected runs from a different database ("-delete-from DB"), which can be used to delete runs which were added by copying from another database. check-in: 2f76c2261a user: Uilebheist tags: trunk | |
| 13:55 | Adding a "-normalise" (or "-normalize") to show individual test timings normalised against the total duration, to help comparing results from similar systems but with different overall speed check-in: c491bd90ab user: Uilebheist tags: trunk | |
|
2022-03-15
| ||
| 13:41 | New option, discard_output, to prevent reading the test/benchmark output into a TCL string. This allows very large datasize values without having TCL try to create a string too big and fail. Default is "Off" i.e. same code as before. check-in: c1678c4bae user: Uilebheist tags: trunk | |
|
2022-03-10
| ||
| 16:13 | Removing extra "incr tc" which causes only half of the test to display in "-details" mode check-in: 2d92923346 user: Uilebheist tags: trunk | |
| 15:44 | Deleting empty values before "-add" of a new value, to stop duplicate issues. Fixing wrong docs in the comments at the top of add-missing-comments check-in: 8edc36f300 user: Uilebheist tags: trunk | |
| 14:34 | Update work loop with hostname, Promotheus, SIGUSR for restarts and more failure recovery. check-in: 018bfeff21 user: rubdos tags: trunk | |
|
2022-03-09
| ||
| 20:10 | Be clearer about how very early the Lumion spec is check-in: 43894c12fd user: danshearer tags: trunk | |
|
2022-03-07
| ||
| 16:44 | Fixing "make targets" which had an error creeping in in a previous edit check-in: 77156eeba1 user: Uilebheist tags: trunk | |
| 15:30 | Producing an explicit error message if tclsh cannot be found check-in: ff2e4146a3 user: Uilebheist tags: trunk | |
|
2022-03-04
| ||
| 13:47 | Adding a small script to help fixing old benchmark databases without CPU_COMMENT and DISK_COMMENT check-in: 4d12be27f7 user: Uilebheist tags: trunk | |
|
2022-02-23
| ||
| 13:38 | More cleanup of rowsum / lumo extensions code, and now rowsum also works on indices as well as tables with ROWID. check-in: 5948723391 user: Uilebheist tags: trunk | |
|
2022-02-22
| ||
| 12:54 | Moving some of the rowsum code to a separate file to simplify the vdbe patch and also to clean things up; it now works with sqlite 3.35.0 or newer, but no longer works with 3.34 due to changes in sqlite. Rowsum on indices is still not there yet. check-in: 5b4e671d94 user: Uilebheist tags: trunk | |
| 12:03 | build.tcl seems to have lost SQLITE targets at some point, adding them back. Also adding an option KEEP_SOURCES to keep sources in the build directory even if no longer requires, as the sources help if using a debugger; default is still to delete them check-in: ab4069d24b user: Uilebheist tags: trunk | |
|
2022-02-21
| ||
| 12:30 | Adding a new operation, "cleanup" which removes any targets in the build directory which would need to be rebuilt before using check-in: 06e9b40ead user: Uilebheist tags: trunk | |
| 12:10 | Adding a special version number, "commit-ID" to select a commit ID as opposed to a formal release. check-in: ada192c84f user: Uilebheist tags: trunk | |
|
2022-02-18
| ||
| 14:39 | More formatting fixes. check-in: ec7bd745dc user: danshearer tags: trunk | |
| 14:35 | Fix incorrect SPDX license; update dates; fix formatting. check-in: 216c1503eb user: danshearer tags: trunk | |
| 14:32 | Checked this over because LumoSQL is proposing to have a physical in-person meeting, hosted in a similar venue to where repeated violations occur annually in Brussels at FOSDEM. Various fixes including keeping it succinct, correcting spelling, attributing Mozilla in a nicer way, etc. check-in: 3addccc09a user: danshearer tags: trunk | |
| 14:16 | Fix bugs, especially to note that #action should always be followed by an irc nick, otherwise the actions will not be assigned by the meetbot. check-in: 714aa58379 user: danshearer tags: trunk | |
| 11:15 | The chair must use "#meetingname Some Sensible Name" because the name is now extracted and shown in the description column at https://lumosql.org/meetings. Condense the list of commands into a kind of LumoSQL meeting handbook. check-in: d7b55e5c13 user: danshearer tags: trunk | |
| 09:03 | Understanding another different cpuinfo format check-in: a3d4f1b3e4 user: Uilebheist tags: trunk | |
| 08:53 | Small typo fixed check-in: 652fe04b24 user: Uilebheist tags: trunk | |
| 08:37 | More code cleanup; adding support for detecting virtio devices on FreeBSD check-in: f4facd26ac user: Uilebheist tags: trunk | |
| 02:43 | Improve comments. check-in: 36be59895b user: danshearer tags: trunk | |
| 02:26 | Clean up disk device detection a lot more. check-in: 10357b27c4 user: danshearer tags: trunk | |
| 01:13 | Generalise detection method for Virtio devices, having found the official up-to-date spec which says there is always a Vendor ID, just not a human- readable string. This needs to be filled out for non-Linux OSs. check-in: 4723297999 user: danshearer tags: trunk | |
|
2022-02-17
| ||
| 23:53 | Fix scope issue preventing Virtio block devices from being reported check-in: 5501d90a9d user: danshearer tags: trunk | |
| 20:21 | More types of network filesystems; CPU detection for FreeBSD and NetBSD; detection of Virtio block devices on Linux check-in: a9486e3846 user: Uilebheist tags: trunk | |
| 15:38 | Some changes to run on FreeBSD: finding tclsh; finding editline/readline; finding a C compiler. Removing code which went into not-forking, and requesting at least not-forking 0.5 check-in: df0ca754bf user: Uilebheist tags: trunk | |
|
2022-02-16
| ||
| 15:30 | Making sure -copy runs in a transaction check-in: 245372f057 user: Uilebheist tags: trunk | |
| 14:33 | Adding a simple tool to add a database of results to a combined database check-in: 875c786139 user: Uilebheist tags: trunk | |
| 13:42 | Making sure -datasize r,w works in benchmark-filter; also fixing an issue when it did not know that the BDB backend has SQLite 3.18.2 so it would not show these results when filtering by that version check-in: 2272416f95 user: Uilebheist tags: trunk | |
| 11:27 | Making indices unique to help spotting duplicates and errors; adding a N_TESTS column to -list output. check-in: d3835cde3b user: Uilebheist tags: trunk | |
| 10:48 | Update -help text check-in: e26311e2bf user: danshearer tags: trunk | |
| 10:39 | Prefix relative path with ./ It works locally with --page doc/ckout/README.md but not online. check-in: d59bd47461 user: danshearer tags: trunk | |
| 09:51 | Checking for duplicate run IDs when adding to a database check-in: fe10dfa097 user: Uilebheist tags: trunk | |
| 09:02 | Adding a -count output option; also fixing the -invalid check to know about the new backend-date field check-in: 77a90ba4cb user: Uilebheist tags: trunk | |
| 08:36 | Soft-require Tclx to catch signals if it happens to be present. check-in: 6365eedbc5 user: danshearer tags: trunk | |
|
2022-02-15
| ||
| 17:51 | Adding an option to pass "--cache" to not-fork, to be able to keep all configuration in a single place check-in: 95416c74a4 user: Uilebheist tags: trunk | |
| 17:30 | Allowing abbreviated run IDs in benchmark-filter check-in: 14f68bcc8f user: Uilebheist tags: trunk | |
| 16:31 | Adding a note about sharing not-forking repo, and calling it from the repo instead of installing it. check-in: e70a154448 user: Uilebheist tags: trunk | |
| 16:11 | Add info on shared build/bench infrastructure for use on a cluster, and point to the example Kubernetes directory. check-in: 79c5b1a8d1 user: danshearer tags: trunk | |
| 16:04 | Cleanups and added details of the Linux image check-in: c33c42083e user: danshearer tags: trunk | |
| 09:49 | Add accessibility section and update contacts. Remove nearly everything else. check-in: 057dbe7ecc user: danshearer tags: trunk | |
|
2022-02-14
| ||
| 14:43 | Forcing a rebuild for (older) objects which do not have commit timestamps check-in: 63d409b70d user: Uilebheist tags: trunk | |
| 13:48 | Adding timestamps from sqlite, the backend, and the notforking sources to the run_data in the benchmark/test database, for future reference. check-in: 167a17fd04 user: Uilebheist tags: trunk | |
|
2022-02-13
| ||
| 19:13 | Fixing small issue introduced in previous change to build.tcl check-in: b58723e4cd user: Uilebheist tags: trunk | |
| 18:49 | Adding some more options to benchmark-filter.tcl and updating documentation; also replacing examples to show the current output format check-in: 7c62da5f9e user: Uilebheist tags: trunk | |
|
2022-02-11
| ||
| 10:57 | Adding a different output mode for -summary and -quick, showing one test/benchmark per column and one target per row, instead of the default of one target per column, one test/benchmark per row. check-in: 07ab6b9668 user: Uilebheist tags: trunk | |
| 09:39 | Allowing the DATASIZE option to have one or two numbers, to specify a separate "read" and "write" data size, with code to remain compatible with older targets which specify just one number. check-in: 79fadb6ad8 user: Uilebheist tags: trunk | |
|
2022-02-10
| ||
| 18:22 | Changing minimum LMDB version we use to 0.9.14, as older versions have issues check-in: 78c38ba2a4 user: Uilebheist tags: trunk | |
| 14:53 | Add missing variable (-fields was crashing) check-in: 0027f15981 user: danshearer tags: trunk | |
| 13:40 | Remove lumosql- prefix for Kubernetes service benchmarks check-in: 1b93639321 user: rubdos tags: trunk | |
| 12:32 | Fixes for ARMv7 which probably also apply to other 32 bit processors check-in: 212bde35a6 user: Uilebheist tags: trunk | |
| 10:14 | Add Kubernetes-based benchmarking service configuration. check-in: 88553e315e user: rubdos tags: trunk | |
|
2022-02-09
| ||
| 22:08 | Removing some debugging output left over from previous changes check-in: 1bb3fb9cc0 user: Uilebheist tags: trunk | |
| 21:39 | Removing variable added by mistake check-in: b514e28fdf user: Uilebheist tags: trunk | |
| 19:15 | Reducing the space needed in the "build" directory by deleting sources after they are no longer needed; adding information to help a later run make a better decision on whether a target needs to be rebuilt. check-in: 048357a41a user: Uilebheist tags: trunk | |
| 18:59 | Slight tidy up of hardware-detect, and making it work for the SD card on a Raspberry PI check-in: afefad759f user: Uilebheist tags: trunk | |
| 18:51 | Edit meeting notes to point to the new meeting repository used with the wonderful Meetbot. Update for ACTION items done just for historical completeness. check-in: e104e14126 user: danshearer tags: trunk | |