This runs a hand-authored SQL test suite in test/sql/ that exercises the storage backend. The SQL language layer is unchanged by LumoSQL and covered by
SQLite's own tests, so we do not try to repeat the thousands of tests SQLite is already doing on higher-level functionality.
Builds binaries on demand, runs every *.test against each built sqlite3, and appends results to test-sql.sqlite in the usual way.
make test-sql # runs all tests
make test-sql LUMO_TEST_FILES='spill.test churn.test' # just these two
tclsh tool/test-runner.tcl /path/to/sqlite3 test/sql/churn.test # stand-alone
Testing with file tests
test/file-checks.sh runs checks on database files that need several processes, which .test files cannot express: snapshot reads beside a writer,
the writer lock, read-to-write upgrade, recovery after a killed writer, VACUUM, backup and read-only opens. It takes one built binary, prints PASS, FAIL or
SKIP for each check and writes no results database. It is most relevant to native SQLite in WAL mode and to libdb.
Example:
sh test/file-checks.sh build/3.53.2+lmdbv1-1.0/sqlite3 # always needs a full binary path supplied
A check that does not apply to a backend's semantics is skipped, with the reason given.
Concurrency tests
test/concurrency.sh runs two or more processes on one database and checks what SQLite applications rely on: no lost update, no write skew, a
stable read transaction, a refused read-to-write upgrade after another commit, schema changes and savepoints beside another writer, a survivor that
can write after another writer is killed, the busy timeout, and many short writers. Each test runs several times. Where a second writer encounters an
open write transaction, the result says whether it was refused (busy), waited for the commit, or ran beside it (concurrent):
sh test/concurrency.sh build/3.53.2+libdb-2026.10.7/sqlite3
sh test/concurrency.sh --wal --rounds 10 build/3.53.2/sqlite3
second:waited 3/3 means the second writer waited in all three rounds. A capability test prints HAS or LACKS instead of PASS or FAIL:
concurrent-writers shows whether a second writer on another table runs beside an open write transaction. A backend may lack a capability without
fault, so LACKS does not change the exit status. An INFO line reports throughput for one writer and for four, and their ratio.
Both scripts skip a test that does not fit a backend, with the reason. --apply-all-tests runs it anyway.
Writing test blocks in .test files
do_execsql_test NAME {
SQL ...
} {expected output}
Expected values containing error, not a database, unable to open, or malformed match any error. Whitespace is collapsed.
Roles and cross-binary testing
The command-line binary is the default role. Extra roles are declared in a header and filled from build/<target>/sqlite3, matched on capabilities parsed from the target name:
| Target name | Capabilities |
|---|---|
3.53.0 |
version=3.53.0, backend=btree, rowsum=off |
3.53.0+lmdb-0.9.33 |
version=3.53.0, backend=lmdb, rowsum=off |
3.53.0+rowsum-on |
version=3.53.0, backend=btree, rowsum=on |
3.53.0+lmdb-0.9.33+rowsum-on |
version=3.53.0, backend=lmdb, rowsum=on |
Requirements header:
requires role NAME { key=val ... }
requires db (shared|each) (file|:memory:)
If no built binary has the capabilities a role requires, the test runner skips the whole file. shared file gives all roles one database; each gives each role its own. Default is each :memory:.
Cross-binary directives:
| Directive | Purpose |
|---|---|
do_execsql_test NAME [ROLE] { SQL } { expected } |
Run on ROLE (default if omitted) |
do_compare_test NAME { roles } [differ] { SQL } |
All roles equal, or with differ each differs from first |
do_golden_test NAME GOLDEN { others } { SQL } |
Others match GOLDEN |
do_corrupt_test NAME { marker } { read_sql } |
Flip byte after marker on disk; pass if read_sql reports corruption |
Querying results
tclsh tool/test-sql-filter.tcl # summary
tclsh tool/test-sql-filter.tcl -failures # all failures
tclsh tool/test-sql-filter.tcl -diff 1 2 # changes between runs
tclsh tool/test-sql-filter.tcl -help # full options
Modes: -runs, -summary, -failures, -skips, -categories, -diff, -compare, -test, -export. Filters: -target, -file, -category, -format.
Relationship to benchmarking
make benchmark |
make test-sql |
|
|---|---|---|
| Tests | Performance | Correctness |
| Input | not-fork.d/*/benchmark/*.test |
test/sql/*.test |
| Output | benchmarks.sqlite |
test-sql.sqlite |
| Query tool | tool/benchmark-filter.tcl |
tool/test-sql-filter.tcl |
| Cross-binary | No | Yes |
| Corruption testing | No | Yes |
Both use append-only SQLite databases with queryable history, comparable across runs.