Lumosql

Testing with ≺code≻make test-sql≺/code≻
Login

Testing with ≺code≻make test-sql≺/code≻

This runs a hand-authored SQL test suite in test/sql/ that exercises the storage backend. The SQL language layer is unchanged by LumoSQL and covered by SQLite's own tests, so we do not try to repeat the thousands of tests SQLite is already doing on higher-level functionality. Builds binaries on demand, runs every *.test against each built sqlite3, and appends results to test-sql.sqlite in the usual way.

    make test-sql  # runs all tests
    make test-sql LUMO_TEST_FILES='spill.test churn.test'  # just these two
    tclsh tool/test-runner.tcl /path/to/sqlite3 test/sql/churn.test  # stand-alone

Testing with file tests

test/file-checks.sh runs checks on database files that need several processes, which .test files cannot express: snapshot reads beside a writer, the writer lock, read-to-write upgrade, recovery after a killed writer, VACUUM, backup and read-only opens. It takes one built binary, prints PASS, FAIL or SKIP for each check and writes no results database. It is most relevant to native SQLite in WAL mode and to libdb.

Example:

    sh test/file-checks.sh build/3.53.2+lmdbv1-1.0/sqlite3  # always needs a full binary path supplied

A check that does not apply to a backend's semantics is skipped, with the reason given.

Concurrency tests

test/concurrency.sh runs two or more processes on one database and checks what SQLite applications rely on: no lost update, no write skew, a stable read transaction, a refused read-to-write upgrade after another commit, schema changes and savepoints beside another writer, a survivor that can write after another writer is killed, the busy timeout, and many short writers. Each test runs several times. Where a second writer encounters an open write transaction, the result says whether it was refused (busy), waited for the commit, or ran beside it (concurrent):

    sh test/concurrency.sh build/3.53.2+libdb-2026.10.7/sqlite3
    sh test/concurrency.sh --wal --rounds 10 build/3.53.2/sqlite3

second:waited 3/3 means the second writer waited in all three rounds. A capability test prints HAS or LACKS instead of PASS or FAIL: concurrent-writers shows whether a second writer on another table runs beside an open write transaction. A backend may lack a capability without fault, so LACKS does not change the exit status. An INFO line reports throughput for one writer and for four, and their ratio.

Both scripts skip a test that does not fit a backend, with the reason. --apply-all-tests runs it anyway.

Writing test blocks in .test files

do_execsql_test NAME {
  SQL ...
} {expected output}

Expected values containing error, not a database, unable to open, or malformed match any error. Whitespace is collapsed.

Roles and cross-binary testing

The command-line binary is the default role. Extra roles are declared in a header and filled from build/<target>/sqlite3, matched on capabilities parsed from the target name:

Target name Capabilities
3.53.0 version=3.53.0, backend=btree, rowsum=off
3.53.0+lmdb-0.9.33 version=3.53.0, backend=lmdb, rowsum=off
3.53.0+rowsum-on version=3.53.0, backend=btree, rowsum=on
3.53.0+lmdb-0.9.33+rowsum-on version=3.53.0, backend=lmdb, rowsum=on

Requirements header:

requires role NAME { key=val ... }
requires db (shared|each) (file|:memory:)

If no built binary has the capabilities a role requires, the test runner skips the whole file. shared file gives all roles one database; each gives each role its own. Default is each :memory:.

Cross-binary directives:

Directive Purpose
do_execsql_test NAME [ROLE] { SQL } { expected } Run on ROLE (default if omitted)
do_compare_test NAME { roles } [differ] { SQL } All roles equal, or with differ each differs from first
do_golden_test NAME GOLDEN { others } { SQL } Others match GOLDEN
do_corrupt_test NAME { marker } { read_sql } Flip byte after marker on disk; pass if read_sql reports corruption

Querying results

tclsh tool/test-sql-filter.tcl           # summary
tclsh tool/test-sql-filter.tcl -failures # all failures
tclsh tool/test-sql-filter.tcl -diff 1 2 # changes between runs
tclsh tool/test-sql-filter.tcl -help     # full options

Modes: -runs, -summary, -failures, -skips, -categories, -diff, -compare, -test, -export. Filters: -target, -file, -category, -format.

Relationship to benchmarking

make benchmark make test-sql
Tests Performance Correctness
Input not-fork.d/*/benchmark/*.test test/sql/*.test
Output benchmarks.sqlite test-sql.sqlite
Query tool tool/benchmark-filter.tcl tool/test-sql-filter.tcl
Cross-binary No Yes
Corruption testing No Yes

Both use append-only SQLite databases with queryable history, comparable across runs.