Building a LumoSQL Amalgamation

A LumoSQL amalgamation is a sqlite3.c + sqlite3.h pair, generated from a not-fork-patched SQLite tree, that compiles into a LumoSQL build. It may have rowsum extensions, the LMDB backend, or both.

The sqlite3.c file #includes a fixed set of helper sources which must be on the include path. The LMDB backend also links two LMDB object files.

Files

make (see "Generate the amalgamation") produces these under build/<target>/sources/sqlite3/:

file	role
`sqlite3.c`, `sqlite3.h`	the amalgamation and its header
`shell.c`	the LumoSQL-aware shell source (optional)
`src/lumo-vdbeInt.h`, `src/lumo-vdbeAdd.c`, `src/lumo-func.c`, `src/lumo-sha3.c`, `src/lumo-blake3*.c`	helpers `#include`d from `sqlite3.c`
`src/blake3.h`, `src/blake3_impl.h`	helper headers

LMDB variants also need:

file	location	role
`mdb.o`, `midl.o`	`build/<target>/sources/lmdb/`	link into a static binary
`mdb.c`, `midl.c`	`build/<target>/sources/lmdb/`	rebuild with `-fPIC` for a shared object
`lmdb.h`, `midl.h`	`build/<target>/lumo/build/`	LMDB headers
`backend/`	`build/<target>/lumo/build/.lumosql/`	the backend shim directory

For an lmdbv1 target, the .c/.o directory is sources/lmdbv1/; the headers and backend/ stay under lumo/build/.

The backend/ directory holds lumo_btree.c, lumo_pager.c, lumo_backup.c, lumo_btmutex.c, lumo_wal.c, and lumo_btreeInt.h (some empty for the LMDB 0.9.x backend). Take these from lumo/build/.lumosql/backend/. The same names appear under sources/sqlite3/.lumosql/backend/ as one-line redirect stubs that resolve only inside the LumoSQL build tree.

sqlite3.c keeps #include "lumo-*.c/.h" and #include "backend/lumo_*.c" directives verbatim, and each backend/lumo_*.c #includes lmdb.h. The helpers, the backend/ directory, and lmdb.h must therefore be on the include path when sqlite3.c is compiled ("Use the amalgamation" arranges this).

Variant matrix

variant	extra defines	extra link
stock-btree, rowsum-off	(none)	-
stock-btree, rowsum-on	`-DLUMO_EXTENSIONS=1 -DLUMO_ROWSUM=on`	-
lmdb, rowsum-off	`-DLUMO_LMDB_FIXED_ROWID -DLUMO_BACKEND_PRAGMA`	`mdb.o midl.o -lpthread`
lmdb, rowsum-on	`-DLUMO_EXTENSIONS=1 -DLUMO_ROWSUM=on -DLUMO_LMDB_FIXED_ROWID -DLUMO_BACKEND_PRAGMA`	`mdb.o midl.o -lpthread`
lmdbv1, rowsum-off	`-DLUMO_LMDB_FIXED_ROWID -DLUMO_BACKEND_PRAGMA`	`mdb.o midl.o -lpthread`
lmdbv1, encrypt-on	`-DLUMO_LMDBV1_ENCRYPT -DLUMO_BACKEND_PRAGMA`	`mdb.o midl.o -lpthread -lsodium`

Every variant also needs -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_FTS5 -DSQLITE_THREADSAFE=1, the SQLite feature flags the LumoSQL test suite assumes.

The lmdbv1 rows use the LMDB 1.0 backend (not-fork.d/lmdbv1/). The encrypt-on row links libsodium and uses two lumo-crypto.* files that "Use the amalgamation" carries automatically. Add -DLUMO_EXTENSIONS=1 -DLUMO_ROWSUM=on to either lmdbv1 row for rowsum-on.

Generate the amalgamation

From the top of the LumoSQL checkout:

make build TARGETS="3.53.1+lmdb-0.9.35" KEEP_SOURCES=1 ALWAYS_REBUILD=1

Other targets:

make build TARGETS="3.53.1++rowsum-on" KEEP_SOURCES=1
make build TARGETS="3.53.1+lmdb-0.9.35+rowsum-on" KEEP_SOURCES=1
make build TARGETS="3.53.1+lmdbv1-1.0" KEEP_SOURCES=1
make build TARGETS="3.53.1+lmdbv1-1.0+lmdbv1_encrypt-on" KEEP_SOURCES=1   # needs libsodium-dev

KEEP_SOURCES=1 is required; without it the patched tree and sqlite3.c are deleted at the end of the build. <target> in the paths below is the literal target string, e.g. 3.53.1+lmdb-0.9.35+rowsum-on.

Sanity-check the result:

cd build/<target>/sources/sqlite3
grep -c '#include "lumo-vdbeInt.h"'   sqlite3.c   # >= 1
grep -c '#include "lumo-vdbeAdd.c"'   sqlite3.c   # 1
grep -c 'lumoExtensionAdd\|lumoRowsumFunc' sqlite3.c  # > 0 if rowsum-on
grep -c '#include "backend/lumo_'     sqlite3.c   # > 0 for LMDB variants, 0 for stock-btree

Use the amalgamation

Pick a directory $DST in your project where sqlite3.c and sqlite3.h will live. Copy the files there:

SRC=/path/to/lumosql/build/<target>/sources/sqlite3
LM=/path/to/lumosql/build/<target>/sources/lmdb    # sources/lmdbv1 for an lmdbv1 target
LB=/path/to/lumosql/build/<target>/lumo/build
DST=/path/to/your/project/sqlite-dir

cp $SRC/sqlite3.c $SRC/sqlite3.h $DST/
cp $SRC/src/lumo-vdbeInt.h $SRC/src/lumo-vdbeAdd.c $SRC/src/lumo-func.c $SRC/src/lumo-sha3.c $DST/
cp $SRC/src/lumo-blake3*.c $SRC/src/blake3.h $SRC/src/blake3_impl.h $DST/

# LMDB variants only:
cp -r $LB/.lumosql/backend $DST/backend   # the backend shim (see "Files")
cp $LB/lmdb.h $LB/midl.h    $DST/          # headers, from lumo/build/
cp $LM/mdb.o  $LM/midl.o    $DST/          # objects, for a static binary
# For a shared library, rebuild PIC objects (-I$DST so mdb.c finds lmdb.h):
# cc -fPIC -O2 -I$DST -c $LM/mdb.c  -o $DST/mdb_pic.o
# cc -fPIC -O2 -I$DST -c $LM/midl.c -o $DST/midl_pic.o

Compile sqlite3.c with -I$DST as the only LumoSQL include path. From there, #include "backend/lumo_btree.c" resolves to $DST/backend/lumo_btree.c and its #include "lmdb.h" resolves to $DST/lmdb.h.

Example: a standalone shell, lmdb + rowsum-on. Use the defines from the variant matrix.

cd $DST
cc -O2 -I. \
   -DLUMO_EXTENSIONS=1 -DLUMO_ROWSUM=on \
   -DLUMO_LMDB_FIXED_ROWID -DLUMO_BACKEND_PRAGMA \
   -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_FTS5 -DSQLITE_THREADSAFE=1 \
   sqlite3.c shell.c mdb.o midl.o \
   -o lumo_sqlite3 -lpthread -lm -ldl

Drop mdb.o midl.o and the LMDB defines for stock-btree. Drop -DLUMO_EXTENSIONS=1 -DLUMO_ROWSUM=on for rowsum-off.

For encrypted lmdbv1, swap -DLUMO_LMDB_FIXED_ROWID for -DLUMO_LMDBV1_ENCRYPT and add -lsodium:

cd $DST
cc -O2 -I. \
   -DLUMO_LMDBV1_ENCRYPT -DLUMO_BACKEND_PRAGMA \
   -DSQLITE_ENABLE_RTREE -DSQLITE_ENABLE_FTS5 -DSQLITE_THREADSAFE=1 \
   sqlite3.c shell.c mdb.o midl.o \
   -o lumo_sqlite3 -lsodium -lpthread -lm -ldl

An encrypted database opens with a key in the connection URI, e.g. lumo_sqlite3 'file:/path/to/db?lumo_key=your passphrase'. The first open writes lumo.salt (mode 0600) in the database directory; later opens reuse it. The wrong key or no key fails with SQLITE_AUTH. A URI key is visible on the process command line (argv, shell history, /proc); use it for evaluation only.

Verify

A LumoSQL binary passes the LumoSQL SQL test suite. From the LumoSQL checkout:

make test-sql TARGETS="<target>"
tclsh tool/test-sql-filter.tcl -db test-sql.sqlite

A standalone lumo_sqlite3 from "Use the amalgamation" tests the amalgamation alone. If it passes and your application build fails, the difference is in your build.