Login
Running libfossil and Fossil on LumoSQL (LMDB backend)
Login

These are instructions to build libfossil and Fossil against a LumoSQL sqlite3 amalgamation. A Fossil repository is an LMDB database consisting of a directory holding data.mdb and lock.mdb. The SQL, Fossil protocol, and artefact levels should be the same as an unmodified SQLite-backed repository.

Prerequisites

Generate a LumoSQL amalgamation for the target you will build against, with KEEP_SOURCES=1. See LUMOSQL-AMALGAMATION.md.

make build TARGETS="3.53.1+lmdb-0.9.35" KEEP_SOURCES=1 ALWAYS_REBUILD=1

The recipes below reference three directories from that build:

SRC=/path/to/lumosql/build/<target>/sources/sqlite3   # amalgamation + lumo helpers
LM=/path/to/lumosql/build/<target>/sources/lmdb       # LMDB .c/.o (sources/lmdbv1 for an lmdbv1 target)
LB=/path/to/lumosql/build/<target>/lumo/build         # LMDB headers + backend shim under $LB/.lumosql/backend/

The LMDB headers (lmdb.h, midl.h) and the backend shim live under $LB. Copying them from $LM is the most frequent build failure.

For LMDB the defines are -DLUMO_LMDB_FIXED_ROWID -DLUMO_BACKEND_PRAGMA -DSQLITE_ENABLE_RTREE -DSQLITE_THREADSAFE=1. For rowsum or lmdbv1-encryption variants take the defines and link items from LUMOSQL-AMALGAMATION.md variant matrix. Use rowsum-off; Fossil's model is already a Merkle DAG of checksums.

Why code changes are needed

Fossil rejects a database that is not a single SQLite file, while libfossil accepts the directory form via the --lumosql configure flag. Fossil needs the patch fossil-db-looks-like-a-repository.diff (see "Apply the Fossil patch").

libfossil

Prepare the source tree

cd libfossil
cp $SRC/sqlite3.c $SRC/sqlite3.h extsrc/
cp $SRC/src/lumo-vdbeInt.h extsrc/
cp $SRC/src/lumo-vdbeAdd.c  extsrc/
cp $SRC/src/lumo-func.c     extsrc/
cp $SRC/src/lumo-sha3.c     extsrc/
cp $SRC/src/lumo-blake3*.c  extsrc/
cp $SRC/src/blake3.h        extsrc/
cp $SRC/src/blake3_impl.h   extsrc/
cp -r $LB/.lumosql/backend  extsrc/backend
cp $LB/lmdb.h $LB/midl.h    extsrc/

$LB/.lumosql/backend holds the expanded backend shim the amalgamation #includes. ($SRC/.lumosql/backend holds redirect stubs that resolve only inside the LumoSQL build tree.)

Build the LMDB objects with -fPIC (libfossil is a shared object). -I$LB lets mdb.c find lmdb.h:

cc -fPIC -O2 -I$LB -c $LM/mdb.c  -o extsrc/mdb_pic.o
cc -fPIC -O2 -I$LB -c $LM/midl.c -o extsrc/midl_pic.o

Configure and build

./configure \
  --lumosql \
  --with-sqlite=extsrc \
  CPPFLAGS="-DLUMO_LMDB_FIXED_ROWID -DLUMO_BACKEND_PRAGMA -DSQLITE_ENABLE_RTREE" \
  LDFLAGS="$PWD/extsrc/mdb_pic.o $PWD/extsrc/midl_pic.o"
make

make produces libfossil.so and the f-apps, under f-apps/ or the checkout root depending on version.

Some libfossil versions build with a rule that uses $(CFLAGS.core) and ignore the configure-time CPPFLAGS, giving lmdb.h: No such file or directory. Pass the flags at make time instead:

make CFLAGS.core='-Wall -Werror -fPIE -fPIC -O2 -Iextsrc \
  -DLUMO_LMDB_FIXED_ROWID -DLUMO_BACKEND_PRAGMA \
  -DSQLITE_ENABLE_RTREE -DSQLITE_THREADSAFE=1 \
  -Wno-unused-function -Wno-maybe-uninitialized'

Smoke test

export LD_LIBRARY_PATH=$PWD USER=$(id -un)
export HOME=$(mktemp -d) XDG_CONFIG_HOME="$HOME/.config"; mkdir -p "$XDG_CONFIG_HOME"
APPS=./f-apps        # or APPS=. if the f-* binaries are in the checkout root

$APPS/f-new   /tmp/t.f -m init
$APPS/f-open  /tmp/t.f --force
echo hello > a.txt
$APPS/f-add   a.txt
$APPS/f-ci    a.txt -m "first" --user "$USER"
$APPS/f-timeline
echo "Home page body." > body.txt
$APPS/f-wiki  save HomePage body.txt --new --user "$USER"
$APPS/f-annotate a.txt

All commands exit 0. f-new creates an LMDB directory repository. f-timeline lists the check-in.

Fossil

Build Fossil after the libfossil smoke test passes. Fossil uses a one-line src/db.c patch and two makefile edits.

Prepare the source tree

cd fossil
cp $SRC/sqlite3.c $SRC/sqlite3.h extsrc/
cp $SRC/shell.c             extsrc/
cp $SRC/src/lumo-vdbeInt.h  extsrc/
cp $SRC/src/lumo-vdbeAdd.c  extsrc/
cp $SRC/src/lumo-func.c     extsrc/
cp $SRC/src/lumo-sha3.c     extsrc/
cp $SRC/src/lumo-blake3*.c  extsrc/
cp $SRC/src/blake3.h        extsrc/
cp $SRC/src/blake3_impl.h   extsrc/
cp -r $LB/.lumosql/backend  extsrc/backend
cp $LM/mdb.o $LM/midl.o     extsrc/
cp $LB/lmdb.h $LB/midl.h    extsrc/

shell.c replaces Fossil's extsrc/shell.c, which references shell symbols absent from the LumoSQL amalgamation. The LMDB objects are static here; Fossil links a static binary.

Apply the Fossil patch

patch -p1 < /path/to/lumosql/fossil/fossil-db-looks-like-a-repository.diff

The patch touches db_looks_like_a_repository() in src/db.c. If patch refuses the diff, make the edit by hand so the function reads:

  sz = file_size(zDbName, ExtFILE);
  if( file_isdir(zDbName, ExtFILE)==1 ){
    /* LumoSQL/LMDB stores a "database" as a directory (data.mdb+lock.mdb).
    ** Size/512-alignment heuristics do not apply; rely on the schema probe
    ** below. */
    sz = 16834;
  }
  if( sz<16834 ) return 0;
  db = db_open(zDbName);
  if( !db ) return 0;
  if( !g.zVfsName && (sz%512) && !file_isdir(zDbName, ExtFILE) ) return 0;

Configure and add LumoSQL flags

./configure --with-openssl=none

Edit src/main.mk. Append to the SQLITE_OPTIONS block, ending with -DSQLITE_THREADSAFE=1:

SQLITE_OPTIONS = ... \
                 -Iextsrc \
                 -Wno-unused-function -Wno-declaration-after-statement \
                 -Wno-maybe-uninitialized \
                 -DLUMO_LMDB_FIXED_ROWID -DLUMO_BACKEND_PRAGMA \
                 -DSQLITE_ENABLE_RTREE -DSQLITE_THREADSAFE=1

-DSQLITE_THREADSAFE=1 is required. With unmodified SQLite, Fossil's SQLITE_OPTIONS sets -DSQLITE_THREADSAFE=0, but the LumoSQL shim needs THREADSAFE=0 to compile. This appended -D overrides Fossil's.

Edit the generated Makefile and append the LMDB objects to the LIB line:

LIB += extsrc/mdb.o extsrc/midl.o

Build

make

This produces a fossil binary in the checkout root.

Smoke test

export USER=$(id -un)
export HOME=$(mktemp -d) XDG_CONFIG_HOME="$HOME/.config"; mkdir -p "$XDG_CONFIG_HOME"
./fossil new   /tmp/r.fossil
./fossil open  /tmp/r.fossil
echo hello > a.txt
./fossil add   a.txt
./fossil commit -m "first"
./fossil timeline
./fossil test-integrity /tmp/r.fossil

All commands should exit 0. fossil new creates an LMDB directory repository. On a rowsum-off repository, test-integrity reports SQLITE_CORRUPT for PRAGMA integrity_check, which has nothing to check against so it returns low-level database integrity-check: ok.

Build errors

error cause fix
undefined reference to lumo_backend_pragma (link) backend/ copied from $SRC/.lumosql/ (redirect stubs) cp -r $LB/.lumosql/backend extsrc/backend
lmdb.h: No such file or directory (compile) headers copied from $LM, or flags did not reach the sqlite3.c rule copy headers from $LB; for libfossil pass flags via make CFLAGS.core=... (libfossil Configure and build)
undefined reference to sqlite3_incomplete (link, Fossil) Fossil's stock extsrc/shell.c in place copy LumoSQL's shell.c (Fossil Prepare the source tree)
expected identifier before '{' in lumo_btree.c (compile) SQLITE_THREADSAFE=0 append -DSQLITE_THREADSAFE=1 (Fossil Configure)
'p' may be used uninitialized made fatal (compile) -Werror with GCC 13+ add -Wno-maybe-uninitialized