Knowledge Relevant to LumoSQL
A terse list of published source code relevant to LumoSQL developers. Standalone document, referred to throughout the LumoSQL documentation.
Everything listed here is open source (or SQLite, which is its own special source status.) There are many closed-source products that extend and reuse SQLite, none of which can be considered for LumoSQL since we do not have the source.
List of SQLite Code-related Knowledge
SQLite code has been incorporated into many other projects. There are also many relevant key-value stores and libraries.
| Project | Last modified | Description |
|---|---|---|
| sqlightning | 2013 | SQLite ported to the LMDB key-value store; all in C. Abandoned, but inspired LumoSQL |
| Original MDB Paper | 2012 | Paper by Howard Chu describing the motivations, design and constraints of the LMDB key-value store |
| SQLHeavy | 2016 | sqlightning updated, and ported to LevelDB, LMDB, RocksDB and more, with a key-value store library abstraction; all in C. Abandoned |
| Oracle BDB port of SQLite | 2019 | The original ubiquitous Unix K-V store, disused in open source since Oracle's 2013 license change to AGPL; the API template for most of the k-v btree stores around. Now includes many additional features including full MVCC transactions, networking and replication. The SQLite port was discontinued after version 18.1.32. Dead |
| libSQL | current | Open-source, open-contribution fork of SQLite by Turso. MIT licensed, 5k+ stars. Adds page-level encryption, vector search, embedded replicas, and remote access while maintaining SQLite file-format compatibility when the new features are not used. The active counterpoint to LumoSQL's "modification, not a fork" framing |
| Turso Database | current | A from-scratch SQLite-compatible rewrite in Rust by the same team. Adds BEGIN CONCURRENT MVCC for concurrent writes, change data capture, and experimental encryption-at-rest. Currently in beta. Out of LumoSQL's stated C/Go scope, but where the SQLite ecosystem is heading |
| rqlite | current | Distributed database with networking and Raft consensus on top of SQLite nodes |
| Bedrock | current | WAN-replicated blockchain multimaster database built on SQLite. Has MySQL emulation |
| Comdb2 | current | Bloomberg's clustered HA RDBMS, built on a SQLite VDBE-layer port over a forked Berkeley DB 4.2.52. The bdb/ adapter is heavily entangled with the distributed-DB layer, but the underlying berkdb/ engine retains the original Sleepycat license (pre-Oracle, pre-AGPL) and is conceptually extractable. Comdb2's SQLite integration sits at the VDBE/row level rather than LumoSQL's btree.c interception point, so it cannot be ported directly |
| Litestream | current | Streaming WAL replication for SQLite to S3-compatible object storage. Go, Apache 2.0. v0.5 introduces the LTX file format (transaction-aware page sets) and an optional read-only VFS that serves queries directly from replica storage. More SQLite-specific than WAL-G |
| LiteFS | current | Distributed FUSE filesystem for live SQLite replication with Consul-based failover. Go, Apache 2.0. Proxies SQLite via a fake filesystem to be transaction-aware without modifying the application |
| cr-sqlite | 2024 | Run-time loadable extension adding multi-master replication and CRDTs to SQLite and libSQL. Rust core. Demonstrates a CRDT-based model for distributed SQLite distinct from the consensus-based approach of rqlite/Bedrock/Comdb2. Active development paused since early 2024 but the codebase remains usable as documentation of the CRDT-on-SQLite pattern |
| sql.js | current | SQLite compiled to JavaScript WebAssembly through Emscripten |
| WAL-G | current | Archival/PITR tool for PostgreSQL, MySQL/MariaDB, MS SQL Server, and Greenplum (production); MongoDB, Redis, and etcd (beta). Illustrates how a WAL is useful as a list of transactions for archival |
| sqlite3odbc | current | ODBC driver for SQLite by Christian Werner as used by many projects including LibreOffice |
| Spatialite | current | Geospatial GIS extension to SQLite, similar to PostGIS |
| Gigimushroom's Database Backend Engine | 2019 | Alternative BTree storage engine implemented using SQLite's Virtual Table Interface. Demonstrates that virtual tables can work as storage backends and that such engines could be ported to LumoSQL backends |
List of MVCC-capable KV store-related Knowledge
There are hundreds of active K-V stores, but few do MVCC. MVCC is a requirement for any performant backend to SQLite, because SQLite translates SQL transactions into KV store transactions and assumes the KV store will handle contention. Section 8 Related Work in Microsoft's SIGMOD paper on the FASTER KV store lists many non-MVCC stores (Redis, RocksDB, Aerospike, etc). We have not considered them for LumoSQL backends.
| Project | Last modified | Description |
|---|---|---|
| SQLite's built-in KV store | current | Included as a reminder that SQLite is also the world's most widely-used key-value store, and it does full MVCC. LumoSQL plans to expose this via an API for the first time |
| LMDB | current | Maintained by the OpenLDAP project, LMDB is a standalone KV store written in C, and is the only KV store that is implemented using mmap. LMDB was the first backend target for LumoSQL |
| Oracle BDB KV Store | current | Version 18.1.40 and later of BDB do not include the SQLite port, but they are still the same MVCC KV store |
| Bloomberg berkdb fork | current | A fork of Berkeley DB 4.2.52 maintained inside the Comdb2 source tree, with 20 years of Bloomberg modifications including page-level locking and snapshot isolation. Retains the original 2003 Sleepycat license — predates Oracle's AGPL relicensing, sidestepping the licensing issue that killed the official Oracle BDB+SQLite port. Not packaged as a standalone library; extraction would be required |
| libmdbx | current | Heavily modified fork of LMDB, written in C, used in Ethereum and hundreds of other projects. Ships as an amalgamated single source file like SQLite, no additional dependencies. |
| WiredTiger | current | MongoDB's default storage engine since 3.2. Written in C, co-founded in 2012 by Keith Bostic and Michael Cahill (Bostic was also a co-founder of Sleepycat, which commercialized Berkeley DB). Supports both B-tree and LSM-tree modes, MVCC, schemas, compression, checkpoints. More elaborate than LMDB but the B-tree mode is well tuned and heavily deployed |
List of On-disk SQLite Format-related Knowledge
The on-disk file format matters to many SQLite use cases, and introspection tools are rare. There are advantages to having investigative tools that do not use the canonical source code to read and write these databases. The SQLite file format is promoted as a stable, backwards-compatible transport (recommended by the Library of Congress as an archive format) but it also has significant drawbacks as discussed elsewhere in the LumoSQL documentation.
| Project | Last modified | Description |
|---|---|---|
| A standardized corpus for SQLite database forensics | current | Sample SQLite databases and evaluations of 5 tools that do extraction and recovery from SQLite, including Undark and SQLite Deleted Records Parser |
| FQLite | current | Open-source SQLite forensic toolkit (Java) that finds and recovers deleted records from the main database, freelist, and WAL. Backed by a peer-reviewed paper in IJCFATI (2021). The maintained successor to Undark and SQLite Deleted Records Parser |
| SQBrite | 2022 | Independent Python 3 reimplementation of SQLite deleted-record recovery, with a from-scratch parser of the SQLite on-disk file format. Useful diversity alongside FQLite |
| bring2lite | 2019 | FSI:DI 2019 academic paper presenting a structural concept and tool for forensic recovery of deleted SQLite records, complementary to FQLite/SQBrite |
(The forensics industry has many SQLite diagnostic tools, mostly closed source. Commonly cited ones we have not tried: Belkasoft Evidence Center, BlackBag BlackLight, Cellebrite UFED Physical Analyser, DB Browser for SQLite, Magnet AXIOM and Oxygen Forensic Detective.)
List of Relevant SQL Checksumming-related Knowledge
Most databases and some filesystems already do page-level or block-level checksumming, but none extend this to row-level content verification that survives export and reimport. LumoSQL's checksumming goals are informed by this prior art.
| Project | Last modified | Description |
|---|---|---|
| eXtended Keccak Code Package | current | Code from https://keccak.team for very fast peer-reviewed hashing. Potential hash function for LumoSQL row checksums |
| BLAKE3 | current | Fast cryptographic hash function with SIMD/multi-threaded parallelism, dual public-domain (CC0) and Apache 2.0 license. Reference C implementation. Outperforms SHA-2 family on modern hardware. Quantum-resistant for collision (Grover halves to ~128-bit security). Alternative or complement to Keccak for LumoSQL row checksums |
| PostgreSQL Data Checksums | current | Per-page CRC checksums on all data files, enabled at initdb. Default-on since PostgreSQL 18. Detects storage-level corruption but not logical corruption |
| InnoDB Page Checksums | current | MySQL/MariaDB InnoDB uses configurable per-page checksums (CRC32, innodb, none). The innochecksum tool verifies offline. Detects torn pages and storage corruption |
| ZFS/Btrfs Block Checksums | current | Copy-on-write filesystems with per-block checksumming and self-healing from redundant copies. Operates below the database layer — protects against storage corruption but is invisible to SQL |
| pgaudit | current | PostgreSQL Audit Extension, the standard for structured compliance-grade audit logging in PostgreSQL. Captures object-level and session-level statement records. Solves an overlapping integrity-evidence problem from the audit-log direction rather than the cryptographic-hash direction |
| SQL code for Public Key Row Tracking | 2018 | Percona's SQL row integrity solution for Postgresql using public key crypto |
| Richard Hipp's SQLite Checksum VFS | 2020 | This shows the potential benefits from maintaining checksums. There are many limitations, but its more than any other mainstream database ships by default |
List of Relevant Benchmarking and Test Knowledge
SQLite and other top databases are not benchmarked in a realistic or consistent way. SQL server benchmarking using tools like TPC is an obsessive industry, and myriad testing tools ship with SQLite, PostgreSQL, MariaDB etc. But in practical terms there is no way of comparing the most-used databases with each other, or of being sure that existing tests are realistic, or of simply reproducing results that other people have found. LumoSQL covers so many codebases and use cases that better SQL benchmarking is a project requirement.
The well-described testing of SQLite involves some open code, some closed code, and many ad hoc processes. The SQLite team clearly have an internal culture of testing that has benefited the world. But reproducible testing is not reproducible benchmarking, and neither addresses whether benchmarks approximate actual use cases. The TCL testing harness shipped with SQLite code contains specific dependencies on the behaviour of the SQLite btree backend. LumoSQL with the original btree backend aims to always pass these tests, but differences in locking behaviour, assumed key lengths, and even the number of database files a backend uses mean the SQLite TCL test suite is not generally useful.
There are virtually no test harnesses that cover encrypted databases or encrypted database connections, despite encryption being frequently required and crypto implementation decisions making a large difference in performance. Privacy is a valid dimension for database testing and a fundamental goal for LumoSQL.
A 2024 systematic literature review of DBMS performance comparisons (Taipalus, J. Syst. Softw. 208:111872) found that "performance is usually tested in a way that does not reflect real-world use cases, and tests are typically reported in insufficient detail for replication or for drawing conclusions from the stated results." This is the same conclusion the 2017 Purohith paper reached for SQLite specifically, now confirmed across the broader DBMS literature seven years later.
SQL testing is difficult for a separate reason. As the Regression Testing paper below says: "A problem with testing SQL in DBMSs lies in the fact that the state of the database must be considered when deducing testing outcomes". SQL statements frequently change server state during execution, in different ways on different servers, affecting the behaviour of subsequent statements.
| Project | Last modified | Description | |
|---|---|---|---|
| Dangers and complexity of sqlite3 benchmarking | n/a | Helpful 2017 paper: "...changing just one parameter in SQLite can change the performance by 11.8X... up to 28X difference in performance" | |
| DBMS Performance Comparisons: Systematic Literature Review | n/a | Taipalus 2024 (J. Syst. Softw. 208:111872), open-access. Covers DBMS benchmarking practice across the literature: "performance is usually tested in a way that does not reflect real-world use cases, and tests are typically reported in insufficient detail for replication." The same conclusion the 2017 Purohith paper reached for SQLite, now confirmed across the broader DBMS literature seven years later | |
| SQLite speedtest1 | current | The canonical performance benchmark shipped in the SQLite source tree itself. The basis of LumoSQL's own benchmarking tool, and of the Phoronix Test Suite's pts/sqlite-speedtest profile | |
| Phoronix Test Suite SQLite profiles | current | GPLv3 cross-platform benchmarking framework with maintained pts/sqlite and pts/sqlite-speedtest profiles, last updated October 2025 against SQLite 3.50.4. OpenBenchmarking.org hosts hundreds of public, comparable result sets across hardware. Complementary to LumoSQL benchmarking: PTS does cross-machine comparison of a single SQLite configuration; LumoSQL does single-machine comparison of a (version x backend x options) matrix. A LumoSQL-->PTS bridge would give LumoSQL the cross-hardware data it currently lacks | |
| sqllogictest-rs | current | Rust implementation of SQLite's sqllogictest dialect for cross-engine SQL correctness testing. The de-facto successor to sqlite.org's original Python/C sqllogictest, which is no longer actively developed. Adopted by Apache DataFusion, Databend, and RisingLightDB. Available as library (sqllogictest crate) and CLI (sqllogictest-bin). The dialect specification at sqlite.org remains the authoritative reference | |
| TCL SQLite tests | current | These are a mixture of code coverage tests, unit tests and test coverage. Actively maintained. | |
| BenchBase | current | Multi-DBMS SQL benchmarking framework via JDBC, maintained by CMU's database group. Successor to OLTP-Bench (Difallah et al., VLDB 2014). Includes implementations of TPC-C, TPC-H, TATP, YCSB, Wikipedia, Twitter, Epinions, AuctionMark, SmallBank, CH-Benchmark, and many more workloads. The standard academic benchmark testbed | |
| go-ycsb | current | Go port of the Yahoo! Cloud Serving Benchmark, actively maintained by PingCAP. The original Java YCSB has been largely unmaintained since 2019; go-ycsb is the maintained alternative. Supports SQLite, BoltDB, RocksDB, Badger, FoundationDB, plus the original cloud DB targets (Cassandra, MongoDB, Redis, etc.) | |
| Sysbench | mostly dormant | A multithreaded generic benchmarking tool, with one well-supported use case being networked SQL servers, and MySQL in particular. Upstream has been largely dormant since 2020 but it remains the standard cited tool in MySQL/PostgreSQL benchmarking literature and tutorials | |
| Regression Testing of SQL | n/a | 2014 paper "a framework for minimizing the required developer effort for managing and running SQL regression tests" | |
| Enhancing the Performance of SQLite | n/a | 2013 paper that does profiling and develops performance testing metrics for SQLite | |
| SQLCipher Performance Optimisation | n/a | 2014 comments on the additional performance metric problems that come with SQLite encryption | |
| Performance analysis ... on flash file systems | 2013 | Discussion of SQLite and 2 others on Flash, examining the cost to flash of SQL operations |
List of Just a Few SQLite Encryption Projects
Encryption is a major problem for SQLite users looking for open code. There are no official open source implementations, although the APIs are documented and most solutions use the SQLite extension interface. The result is many mutually-incompatible implementations. None of the open source ones have received FIPS 140 certification, and none publish test results for compatibility with SQLite upstream or with the file format. Besides the closed source solution from sqlite.org there are at least three other closed source options not listed here. The choice between closed source and fragmented open source is a poor security outcome.
No SQLite encryption project currently addresses post-quantum cryptography. NIST finalised its first three post-quantum standards in August 2024 — FIPS 203 (ML-KEM), FIPS 204 (ML-DSA), and FIPS 205 (SLH-DSA) — and intends to deprecate quantum-vulnerable algorithms by 2035. Because SQLite is recommended by the Library of Congress as an archive format, SQLite databases are a canonical "harvest now, decrypt later" target: anything encrypted today with RSA or elliptic-curve key wrapping is exposed once a cryptographically-relevant quantum computer exists. Symmetric primitives (AES, ChaCha20) and standard hash functions (SHA-2, SHA-3, BLAKE3) are considered quantum-resistant in practice, so the exposure is specifically in key encapsulation and signatures.
| Project | Last modified | Description | |
|---|---|---|---|
| SQLite Encryption Extension(SEE) | current | Info about the proprietary, closed source official SQLite crypto solution, illustrating that there is little to be compatible with in the wider SQLite landscape. This is a standalone product. The API is published and used by some open source code. | |
| SQLCipher | current | Adds at-rest encryption to SQLite at the pager level, using OpenSSL (the default) or optionally other providers. Uses an open core licensing model, and the less-capable open source version is BSD licensed with a requirement that users publish copyright notices. Uses the SEE API. | |
| Oracle BDB Encryption | 2018 | Exposes the (old and insecure) BDB encryption facilities via the SEE API with one minor change. | |
| sqleet | 2020 | Implements ChaCha20-Poly1305 encryption at the pager level. Public Domain (UNLICENSE). The project itself has been unmaintained since v0.31.1 in January 2020, but its codebase has been folded into SQLite3 Multiple Ciphers (below) as one of the supported cipher schemes | |
| wxSqlite3 | current | wxWidgets C++ wrapper, that also implements SEE-equivalent crypto. Licensed under the LGPL | |
| SQLite3 Multiple Ciphers | current | Successor to wxSQLite3's crypto code, decoupled from wxWidgets. Supports SQLCipher-compatible mode, ChaCha20, AES-256, and the sqleet cipher. The most actively maintained open encryption option for SQLite as of 2026 | |
| LMDB v1.0 with encryption | current | The LMDB branch mdb.master3 is LMDBv1.0, with page-level encryption and other features, shipped as a LumoSQL backend | |
| libSQL native encryption | current | libSQL adds optional page-level encryption that produces standard SQLite files when not used. Cross-reference: see also the SQLite Code section above | |
| liboqs (Open Quantum Safe) | current | MIT-licensed C library implementing NIST-standardised post-quantum algorithms — ML-KEM (FIPS 203) and ML-DSA (FIPS 204) — plus experimental candidates. Backed by AWS, Cisco, IBM Research, Microsoft Research, Linux Foundation. No post-quantum-aware SQLite encryption project exists yet; this is the obvious building block for one | |
| sqlite-zstd | current | Transparent dictionary-based row-level compression for SQLite, in Rust. Dictionary-based compression interacts with encryption design (compression-then-encryption is the safe order) and the encryption ecosystem will eventually need to address compressed payloads | |
| Discussion of SQLCipher vs sqleet | 2019 | Authors of sqleet and wxSQLite3 discuss SQLCipher, covering many weaknesses and some strengths |
... there are many more crypto projects for SQLite.