Non-technical Introduction

This is the home of LumoSQL, which is a modification of SQLite, the world’s most-used software. LumoSQL adds privacy, security and performance options to SQLite, but stays carefully within the SQLite project’s software guidelines and does not seek to replace SQLite. Several of LumoSQL’s features can be enabled in a standard SQLite database without disturbing traditional SQLite in any way. You can fetch the source code with fossil clone https://lumosql.org/src/lumosql or the git mirror git clone https://codeberg.org/lumosql/lumosql.

All LumoSQL code is licensed under the MIT open source license, which is likely the most-used software license.

LumoSQL is compliant with the mandatory privacy and security requirements of legislation based on Article 7 and Article 8 of the EU Charter of Fundamental Rights. Many countries outside Europe have similar legislation. SQLite cannot offer this, and yet is used at enormous scale for handling personal data within the EU and these other countries.

When LumoSQL becomes a readily-available option just like SQLite, this will have implications for almost all mobile phone users, motor vehicles, web browsers and many other kinds of embedded software. This software will then have a database which:

• Is not stored in plain text. LumoSQL’s at-rest encryption means that personal data is more secure.
• Checks for corruption. LumoSQL notices if the data it reads is not identical to the data it wrote. That might sound obvious, but no mainstream database does this yet!
• May be faster, depending on what your application is doing.
• Gives the user the option to keep data encrypted even if it is transferred to the cloud. Only when the user hands over a password can the data be decrypted, and even then the user might decide to only give the password for a portion of the data.

In the latter half of 2022 we hope there will be a usable general release, accompanied by documentation for end users.

Technical Introduction

LumoSQL provides new features for SQLite:

• Security. LumoSQL’s design provides page-level encryption. There are commercially-licensed code patches that will add this to SQLite, but open source solutions are very limited. Security cannot be assured without open source, so this is an essential feature we have added to SQLite.
• Corruption detection and prevention. LumoSQL offers per-row checksums invisible to the application by default, but optionally visible and available to be operated on. This feature behaves like the existing SQLite ROWID column, implemented via additional hidden columns. No other mainstream database has this feature.
• Fine-grained Security. Using the hidden columns feature, LumoSQL is able to provide per-row encryption, meaning that some rows might be visible to a particular user while other rows are not.
• An encrypted equivalent to JSON called Lumions, usable in any application. Lumions are an early draft standard and offer Attribute Based Encryption, versioning, strong GUID, checksumming and more. This RFC is ambitious but sufficiently limited in scope that it seems possible it could become a universal tool for privacy-compliant cloud-portable data.
• Alternative key-value stores. Every database has a key-value store underneath it, but only LumoSQL has the ability to swap key-value stores with full functionality. The native SQLite Btree can be replaced with LMDB, and with a sufficiently general API that other key-value stores are equally possible. We are very interested in 21st-century K-V stores such as Adaptive Radix Trees and Fractal Trees.

The techniques used to implement LumoSQL contain some important advances:

• LumoSQL does not fork SQLite. SQLite is conservative about breaking compatibility due to its immense userbase, but LumoSQL applies new features to the current codebase or any previous version. This means that the SQLite project can experiment with alternate futures for its architecture and design, while LumoSQL does not have to carry the burden of forking such a successful and intricate codebase.
• LumoSQL provides a user-selectable matrix of code versions. As of today, some 600 combinations of SQLite versions and LMDB versions can be builts, tested and benchmarked. This matrix will grow as more key-value store backends are implemented via the LumoSQL backend API.
• Measurement-based. LumoSQL has an extensive benchmarking toolset which allows anyone to run their own benchmarks, store the results in a standard SQLite file, and then aggregate them. We have some early graphical results for the first ten thousand benchmark runs. We measure across platforms, versions, time and data size among other variables, and we are doing careful statistical modelling.
• Recording prior art. For both the encryption and database design parts of LumoSQL we have gathered and annotated references in BibLaTeX format. Some of these are exported as part of the Lumions RFC.

LumoSQL is supported by NLnet.