The LumoSQL benchmarking suite has detailed documentation, and separately, a extensive statistical analysis tool which examines the data held in the benchmarking database. The system is designed for consolidating results from many people, and captures many details that give context. It seems quite odd that there is no agreed SQL/SQLite benchmarking system that captures real-world usage, configuration settings and does like-for-like comparisons between databases. Perhaps this system will help promote the idea that measurements matter. of these improvements are inherited by LumoSQL - and besides, its interesting.
The difficulty with having so much data is asking the right question. At this early stage of LumoSQL, we only ask two questions:
- Given one specific recent version of SQLite, what happens to performance as we replace the backend, turn encryption and checksums on and off, change runtime settings, and do all of the above with steadily-increasing data sizes and loadings? And,
- Given many different versions of SQLite and backends released over the last decade, what happens when we test their cross-product with many data sizes and a selection of features including encryption?
We also ask one non-LumoSQL question:
- How have SQLite vs SQLite benchmarks changed over the last few years? LumoSQL benefits from the SQLite performance work, and besides, its interesting.
There are, briefly, other observations we have from other aspects of this data, for example, look what happened to indexed SELECT statements from SQLite
version 3.30. Version 3.31 and 3.42 are slower than 3.30, but from 3.43 there was steady improvement until 3.53.2, which is 52% faster than 3.30. This sort of
effect means context is essential when saying "LMDB-backed SQLite has become X% faster or slower", because it may be a meaningless statement. Similarly we
observe that LMDB v0.9x results are almost unchanged over the course of a decade indicating great stability. And LMDB v1.0 is often faster and never slower than
LMDB v0.9x. But we are only focussing on the two questions above, these are all interesting sidequests that people might like to explore in the data, or run
benchmarks themselves.
We do not have a large public dataset yet. We have recorded several tens of thousands of runs in the process of producing LumoSQL v0.82, which was the first fully-functional version able to run behind real-world apps such as Fossil and rpm. None of these runs are kept, because nobody cares how performant a rapidly-changing piece of very early software might have been. In due course once LumoSQL architecture is more or less stable, we expect to make rich benchmarking data available.
It is very significant for our benchmarking problem that LumoSQL does not yet report a detailed version number. This is a non-trivial problem as recorded in the LumoSQL TODO list and seems pretty essential for durable benchmarking data. We would love to hear from SQLite experts on this topic.