The Not-forking Software Reproducibility Tool

Not-forking is for the surprisingly-common situation where you need somebody else's source code, but using it unchanged is not an option and maintaining your own fork sounds like a really terrible idea. What you want to do is say "this is my intention when combining these source trees" but in very precise terms. Not-forking is deterministic, and does not involve any form of AI in any part of its operation.

Not-forking applies changes to a source tree when a normal patch won't work: renamed files, replaced files, generated files, source trees assembled from several upstreams, or text changes which are only valid for some versions. The effect is something like patch + sed + diff + cp + mv + version selection + source retrieval where one recipe describe the desired operations. When upstreams change you run the recipe again, and it adjusts to suit and produces the build artefacts you intended. Not-forking also runs a comprehensive (but not shared) cache, which really helps when dealing with gigabytes of source.

Not-forking is a programming tool for answering this question:

What is the minimum difference between multiple source trees, and how can this difference be applied as versions change over time?

The word difference is more general than a normal diff which deals in approximately adjacent lines. It's also more than a version-control merge which handles histories. A maintainer often wants to express an intention such as use this implementation instead, take only this subtree, make this backend look like that API, or apply this change until upstream absorbs it. Not-forking recipes describe those sorts of maintainer intentions as a reproducible collection of operations.

The Not-forking TODO records future development plans.

The result is a buildable tree without turning the upstream project into a local, reluctant fork. Most experienced developers have handled such a reluctant fork.

The Reluctant Project Fork

Suppose an external project, Upstream, forms part of a new Combined Project. Upstream is source code copied into the Combined Project tree, not a library already installed on the system.

Diagram 1: Not forking with one Upstream

The obvious questions arrive immediately:

Should Upstream be imported into the Combined Project repository? That puts all source under version control, but loses the clean boundary and makes the upstream history awkward to retain.
If Upstream changes, how can those changes be pulled in safely?
If Combined Project changes Upstream files, how can both sets of changes be merged next time?

The easy answer is to copy Upstream into the project and fix conflicts by hand. That answer works once. It becomes a maintenance problem when the copy is made again six months later, when the person who made the original merge has left, or when three upstreams need to move independently.

That is the reluctant project fork: a local copy that nobody intended to maintain as a separate project, slowly drifting from the project whose authors still maintain it. Debian calls a related practice vendoring. It is often necessary, but the maintenance cost is real.

Not-forking keeps the upstream source and the local recipe separate. Each build starts from a selected upstream version and applies the local operations again. A lockfile can record the exact Git commit or Fossil artifact used, so reproducibility does not depend on a tag remaining still.

More Than One Upstream

The problem gets more interesting when Combined Project has two unrelated upstreams.

Diagram 2: Not-forking with two upstreams

The upstreams may have different release schedules, different version numbering, and different version-control systems. One may be a Git repository, one a Fossil repository, and a third a tarball service. The local project may need SQLite 3.53, LMDB 0.9.x, and a development branch of another component at the same time.

LumoSQL is a practical example. It combines SQLite with LMDB and maintains separate recipes for SQLite, LMDB 0.9.x, and LMDB 1.0. Its recipes use source subtrees, version boundaries, multiple repository blocks, branch-tip versions, generated replacements, source mirrors, lockfiles, and anchored changes inside SQLite functions. The complete LumoSQL recipe set tracks upstreams which differ in release schedule, version control system, and source layout.

Diagram 3: Not-forking with multiple versions and multiple upstreams

Not-forking can drive a matrix of these combinations without requiring a separate hand-written merge script for every version pair.

This is not an exotic edge case. Some of the 132 forks of LibVNC are maintained, shipping products which have fallen hundreds of commits behind their original upstream. They may be perfectly legitimate forks, but a recipe-based source integration could remove much of the manual resynchronisation work in cases where the fork was never meant to become permanent.

Why Not Just Use Git, Fossil, or Patches?

Git rebase cannot solve the whole problem. Neither can Git submodules, Fossil merge, or a quilt patch series.

A VCS is excellent at recording a project history. A patch tool is excellent at applying a known line-level difference. Neither is a description of a source integration made from several histories where files may be renamed, replaced, generated, or selected by version.

Consider the sorts of instructions a real maintainer gives:

use foo.c from one upstream but bar.c from another because they expose the same API;
apply this patch to SQLite 3.43 through 3.47, but stop applying it when the upstream implementation changes in 3.48;
replace a complete implementation rather than attempting to merge it line by line;
apply a text substitution to every matching file, even though a VCS merge cannot find a common history;
combine upstream A version 2 and upstream B version 3, both ported to upstream C version 5;
fetch the source from Git, Fossil, or a tarball, then use a local mirror when building the same matrix repeatedly;
stop when an anchored change no longer matches instead of silently creating a source tree with half an intended modification.

Not-forking does not replace the ordinary tools. It puts patch, sed, file replacement, source retrieval, version comparison, and cache management in a single recipe language so that the build system can express the whole job.

What Not-forking Does

An upstream definition says where source comes from and how its versions are named:

vcs = git
repos = https://github.com/sqlite/sqlite
branch = master
version = version-
version_from_branch = 3.54.0
srcmirror = sqlite3-$C sqlite3-$C.tar.gz

version_from_branch gives a moving branch tip a usable version name. A lockfile resolves that name to a concrete commit when it is generated. srcmirror lets repeated builds use an unpacked checkout or archive from a local mirror; $C expands to the selected commit and $V to the selected version.

The LMDB 0.9.x recipe shows how a version line can span repositories:

vcs = git
version_filter = >= 0.9.14 < 1.0-alpha
subtree = libraries/liblmdb
version = LMDB_

block
repos = https://github.com/openldap/openldap

block
repos = https://github.com/LMDB/lmdb

The modification recipe can replace complete files:

method = replace
--
src/btree.c    = files/btree.c
src/btreeInt.h = files/btreeInt.h

It can also append text, make pattern substitutions, apply a conventional unified patch, or apply a patch inside an anchored source fragment. The fragment_patch method is deliberately strict: its in-tree applier uses exact matches and deterministic forward hunk application. It does not use patch(1) fuzz, offset search, or whitespace tolerance. If the code inside an anchor has drifted, the recipe stops and asks for a real maintenance decision.

What It Does Not Do

The recipe author still chooses the upstreams, version names, source subtrees, replacement files, anchors, and version boundaries. Not-forking does not understand Python/C/Rust etc types or control flow, prove that a combination is sensible, or resolve an irreducible conflict.

That is a feature rather than a limitation disguised as magic. A failed source integration should be visible in a build. It should not silently produce a tree which compiles while omitting half of the intended change.

Not-forking is not intended for permanent whole-project forks such as LibreOffice splitting from OpenOffice.org, or MariaDB splitting from MySQL. Those are planned project events. It is intended for local modifications to source whose original authors still maintain the upstream.

Nor is it intended for extreme vendoring where a large, well-funded upstream guarantees that it will maintain all of its own dependencies. Debian discussed that boundary in its January 2021 technical committee decision.

Getting Started

Download the repository from Fossil or the read-only Git mirror:

fossil clone https://lumosql.org/src/not-forking
git clone https://codeberg.org/not-forking/not-forking
curl -L -O https://lumosql.org/src/not-forking/tarball/trunk/lumosql.tar.gz

Build it without installing:

perl Makefile.PL
make
perl -Iblib/lib bin/not-fork -V

Check all optional dependencies:

perl -Iblib/lib bin/not-fork --check-recommend

Try the included examples:

perl -Iblib/lib bin/not-fork -idoc/examples [options] ...

The full manual covers configuration syntax, source caches, lockfiles, modification methods, version filters, mirrors, and the command-line interface. The fragment patch specification defines the deterministic hunk behaviour.

The Name

“Fork” has several meanings. Not-forking addresses the one where source maintained by other people elsewhere is modified locally, creating the burden of maintaining the original codebase as well as the local changes.

In Git, a fork can mean another clone. In Fossil, it can mean a branch point. Neither meaning is the problem here. Fossil has a useful discussion of forking and branching, and GitHub uses “fork” for another clone of a repository. The problem is the accidental, reluctant fork which starts with one copied file and ends with a private upstream that must be merged by hand forever.

Project Links

The main repository is the Fossil repository. The Codeberg repository is a read-only mirror. The complete LumoSQL recipes are available under not-fork.d/.

Licence

Not-forking is released under the MIT licence. Documentation is released under CC BY-SA 4.0. See the files in LICENCES/.