The Not-forking Software Reproducibility Tool

Not-forking is for the surprisingly common situation where you need somebody else's source code, but using it unchanged is not an option and maintaining your own fork sounds like a terrible idea. You need a precise way to express this is my intention when combining these source trees. Not-forking is deterministic, and does not involve any form of AI in any part of its operation.

Not-forking applies changes to a source tree when a normal patch won't work: renamed files, replaced files, generated files, source trees assembled from several upstreams, or text changes which are only valid for some versions. The effect is something like patch + sed + diff + cp + mv + version selection + source retrieval where one recipe describes the desired operations. When upstreams change you run the recipe again, and it produces the source tree you intended from the new upstream versions. Not-forking also keeps a cache of upstream repositories, so a repeated run does not fetch gigabytes of source again.

Not-forking answers this question:

What is the minimum difference between multiple source trees, and how can this difference be applied as versions change over time?

The word difference is more general than a normal diff which deals in approximately adjacent lines. It's also more than a version-control merge, which handles histories. A maintainer often wants to express an intention such as use this implementation instead, take only this subtree, make this backend look like that API, or apply this change until upstream absorbs it. Not-forking recipes describe those sorts of maintainer intentions as a reproducible collection of operations.

The result is a buildable tree without turning the upstream project into a local, reluctant fork. Most experienced developers have handled such a reluctant fork, and not enjoyed it.

The Reluctant Project Fork

Suppose an external project, Upstream, forms part of a new Combined Project. Upstream is source code copied into the Combined Project tree, not a library already installed on the system.

Diagram 1: Not forking with one Upstream

This raises three questions:

Should Upstream be imported into the Combined Project repository? That puts all source under version control, but loses the clean boundary and makes the upstream history awkward to retain.
If Upstream changes, how can those changes be pulled in safely?
If Combined Project changes Upstream files, how can both sets of changes be merged next time?

The easy answer is to copy Upstream into the project and fix conflicts by hand, but that becomes a maintenance problem when the copy is made again six months later, when the person who made the original merge has left, or when three upstreams need to move independently.

That is the reluctant project fork: a local copy that nobody intended to maintain as a separate project, slowly drifting from the project whose authors still maintain it. Debian calls a related practice vendoring. It is often necessary, and someone has to bring each vendored copy up to date by hand.

Not-forking keeps the upstream source and the local recipe separate. Each build starts from a selected upstream version and applies the local operations again. A lockfile can record the Git commit or Fossil artifact used, so a build can be repeated after an upstream tag or branch moves.

More Than One Upstream

Combined Project may have two or more unrelated upstreams.

Diagram 2: Not-forking with two upstreams

The upstreams may have different release schedules, different version numbering, and different version-control systems. One may be a Git repository, another a Fossil repository, and another a tarball on a web server.

LumoSQL is a practical example. It combines SQLite with LMDB or libdb and maintains separate recipes for SQLite, LMDB 0.9.x, LMDB 1.0 and libdb. Its recipes use source subtrees, version boundaries, multiple repository blocks, branch-tip versions, generated replacements, source mirrors, and anchored changes inside SQLite functions. The complete LumoSQL recipe set tracks upstreams which differ in release schedule, version control system, and source layout.

Diagram 3: Not-forking with multiple versions and multiple upstreams

Not-forking can drive a matrix of these combinations without requiring a separate hand-written merge script for every version pair.

For example, some of the very many GitHub forks of LibVNC are maintained. These are often shipping products hundreds of commits behind their original upstream. They may be justfied and funded forks - but a recipe-based source integration could remove the manual resynchronisation work in cases where the fork was not desired, or never meant to become permanent.

Git, Fossil and Patches

Git rebase cannot solve the whole problem. Neither can Git submodules, Fossil merge, or a quilt patch series.

A VCS is excellent at recording a project history, and a patch tool does a great job applying a known line-level difference, but even together they cannot describe a source integration made from multiple trees with all common operations. Files may be renamed, replaced, generated, modified, merged or selected by version and all that typically happens manually.

Consider the sorts of instructions a maintainer gives:

use foo.c from one upstream but bar.c from another because they expose the same API;
apply this patch to SQLite 3.43 through 3.47, but stop applying it when the upstream implementation changes in 3.48;
replace a complete implementation rather than attempting to merge it line by line;
apply a text substitution to every matching file, even though a VCS merge cannot find a common history;
combine upstream A version 2 and upstream B version 3, both ported to upstream C version 5;
fetch the source from Git, Fossil, or a tarball, then use a local mirror when building the same matrix repeatedly;
stop when an anchored change no longer matches instead of silently creating a source tree with half an intended modification.

Not-forking does not re-invent the ordinary tools. Instead it processes recipes according to a simple grammar, where a single recipe can contain all or any operations such as: patch, sed, file replacement, source retrieval, version comparison, and cache management. With not-forking, the build system can express the whole job.

What Not-forking Does

A recipe is a directory with one subdirectory for each upstream. Each subdirectory holds an upstream.conf, which says where the source comes from and how its release tags map to versions, and any number of .mod files, each describing one change. Whole files that the recipe supplies go in files/. The sqlite example included with not-forking has this layout:

doc/examples/
  sqlite/
    upstream.conf
    btree.mod
    vdbeaux.mod
    vdbeaux-sed.mod
    files/
      btree.c
      btreeInt.h

A minimal upstream.conf gives the version-control system, the repository, and the prefix of its release tags. The not-forking example tracks this tool from its own Fossil repository, where the tag version-0.3 is version 0.3:

vcs = fossil
repos = https://lumosql.org/src/not-forking
version = version-

A .mod file has a header, a line of dashes, and then the change. The header gives the method and, optionally, the versions the change applies to. This sed change renames a function call in SQLite versions before 3.31, and not-forking skips it for later versions:

method = sed
version < 3.31
--
vdbeaux.c : sqlite3PagerIsMemdb = BackendIsMemdb

Each line after the dashes gives a file, a regular expression and its replacement.

A replace change substitutes complete files. Paths on the left are in the upstream tree, and paths on the right are in the recipe's own subdirectory:

method = replace
--
src/btree.c    = files/btree.c
src/btreeInt.h = files/btreeInt.h

Not-forking fetches the selected version of each upstream, applies its .mod files, and writes the result under the output directory:

not-fork -i doc/examples -o sources -v 3.30.1 sqlite

This writes SQLite 3.30.1 to sources/sqlite, with both replacement files and the vdbeaux.c change in place, ready for SQLite's normal build.

One upstream can draw its versions from more than one repository. The LumoSQL recipe for the LMDB 0.9.x backend selects only 0.9.x versions, installs only the libraries/liblmdb subtree, and finds each release in either the OpenLDAP repository or the standalone LMDB repository:

vcs = git
version_filter = >= 0.9.14 < 1.0-alpha
subtree = libraries/liblmdb
version = LMDB_

block
repos = https://github.com/openldap/openldap

block
repos = https://github.com/LMDB/lmdb

A .mod file can also append text, apply a conventional unified patch, or apply a patch inside an anchored source fragment. The fragment_patch method applies each hunk only where its lines match the upstream source exactly, in order. not-forking implements its own internal patch algorithm taken from patch(1) using no fuzz, offset search, or whitespace tolerance. If upstream has changed the code inside an anchored fragment, the build stops, and the recipe must be updated by hand. It is surprising how rarely this occurs.

An upstream can also be a tarball download or a moving branch tip. A lockfile records the commit that each version resolved to, so a later build repeats it, and a local source mirror saves fetching the same source again. The full manual describes these.

Work that still needs to be done

The recipe author still chooses the upstreams, version names, source subtrees, replacement files, anchors, and version boundaries. Not-forking does not understand Python/C/Rust etc types or control flow, or prove that a combination is sensible, or resolve an irreducible conflict.

Not-forking halts with an error when a patch or fragment patch no longer applies.

Not-forking is not intended for permanent whole-project forks such as LibreOffice splitting from OpenOffice.org, or MariaDB splitting from MySQL. Those are planned project events. It is intended for local modifications to source whose original authors still maintain the upstream.

Nor is Not-forking intended for extreme vendoring where a large, well-funded upstream guarantees that it will maintain all of its own dependencies. Debian discussed that boundary in its January 2021 technical committee decision.

Getting Started

Clone the Fossil repository or the read-only Git mirror, or download a tarball using one of:

fossil clone https://lumosql.org/src/not-forking
git clone https://codeberg.org/not-forking/not-forking
curl -L -O https://lumosql.org/src/not-forking/tarball/trunk/not-forking.tar.gz

Build it without installing:

perl Makefile.PL
make
perl -Iblib/lib bin/not-fork -V

Check every program and Perl module that not-forking can use:

perl -Iblib/lib bin/not-fork --check-recommend

Try the included examples:

perl -Iblib/lib bin/not-fork -i doc/examples -o sources -v 3.30.1 sqlite

The full manual covers configuration syntax, source caches, lockfiles, modification methods, version filters, mirrors, and the command-line interface. The fragment patch specification defines the deterministic hunk behaviour.

The Name

"Fork" has several meanings. Not-forking addresses the one where source maintained by other people elsewhere is modified locally, creating the burden of maintaining the original codebase as well as the local changes.

On GitHub, a fork is another clone of a repository. In Fossil, a fork is a check-in with two children on the same branch, as Fossil's discussion of forking and branching explains. Neither meaning is the problem here. The problem is the accidental, reluctant fork which starts with one copied file and ends with a private upstream that must be merged by hand forever.

Project Links

The main repository is the Fossil repository. The Codeberg repository is a read-only mirror. The complete LumoSQL recipes are available under not-fork.d/. The Not-forking TODO records future development plans.

Licence

Not-forking is released under the MIT licence. Documentation is released under CC BY-SA 4.0. See the files in LICENCES/.