Build state and stale sysroots

Verified against

avocado-cli 1.0.0-rc.5 (src/commands/rootfs/install.rs, src/commands/ext/build.rs, src/commands/ext/install.rs, src/commands/ext/image.rs, src/commands/clean.rs, src/utils/stamps.rs, src/utils/container.rs, src/utils/overlay_preprocess.rs). The incidents described come from a real Jetson Orin project built with this CLI on 2026-09-24 and 2026-09-25.

Short version

Removing or renaming something in an overlay, in enable_services or in a package list does not remove it from the image. After any removal or rename, clean that component. Before anything you flash or ship, build from avocado clean.

Three kinds of state, none of which check the others

Layer	What it records	How it goes stale
avocado.lock	Which package versions to install	Can keep entries for packages you’ve removed, and miss ones you’ve added (seen in practice)
Stamps (/opt/_avocado/<target>/.stamps/)	A hash of each step’s inputs from its last success	Never checks the output. A bad artifact is reused for as long as the inputs don’t change.
Sysroots	The actual files	Updates are added on top. Nothing is ever removed, except when the kernel pin changes.

Overlays add and overwrite; they never delete

Rootfs, initramfs and extension overlays are all applied like this, inside the SDK container:

cp -a "/opt/src/<overlay>/." "$SYSROOT/"     # mode: merge (the default)
cp -r "/opt/src/<overlay>/." "$SYSROOT/"     # mode: opaque

Neither command removes files from the destination. “Opaque” mode doesn’t replace anything either. The official docs and the CLI’s own code comments say it “fully replaces directory contents”, and the code doesn’t do that.

The stamps do notice that the overlay changed; they’ve hashed overlay contents since August 2026 (avocado-cli#210). So the step re-runs. But it re-runs as another additive copy.

Real example. A project masked getty.target with rootfs/etc/systemd/system/getty.target -> /dev/null, then renamed that file to getty-off.target to unmask it. After rebuilding, the rootfs held both files. The mask was still active, and the serial login never came up.

enable_services symlinks are never removed

enable_services: [foo.service] makes the build create etc/systemd/system/<target>.wants/foo.service in the extension. If you remove the service from the list, the symlink stays in the sysroot, so the service stays enabled in every image rebuilt from it.

Removed packages stay installed

dnf installs are additive. Dropping a package from packages: stops it being requested, but doesn’t uninstall it from a sysroot that already has it. The CLI’s own source says so (“dnf is additive”). The one case the CLI handles is a kernel pin change: then it wipes the extension, rootfs or initramfs sysroot before reinstalling, so modules for the old kernel don’t linger.

Up-to-date checks trust inputs, not outputs

Before each step, the CLI hashes that step’s inputs (config section, overlay contents, package list, kernel pin, and the dependency chain) and compares them with the stamp. Nothing checks that the image it produced contains what the inputs describe.

Real example (avocado-cli#283). On 1.0.0-rc.5, three extension images were built from sysroots with none of their packages in them:

Extension	Files in sysroot	Files in image
avocado-bsp-jetson-orin-nano-devkit	515	2
avocado-ext-dev	533	5
display-tester	7	4

Two later builds reused those images, because their inputs hadn’t changed, and one of them was flashed. The empty BSP image had no nvbootctrl, which made the device’s extension merge fail, so no extension services started. See Boot and extension merge.

A later comment on #283 from another reporter narrows this: a fetched, manifest-only BSP extension (source: { type: package, version: '*' }) was retested on 1.0.0-rc.4 and populated correctly, nvbootctrl included. The reproduction that reliably drops packages is an extension with an inline packages: map written directly in the project’s avocado.yaml. The avocado-bsp-jetson-orin-nano-devkit/avocado-ext-dev case above used the fetched form, so either the defect isn’t fully scoped yet or it isn’t limited to the inline case either. Treat both shapes as suspect until this is resolved upstream.

Two extension sysroot locations

Extension sysroots live per runtime at /opt/_avocado/<target>/runtimes/<runtime>/extensions/<ext>. There’s also a legacy path, /opt/_avocado/<target>/extensions, used by any step that runs without AVOCADO_RUNTIME set.

The CLI turns the legacy path into a symlink to the runtime tree, but only if it’s missing, already a symlink, or holds no installed packages. It won’t replace a real directory that has packages in it. A step that reads the legacy path while it’s still a real directory sees different sysroots from the runtime tree. That’s a plausible cause of the empty images above, though not a confirmed one.

The symlink follows the last runtime that ran a command with AVOCADO_RUNTIME set (utils/container.rs, ln -sfn on every such run). The CLI’s own comment lists the commands that still read the legacy path: build, image, clean, runtime build, fetch and hitl. So with two runtimes on the same target, a plain avocado install (which walks every runtime) can leave the symlink pointing at the other runtime, and the next avocado build -r <yours> builds from that runtime’s extension sysroots. Seen in practice: after avocado install without -r, the dev build failed in the SSH extension until avocado install -r dev was run again. Give each runtime its own target (target: on the runtime), or always pass -r to install.

The lock can disagree with the build

Packages written with {{ avocado.kernel.version }} are never recorded in avocado.lock. ext install substitutes the kernel version into the name it installs, but it queries the installed versions using the unsubstituted key (ext/install.rs, the package_names.push(package_name…) line). No package is called kernel-module-foo-{{ avocado.kernel.version }}, so nothing is found and nothing is written. Those packages are never pinned: every clean install takes whatever the feed has for the pinned kernel.

Seen in practice, twice:

• After an extension’s packages were changed from usb-f-ecm and usb-f-rndis to templated names ending in usb-f-ncm, avocado install rewrote avocado.lock but kept the old hand-written ECM and RNDIS entries, with no NCM entry. The sysroot contained only NCM.
• After avocado unlock and a fully clean install, the lock had no packages at all for that extension, although its image had all three .ko files.

Don’t treat the lock as a record of what’s in the image.

Inputs the up-to-date check misses

ext build’s input hash (compute_ext_build_input_hash in utils/stamps.rs) folds a fixed list of extension keys plus some file contents. Changing something outside it doesn’t rebuild the extension:

• Compile sources. For an extension built from an sdk.compile section, the hash covers the compile and install scripts, plus any package_files. It doesn’t cover the source files those scripts read. Edit only main.c and the build reports “up to date” and ships the old binary. List the source directory in the extension’s package_files (for example package_files: [ext/my-tool]); note that this also replaces the default file list avocado ext package bundles.
• modprobe:. The hash lists a key called kernel_modules, which the build no longer reads. It doesn’t list modprobe, which the build turns into release-file lines. So changing only an extension’s modprobe: list leaves the old list in the image. (From reading the source; not reproduced.) Clean the extension after changing it.
• Runtime-level compile steps. Runtime packages: entries with compile/install have no package_files, so the compile-source trap has no fix there. Keep inputs inside the scripts, or have the script check its inputs’ checksums.

avocado clean leaves .avocado/ behind

avocado clean removes the build volume and .avocado-state, and nothing in your project folder. In particular it leaves .avocado/overlay-staging/, where preprocessed overlays are written with their templates filled in. (A comment in utils/overlay_preprocess.rs says avocado clean clears it. commands/clean.rs doesn’t.) Each overlay’s staging copy is replaced the next time that overlay is built. Delete .avocado/ by hand if it may hold secrets.

What to do

Situation	Do this
Removed or renamed a file in an extension overlay, removed an enable_services entry, or removed a package	avocado ext clean -r <runtime> <ext>, then avocado install and avocado build
Same, in the rootfs or initramfs	avocado rootfs clean or avocado initramfs clean, then install and build
About to flash, or producing anything you’ll ship	avocado clean, then avocado install and avocado build. Nothing carries over.
A step keeps getting skipped	avocado --no-stamps <cmd> re-runs it. This doesn’t fix stale files, because the re-run is still additive.
Want newer package versions	avocado unlock --<scope>, then avocado install

avocado clean deletes the whole build volume. The next build creates a new volume with a new name, recorded in .avocado-state. Old volume names stop working.

Verify before you flash

The scripts/check-avocado-build.sh script in this repository compares each built extension image with its sysroot, and flags an image that holds fewer files than the sysroot it came from. That’s exactly the failure described above. It runs read-only against the build volume.

scripts/check-avocado-build.sh ~/path/to/project dev

Caution

The script needs Docker access to the build volume. On macOS it uses the avocado-vm Docker socket. It mounts each image read-only in a throwaway --privileged container. It’s new and has only been tried against one project, so read it before you rely on it.

A few manual checks catch the rest:

• Stale masks: ls -la <rootfs sysroot>/etc/systemd/system/ | grep /dev/null lists every masked unit. Compare the list against your overlay.
• Stale enables: list etc/systemd/system/*.wants/ inside each extension sysroot, and compare with enable_services.
• Image sizes: a BSP extension of a few KB is empty. See Inspecting the build volume.