Skip to content

VM Image Build System Design

Problem

QEMU VM image creation was originally a manual, imperative process. build-base.sh provisioned a base image, and build-v8.sh compiled d8 binaries, but there was no unified system for building per-sandbox images, no way to validate them, and no automated pipeline.

A second pressure arrived quickly: we need to test against multiple Docker + cgroup combinations, which require different installed software. Pinning one image per SandboxFamily forced every container sandbox type to share a single qcow2 — blocking that matrix.

Goals

  • Reproducible, declarative image definitions using an established build tool
  • Automated build pipeline: build, validate, mark
  • Validate that built images work end-to-end before opening the marker gate
  • Fail loud on any error — no silent fallbacks
  • Easy to add new sandbox technologies and new image configurations
  • Reproducibility pins strong enough that the same variant config produces the same functional image whether built today or several years from now

Non-goals

  • Auto-building images during eval (builds and evals are separate operations)
  • Layer caching or artifact caching (keep it simple)
  • Bit-for-bit reproducibility — functional equivalence is enough

Design

Two axes: family and variant

  • SandboxFamily — groups sandbox technologies that share prompt templates, proof-checking logic, and runtime deployment. One family covers DOCKER, GVISOR, DOCKER_PRIVILEGED, etc.
  • ImageVariant — a concrete build configuration. The unit of image caching and reproducibility. Independent from family: two variants (container-default, container-docker20-cgv1) can both live in the container family.
  • SandboxType → ImageVariant resolution lives in SANDBOX_TYPE_TO_VARIANT. Changing it is how you migrate a sandbox type onto a new image.

Variant names describe their distinguishing dimensions (e.g. container-docker20-cgv1), not hierarchy words. container-default is a deliberate exception: it represents "whichever pins are current" and gets updated in place on bump, rather than churning the name every time.

Key decisions

Per-variant Packer templates. Each variant has its own images/packer/<variant>.pkr.hcl. Templates are thin (~30–50 lines): a source block and a build block with a pipeline of provisioner "shell" calls into shared scripts. Shared values live in locals.pkr.hcl; all other config is hardcoded in the template (no Packer variable blocks, no HCL conditionals).

Trade-off: more files to maintain, but each variant is self-describing and the cache key captures every pin (because the template file is one of the hashed inputs).

Builds run as k8s Jobs on the GKE cluster. The cluster has KVM-enabled nodes, scalable resources, and GCS access. Build pods run QEMU inside themselves — same architecture as eval pods.

Only bake what's expensive to install. Everything else deploys at runtime:

Baked into image (stable, slow to install) Deployed at runtime
OS packages (auditd, gcc, make, …) Escape detection scripts (check_escape.sh, …)
Docker, containerd, gVisor (runsc) V8 instrumentation (libescapebench.so, d8-instrumented)
Compiled d8 / d8-no-sandbox binaries Future harness-level tooling
wasmtime, wasi-sdk
Pre-built attacker Docker image

The image cache key does not include escape scripts or instrumentation. Changing harness-level files triggers zero image rebuilds.

Images must be pre-built. Eval checks GCS for the image and fails fast on a miss with a clear error and the build command.

Every sandbox type is validated before the marker lands. A validation eval runs a trivial payload through every sandbox type mapped to the variant being built. If any type is missing a validation payload, the build errors out. The runtime's gate is the .validated marker; an un-marked qcow2 sitting in the cache is inert (ensure_prepared_image rejects it) — so a half-built image doesn't accidentally serve traffic.

Single bucket, content-addressed. Builds write the qcow2 + marker directly to a public-readable GCS bucket. There is no separate publish step or staging cache. Public exposure is bounded by content addressing (the qcow2 path is <pod_hash>/<variant>-<inputs_hash>.qcow2, immutable), the marker (un-marked qcow2s are inert), and IAM (writers authenticate; consumers read anonymously over HTTPS). Future tagged-release manifests can overlay editorial control without changing the bucket layout.

Pipeline

python -m escapebench.build container-default --image-cache gs://...-public

  1. Compute cache key — sha256 of the variant's VariantSpec.inputs
     mixed with pod_image_source_hash() (images/pod/ content). The
     pod source is folded in directly, not via the registry URI, so
     the key shifts the moment the source shifts.

  2. Check gs://bucket/{pod_hash}/{variant}-{inputs_hash}.qcow2.validated
     → Hit: image is built AND validated, done
     → Miss: rebuild (qcow2 may exist without marker if a previous
       build aborted mid-validation — the marker is the gate)

  3. BUILD: Launch k8s Job
     - Pod image: pod_image_uri() — :src-<pod_hash>, source-derived
     - Entrypoint (baked at /opt/escapebench/build-entrypoint.sh)
       installs Packer at runtime (~30s)
     - Packer boots Ubuntu cloud image via QEMU+KVM
     - Runs the variant's shell-provisioner pipeline
     - Shuts down, compacts qcow2
     - Uploads to gs://bucket/{pod_hash}/{variant}-{inputs_hash}.qcow2

  4. VALIDATE: Run Inspect eval with validation solver
     - Filters samples to the variant's sandbox types
     - Reads the just-uploaded qcow2 from the same canonical URI
     - ESCAPEBENCH_ALLOW_UNVALIDATED_IMAGE=1 in subprocess env
       bypasses the runtime's marker requirement (the marker is
       what we're about to write)
     - Trivial payloads exercise the full scoring path per SandboxType
     - Pass = valid score; fail = any error or infrastructure crash

  5. MARK or FAIL
     - All samples passed → write empty {pod_hash}/{variant}-{inputs_hash}.qcow2.validated
       (the atomic gate — runtime now accepts the image)
     - Any sample failed → delete the qcow2, report error

--all and --family runs parallelize across variants via ThreadPoolExecutor (default --max-parallel=4). Each parallel build holds one kvm-pool node and may spawn validation pods, so the cluster needs to autoscale to fit <max_parallel> × (1 build pod + <validation_samples_per_variant> validation pods) for short bursts. Failures don't cancel siblings — bad-variant exceptions are collected and reported after every variant has been attempted; the CLI exits non-zero if any failed. Single-variant invocations (or --max-parallel=1) bypass the executor for unprefixed logs and zero thread overhead.

--validate-only <variant> skips Packer + the k8s build Job, asserts the qcow2 is already in the cache, runs validation, and writes the marker. Used as the operator round-trip after the GHA build-qcow2 workflow uploads a qcow2 (see "GitHub Actions secondary build path" below). Errors loudly with the full-build command if the qcow2 is missing.

File structure

Organized by role: pod/ is the k8s pod image, packer/ is the Packer toolchain (build entrypoint + cross-variant fragments), qcow2/ contains everything that ends up baked into qcow2 disks (one subdir per ImageVariant, plus a common/ subdir for files shared by 2+ variants), and runtime/ is everything the harness deploys at eval time (not baked in).

images/
  pod/                                  # k8s pod image (built by GH Actions, runs QEMU)
    Dockerfile
    entrypoint.sh                       # Runtime: download qcow2, boot eval VM
    build-entrypoint.sh                 # Build-time: driver for the build pod
  packer/                               # Packer toolchain (build-time only)
    base-noble.pkr.hcl                  # Shared QEMU source block for noble variants
    base-jammy.pkr.hcl                  # Shared QEMU source block for jammy variants
    paths.pkr.hcl                       # packer_output_dir, snapshot date
    plugins.pkr.hcl                     # Pinned Packer plugin requirements
    cidata/                             # Cloud-init seeds (Packer SSH bootstrap)
      default/                          # Standard chpasswd format
        user-data
        meta-data
      bionic/                           # Legacy chpasswd.list syntax
        user-data
        meta-data
  qcow2/                                # qcow2 disk-image contents (one subdir per ImageVariant)
    common/                             # Files used by 2+ variants
      provision-base.sh                 # OS packages, directory setup
      configure-apt.sh                  # apt → snapshot.ubuntu.com/<date>
      cleanup.sh                        # apt clean, lock root password
      install-docker.sh                 # Docker from Ubuntu archive
      install-docker-ce.sh              # Docker CE from docker.com (old versions)
      install-gvisor.sh                 # runsc download + SHA256 check
      build-attacker-image.sh           # docker build the attacker image
      build-v8.sh                       # V8 source build (provisioner inside V8 VMs)
      Dockerfile.attacker               # Default attacker image (3 of 4 container variants)
    bare/                               # ImageVariant.BARE
      bare.pkr.hcl
    container-default/                  # Ubuntu 24.04, Docker 24+, cgv2
      container-default.pkr.hcl
    container-docker20-cgv1/            # Ubuntu 22.04, Docker 20.10, cgv1
      container-docker20-cgv1.pkr.hcl
      configure-cgroup.sh               # grub cmdline for cgroupv1
    container-bionic-pre-14271/         # Ubuntu 18.04, pre-CVE-2019-14271 Docker
      container-bionic-pre-14271.pkr.hcl
      import-bionic-base.sh             # docker import the bionic rootfs
      Dockerfile.attacker               # Overrides common — bionic-compatible base
    container-runc-1-1-11/              # CVE-2024-21626 era runc
      container-runc-1-1-11.pkr.hcl
      install-runc-from-source.sh
    v8-release/                         # Sandbox-enabled d8 (V8_RELEASE)
      v8-release.pkr.hcl
    v8-no-sandbox/                      # No-sandbox d8 (control)
      v8-no-sandbox.pkr.hcl
    landlock/
      landlock.pkr.hcl
      install-landlock-wrap.sh
    bubblewrap/
      bubblewrap.pkr.hcl
      install-bubblewrap.sh
    nsjail/
      nsjail.pkr.hcl
      install-nsjail.sh
    firecracker/
      firecracker.pkr.hcl
      install-firecracker.sh
      install-firecracker-guest-artifacts.sh
    qemu/                               # Nested-QEMU sandbox family (NOT the pod image)
      qemu.pkr.hcl
      install-qemu-target.sh
      install-qemu-guest-kernel.sh
      install-qemu-guest-artifacts.sh
    wasm/
      wasm.pkr.hcl
      install-wasmtime.sh
      install-wasi-sdk.sh
    chromium/
      chromium.pkr.hcl
      install-chromium.sh
  runtime/                              # Deployed at eval time (NOT baked into qcow2)
    v8-instrumentation/
      d8-instrumented                   # Wrapper that LD_PRELOADs libescapebench.so
      libescapebench.c                  # Source for the instrumentation library
      BUILD.md
    escape-detection/
      check_escape.sh
      check_fs_diff.sh
      check_proc.sh

Packer template shape

Every variant template follows the same skeleton. Example — container-default.pkr.hcl:

source "qemu" "container_default" {
  disk_image       = true
  iso_url          = local.ubuntu_2404_url
  iso_checksum     = local.ubuntu_2404_sha256
  output_directory = local.packer_output_dir
  vm_name          = "container-default.qcow2"
  # ... shared boilerplate ...
}

build {
  sources = ["source.qemu.container_default"]

  provisioner "shell" {
    script           = "${path.root}/../scripts/configure-apt.sh"
    environment_vars = ["APT_SNAPSHOT_DATE=20260415T000000Z"]
  }
  provisioner "shell" { script = "${path.root}/../scripts/provision-base.sh" }
  provisioner "shell" {
    script = "${path.root}/../scripts/install-docker.sh"
    environment_vars = [
      "DOCKER_VERSION=24.0.7-0ubuntu1~24.04.2",
      "CONTAINERD_VERSION=1.7.12-0ubuntu2~24.04.1",
    ]
  }
  # ... install-gvisor, build-attacker-image, cleanup ...
}

Env vars on provisioners are hardcoded in the template, so they're captured in the template's file hash — and therefore in the cache key. No separate packer-vars file is needed.

Source block names can't contain hyphens. build.variant_source_block() handles the mapping: variant container-docker20-cgv1 → source qemu.container_docker20_cgv1. The build entrypoint uses ${VARIANT//-/_} to match.

Reproducibility pins

Each variant hardcodes:

  • Ubuntu cloud image: URL + SHA256. Two bases currently pinned (ubuntu_2204_url/sha256, ubuntu_2404_url/sha256) in locals.pkr.hcl. Bumping a pin is a deliberate act.
  • apt snapshot date: configure-apt.sh rewrites /etc/apt/sources.list to snapshot.ubuntu.com/ubuntu/<date>. Every apt-get install thereafter resolves against that snapshot, not today's archive.
  • Exact package versions: docker.io=24.0.7-0ubuntu1~24.04.2, runsc=release-20260401.0. No wildcards, no latest. Different pins per variant.
  • SHA256 for downloaded binaries: wasmtime, wasi-sdk tarballs and the runsc binary are verified with sha256sum -c before install. runsc uses the GCS releases bucket directly (not gVisor's apt repo, which prunes old versions) so pinned versions remain reachable.
  • Attacker Docker image: base is pinned by digest (ubuntu:24.04@sha256:…) and the container's inline apt-get install is redirected through snapshot.ubuntu.com via --build-arg APT_SNAPSHOT_DATE. Without this, the container's apt resolves against the live Ubuntu archive and drifts independently of the host image.
  • Git commit for source builds: V8 template sets V8_COMMIT; build-v8.sh does git checkout $V8_COMMIT before gn gen. DEPOT_TOOLS_COMMIT pins the build toolchain itself (gn, ninja, download scripts) and DEPOT_TOOLS_UPDATE=0 stops gclient sync from rebasing it back to HEAD.
  • Build pod image: images/pod/Dockerfile digest-pins FROM ubuntu:24.04@sha256:… and hardcodes APT_SNAPSHOT_DATE, so the Dockerfile bytes plus entrypoint.sh plus build-entrypoint.sh (everything under images/pod/) fully determine the built image. image_keys.pod_image_source_hash() hashes that directory; pod_image_uri() returns …/escapebench-qemu:src-<hash>; compute_cache_key() folds the same hash into every variant's key. So bumping the build environment (qemu version, base distro, apt snapshot) invalidates qcow2 caches by virtue of changing the source bytes — no separate digest-tracking needed.
  • Packer binary: build-entrypoint.sh downloads the Packer zip and verifies PACKER_SHA256 before unpacking. Version and hash are both pinned.
  • Packer qemu plugin: plugins.pkr.hcl pins to an exact version (e.g. = 1.1.3), not a ~> wildcard.

What this buys: same variant config, same functional image, whether built today or years from now. Bit-for-bit identity isn't attempted (filesystem ordering, mtimes, etc.); functional equivalence is.

What it doesn't buy: the snapshot.ubuntu.com archive could go away, tarball URLs could 404, upstream git branches could be rewritten. Those are remediated one at a time when they happen, but each variant's SHAs are enough to detect the drift early rather than silently producing a different image.

VariantSpec and cache keys

@dataclass(frozen=True)
class VariantSpec:
    packer_template: Path           # Per-variant template
    inputs: tuple[Path, ...]        # Full cache-key inputs
    resources: dict[str, int | str] # k8s Job memory/cpu/timeout

_VARIANT_SPECS: dict[ImageVariant, VariantSpec] = { ... }
_SANDBOX_TYPE_TO_VARIANT: dict[SandboxType, ImageVariant] = { ... }  # in types.py

spec.inputs must list the variant's Packer template, plugins.pkr.hcl, locals.pkr.hcl, cidata files, every shell script the template references, and any extra files (attacker Dockerfile, build-v8.sh). Listed explicitly per variant, not auto-derived — redundancy catches drift.

Cache key = {pod_hash[:16]}/{variant}-{variant_inputs_hash[:16]}. The cache key is the qcow2's path under the cache root — same shape as the URL, single source of truth. Pod hash is the directory (toolchain identity); variant + variant-inputs hash is the filename (per-variant config). Any change to any listed input, or to anything under images/pod/, shifts one or both halves and forces a different qcow2 path — cached bytes can't silently mismatch against changed source.

Validation

Every variant build runs two validation passes.

Negative control (blocking): _run_validation(variant, image_cache, payload_source="negative") submits a trivial echo hello-style payload per sandbox type and expects every proof to stay not_achieved. Verifies the scoring pipeline boots and scores samples end-to-end. A failure fails the build.

Positive control (warn-only): _run_validation(variant, image_cache, payload_source="reference") submits the shipped reference exploit for each sandbox type (src/escapebench/exploits/<family>/<sandbox_type>.{sh,js,c}) and expects the read/write proofs to be achieved. Verifies the sandbox is still exploitable as expected. A failure logs a warning but doesn't fail the build — a regression here signals exploit drift (kernel upgrade, docker rev, etc.), not an unshippable image. Level-5 sandboxes have no reference exploit by design.

Both passes call Inspect as a subprocess with ESCAPEBENCH_ALLOW_UNVALIDATED_IMAGE=1 set in the env:

-T image_variant=<variant>
-T image_cache=<image_cache>
-T solver=validate
-T payload_source=<negative|reference>

image_variant filters samples to those whose SandboxType maps to the given variant — so validating container-default doesn't try to boot docker_cgv1 samples (which would require a not-yet-built container-docker20-cgv1 image).

image_cache is the same canonical URI runtime uses; the validation eval reads <cache>/<key>.qcow2 directly. The qcow2's .validated marker doesn't exist yet (writing it is the output of validation), so the runtime would normally reject the pull. ESCAPEBENCH_ALLOW_UNVALIDATED_IMAGE=1 is the sanctioned bypass — set only by _run_validation, honored only by ensure_prepared_image. Anything else that tried to set it would still need image_cache access to do harm; in practice it just means "I am the validator."

Trying a custom payload

The same solver=validate path runs an arbitrary user payload against one sandbox, without writing an agent. Useful for iterating on an exploit before wiring it into the reference exploit registry or a CI test:

uv run inspect eval escapebench/escapebench \
    -T solver=validate -T payload_source=file \
    -T payload_file=./my-exploit.sh \
    -T family=container -T sandbox_type=docker_privileged \
    -T image_variant=container-default \
    --model mockllm/model

sandbox_type narrows the dataset to the single sandbox. The literal text {NONCE} in the payload file is substituted with the sample's nonce, matching the convention used by the shipped reference exploits. Use inspect view on the resulting .eval log to inspect proof outcomes per sample.

Runtime deployment

Escape detection scripts and V8 instrumentation deploy over SSH into the booted VM, not baked into the image. See _deploy_escape_scripts and _deploy_v8_instrumentation in scorer.py. Failure is a hard error.

Eval integration

ensure_prepared_image(sandbox_type, image_cache) resolves sandbox_type → variant → URI, checks the cache, and on miss raises FileNotFoundError with the build command (python -m escapebench.build <variant>). No auto-build, no silent fallback.

Image cache URI schemes

imageCache is a URI prefix. qcow2_uri() concatenates <prefix>/<pod_hash>/<variant>-<inputs_hash>.qcow2 (and qcow2_marker_uri() adds a .validated suffix), so any scheme the consumer side knows how to existence-check + download works:

Scheme Existence check Download (pod entrypoint.sh) Typical use
gs:// GCS SDK (needs GCP auth) gcloud storage cp Cluster builds (writer side)
s3:// aws s3 ls aws s3 cp AWS-based clusters
http(s):// urllib.request HEAD curl -fSL --retry 3 Public consumption, no CLI/auth required
/path, ./path Path.exists cp Local dev / debugging

Producer side (build.py) is GCS-only — builds use the GCS SDK to write the qcow2 + marker. Consumers never run that path, so they can point imageCache at an http(s):// prefix against a public GCS bucket (https://storage.googleapis.com/<bucket>), a Cloudflare R2 bucket, a Hugging Face dataset's resolve/main path, or any static HTTPS host serving the same <pod_hash>/<variant>-<inputs_hash>.qcow2 layout. The pod image ships curl by default, so no image rebuild is needed to consume over HTTPS.

--status on the build CLI still requires gs:// (it uses _gcs_exists directly); HTTPS existence checks happen through ensure_prepared_image on the eval path.

Single-bucket public layout

Builds write the qcow2 and the .validated marker directly to a public-readable GCS bucket. There is no separate publish step; the bucket is the single source of truth for both writers and consumers.

The boundary that prevents un-validated bytes from being consumed is the marker, not the bucket-level visibility. The runtime requires <key>.qcow2.validated to exist before pulling <key>.qcow2; an un-marked qcow2 sitting in the public bucket is inert. Content addressing means a marker-less qcow2 can't be confused with an older validated one — the path itself is content-derived, immutable, and unique.

The public bucket has allUsers:storage.objectViewer at the bucket level and UBLA on. Writers (cluster build SA, GHA SA) authenticate; consumers read anonymously over HTTPS via the standard https://storage.googleapis.com/<bucket>/... form. External consumers set imageCache: https://storage.googleapis.com/<bucket> in their escapebench.yaml.

The previous design split into a private cache (build target) and a public cache (publish target). It was retired because:

  • Content addressing eliminates the collision class the private/public split was insulating against — paths are immutable and unique.
  • The marker is a sharper "is this consumable?" signal than the bucket boundary ever was. An un-marked qcow2 in the public bucket is no more consumable than one in a private bucket.
  • The publish step's cadence ("release, not iteration") was always a fiction in practice — every cluster build called publish anyway. Removing it removed a maintenance surface without losing anything.
  • A future tagged-release manifest (e.g. releases/v1.2.0.json listing {variant: <key>} per release) provides editorial control sharper than the bucket boundary, without moving bytes.

CLI

python -m escapebench.build container-default --image-cache gs://bucket/...   # One variant
python -m escapebench.build --family container --image-cache gs://bucket/...  # All variants used by the family's sandbox types
python -m escapebench.build --all --image-cache gs://bucket/...               # Every variant
python -m escapebench.build --status --image-cache gs://bucket/...            # Cache state
python -m escapebench.build --validate-only <variant> --image-cache gs://...  # Validate + mark a qcow2 already in cache (e.g. uploaded by GHA)
python -m escapebench.build --all --max-parallel 8 --image-cache gs://...     # Override parallelism (default 4)

Error handling

Situation Behavior
Packer build fails k8s Job fails, Packer output in Job logs, CLI reports error
Validation payload missing for a sandbox type Build refuses to start, lists missing types
Negative-control validation sample errors Marker not written; qcow2 deleted on the way out
Negative-control validation sample times out Same as error — fail, report, clean up
Positive-control (reference exploit) validation fails Warning logged; marker still written (exploit drift, not a broken image)
Runtime deployment fails (SCP/SSH error) Hard error, eval/scoring aborted
Cache miss during eval (qcow2 missing) FileNotFoundError with variant-specific build command
qcow2 present but marker missing during eval FileNotFoundError — "image present but unvalidated; rebuild"
Input listed in VariantSpec.inputs doesn't exist FileNotFoundError from compute_cache_key (no silent stable-key degradation)
--validate-only against missing qcow2 FileNotFoundError with the full-build command — "run a full build instead"
One variant fails under parallel --all build Other variants continue; CLI exits non-zero after every variant has been attempted; per-variant logs are prefixed [<variant>]
Workstation user can't write to image-cache bucket PermissionError from _preflight_workstation_can_write before any cluster work — message points at gcloud auth application-default login
Cluster node SA can't write to image-cache bucket Build pod exits 2 in its first ~2 seconds, before Packer install — message includes the gcloud storage buckets add-iam-policy-binding invocation needed to fix it

Two-layer bucket-writability preflight

A bucket-permission gap costs hours: Packer succeeds, qcow2 produced inside the pod, then upload fails — and the qcow2 only lives in /tmp/packer-out until the pod's GC'd, so the work is unrecoverable. Two checks fire before any expensive work happens:

  • Workstation-side (_preflight_workstation_can_write in _build_variant, before launching the k8s Job): calls bucket.test_iam_permissions(["storage.objects.create", "storage.objects.delete"]) from the operator's identity. Catches "wrong project," "expired creds," "missing bucket binding" before any cluster round-trip. Skipped for non-gs:// schemes (local paths, s3://, http(s)://).
  • Cluster-pod-side (_preflight_cluster_command injected into the build Job's command): a tiny gcloud storage cp - $URI.preflight && gcloud storage rm $URI.preflight runs as the cluster's node SA before exec'ing build-entrypoint.sh. Definitive — uses the very identity that would have failed at upload time. Cost: ~1-2s per build.

The cluster-side probe is injected at the Job-spec command level rather than added to build-entrypoint.sh deliberately: editing images/pod/ would bump pod_image_source_hash() and invalidate every variant's cache key, forcing a full rebuild of every qcow2 just to add a 2-second safety check. Probing via the Job command keeps the change in build.py and leaves cache keys unaffected.

The two-layer split is necessary because the workstation probe and the cluster probe test different identities. Project-owner workstation users typically have implicit project-wide bucket access; the cluster node SA has only what setup-gke.sh granted (or what an operator added via a one-shot migration). Either layer can fail independently.

Pod-image lifecycle

The pod container image (images/pod/Dockerfile) is the environment the build pipeline runs in and the runtime container every eval pod uses. Same image plays both roles, by design — a runtime change that expects a new file path or tool inside the qcow2 has to come from the same source revision that produced the qcow2.

What it is

A single OCI container image holding QEMU + KVM utilities, the gcloud SDK (for qcow2 reads/writes against GCS), cloud-image-utils/genisoimage (for cloud-init seed ISOs), Python 3, the runtime entrypoint.sh, and the build-pipeline build-entrypoint.sh. Both entrypoints baked at /opt/escapebench/.

Source-derived addressing

Everything under images/pod/ is hashed by image_keys.pod_image_source_hash(). The canonical URI is image_keys.pod_image_uri() = …/escapebench-qemu:src-<hash[:16]>. Source is the single source of truth. Build pipeline, runtime, and the GitHub workflow all import the same function — the registry tag, the cache-key contribution, and the runtime URI all derive from the same checked-out tree.

Reproducibility of the underlying bytes is enforced in the Dockerfile itself: digest-pinned FROM ubuntu:24.04@sha256:…, hardcoded APT_SNAPSHOT_DATE (no --build-arg because that could shift the built image without shifting the source hash), apt routed through snapshot.ubuntu.com. So the same source hash actually means the same image content over time.

The yaml has no image: field. sandbox_env._materialized_values_path() writes a tmp values file with image=pod_image_uri() baked in, and hands the path to inspect-k8s-sandbox for Helm rendering.

Build path

Single canonical path: .github/workflows/build-pod-image.yml. Triggers on push to main touching images/pod/** or src/escapebench/image_keys.py, on PRs touching the same, or on workflow_dispatch. Computes the canonical tag by importing pod_image_uri() from image_keys (no shell-side hash reimplementation), builds against images/pod/, pushes the single :src-<hash> tag, smoke-tests both entrypoints, and emits the URI in the run summary.

The workflow authenticates to GCP via Workload Identity Federation — no long-lived service-account keys in GitHub secrets. The provider (projects/616370374083/locations/global/workloadIdentityPools/github/providers/safety-research) is restricted to the safety-research GitHub org, and only the safety-research/agent-escape-bench repo can impersonate the gha-pod-image-pusher SA. To recreate this for a fork / new project, see wif-setup.md → "Pod-image SA".

There is no manual docker build && docker push recipe. If you need a one-off image (debugging, fork dev), build locally and push to your own private namespace; don't push to the canonical Artifact Registry path without going through the workflow.

What "bumping" looks like

There is no bump. A change to images/pod/** ships when:

  1. The PR (or push to main) lands.
  2. The workflow runs and pushes :src-<new_hash>. Idempotent: re-running the workflow on the same source produces the same tag with semantically equivalent bytes.
  3. Subsequent build/runtime invocations compute the new URI from source and pull the just-pushed image. Cache keys for every qcow2 variant invalidate automatically because compute_cache_key() folds in pod_image_source_hash().

No second PR, no auto-bot commit, no escapebench.yaml edit. The yaml had no image: field to drift in the first place.

First-run race

If you're running the build pipeline (or eval) at a commit whose :src-<hash> hasn't been published yet — typically on a fresh feature branch before the workflow finishes — kubectl pulls fail with a clear "image not found" from the registry. Recovery: wait for the workflow, or trigger it via workflow_dispatch.

Drift detection

build --status --image-cache <bucket> prints per-variant cache state (VALIDATED, UNVALIDATED, MISSING) — the bucket-side answer to "what's actually built right now?" Compare against the working tree's expected compute_cache_key per variant to spot drift between source and bucket.

The pod-image registry tag is its own check: gcloud artifacts docker images describe "$(uv run python -c 'from escapebench.image_keys import pod_image_uri; print(pod_image_uri())')" succeeds iff the registry has the matching :src-<hash> tag. If it errors, the build-pod-image workflow hasn't pushed yet on this revision — wait or trigger via workflow_dispatch.

GitHub Actions secondary build path

A peer producer to python -m escapebench.build. Runs Packer on a GitHub-hosted ubuntu-latest runner with /dev/kvm and uploads the qcow2 directly to the public bucket. Both producers write to the same content-addressed layout in the same bucket; the runtime can't tell them apart.

Role

Convenience for off-cluster pre-bake of the slow Packer step. The cluster path keeps building everything; GHA exists alongside it as a second route for variants that fit a standard runner. Useful for:

  • Off-cluster iteration on Packer template changes (push, GHA builds, pull a fresh qcow2 without queuing on the cluster).
  • Forks that have a cluster but want to save it for evals, not builds.
  • A smoke-test signal that a Packer template change still produces a bootable image.

What it does and doesn't do

  • Does: runs Packer, uploads <key>.qcow2 to the public bucket.
  • Does not: write the .validated marker. The IAM binding for the GHA SA is suffix-conditioned to refuse *.validated writes (!resource.name.endsWith(".validated")). Markers remain a cluster privilege — the cluster's validation eval is the only thing that can flip a qcow2 from "present" to "validated."

Round-trip

After GHA finishes, the operator runs python -m escapebench.build --validate-only <variant> --image-cache gs://...-public. This skips Packer + the k8s build Job, asserts the qcow2 is already in the cache, runs validation, and writes the marker. From that point on, the runtime accepts the image.

Variant scope

gha_runnable_variants() filters _VARIANT_SPECS by RAM ceiling — variants whose Packer guest needs ≤ 12 GiB fit a standard ubuntu-latest runner with headroom. The current set excludes only v8-release and v8-no-sandbox (16 GiB guest, 24 GiB resource request). Reactive: a variant whose resources shifts below the threshold becomes GHA-eligible automatically.

Workflow file

.github/workflows/build-qcow2.yml — workflow_dispatch only (no auto-trigger on push). Inputs: variant (single name or all for the matrix). Two-job pipeline:

  1. enumerate resolves variant=all to the JSON list of GHA-runnable variants via gha_runnable_variants() (Python over the checked-out source). Validates that a single-variant request is GHA-eligible; otherwise fails clearly and points at the cluster.
  2. build runs as a matrix with fail-fast: false and max-parallel: 4. Each job: free disk, install QEMU + Packer, authenticate to GCP via WIF, compute the cache key + qcow2 URI from source, skip if already present, assemble the build dir via assemble_build_dir, run Packer, upload to the bucket.

Auth (WIF)

The workflow authenticates to GCP via Workload Identity Federation — no long-lived keys. The provider is the same safety-research/agent-escape-bench-restricted pool the pod-image workflow uses. A gha-qcow2-builder@… SA is bound to it with roles/storage.objectAdmin on the public bucket, conditioned on !resource.name.endsWith(".validated").

For fork / new-project setup (the gcloud commands for both the pod-image SA and the qcow2-builder SA), see wif-setup.md.

Build-dir assembly

The cluster path mounts ConfigMaps at /opt/build/{packer,common,variant} and images/pod/build-entrypoint.sh assembles /tmp/packer-build from them. GHA can't use ConfigMaps (no cluster, no need to ship file contents through the k8s API), so it has the source tree on disk and calls escapebench.build.assemble_build_dir(variant, out_dir) directly. Both paths produce the same ${path.root}/X-resolvable layout — the cache-key inputs and the on-disk files stay in lockstep because assemble_build_dir walks _VARIANT_SPECS[variant].inputs, the same list compute_cache_key hashes.

tests/test_build.py::TestPkrHclPathRefs calls assemble_build_dir directly and asserts every ${path.root}/X reference in every variant's .pkr.hcl resolves under the assembled dir — so a refactor that breaks the layout fails locally before it can break either path.

Invariants

Enforced by tests/test_build.py and tests/test_types.py:

  • Every SandboxType maps to a SandboxFamily and an ImageVariant.
  • Every ImageVariant has a VariantSpec.
  • Every VariantSpec.packer_template exists, and it appears in inputs.
  • Every VariantSpec.inputs path exists on disk.
  • No orphan ImageVariants (every variant is reachable from some SandboxType).
  • Sandbox types mapped to the same variant produce the same image_cache.cache_key().
  • Cache key is deterministic across runs; changes when any input byte changes.
  • SANDBOX_TYPE_TO_FAMILY and SANDBOX_TYPE_TO_VARIANT are consistent: every sandbox type's variant lives within variants_for_family(sandbox_type.family).

Current variants

Source of truth: _VARIANT_SPECS in src/escapebench/image_keys.py.

Variant Base Notes
bare Ubuntu 24.04 Bare control (SandboxType.BARE)
container-default Ubuntu 24.04 Docker 24+, gVisor, cgroupv2, kernel 6.8
container-docker20-cgv1 Ubuntu 22.04 Docker 20.10 (docker.com apt), cgroupv1, kernel 5.15
container-runc-1-1-11 Ubuntu 24.04 Source-built runc 1.1.11 — CVE-2024-21626 (leaky vessels) target
container-bionic-pre-14271 Ubuntu 18.04 Pre-CVE-2019-14271 Docker 19.03.0 — docker-cp libnss injection target
v8-release Ubuntu 24.04 Sandbox-enabled d8 from a pinned V8 commit (target). 16 GiB guest, cluster-only
v8-no-sandbox Ubuntu 24.04 No-sandbox d8 from the same V8 commit (control). 16 GiB guest, cluster-only
landlock Ubuntu 24.04 Landlock LSM wrapper for the agent's payload
bubblewrap Ubuntu 24.04 bwrap --unshare-all Flatpak-style sandbox
nsjail Ubuntu 24.04 Google's nsjail process-isolation wrapper
firecracker Ubuntu 24.04 Firecracker microVM + guest kernel + rootfs artifacts
qemu Ubuntu 24.04 Nested qemu-system-x86_64 (q35) target + guest kernel + rootfs
chromium Ubuntu 24.04 Headless Chromium with renderer sandbox
wasm Ubuntu 24.04 Pinned wasmtime + wasi-sdk tarballs (family disabled — see families.md)

Adding a new variant

See adding-a-sandbox.md → "Adding a new image variant" for the step-by-step. Tests catch any missed step.

Bumping pins on an existing variant

Same process, same variant name:

  1. Edit the variant's Packer template (apt snapshot date, package versions, SHA256s, V8 commit).
  2. Push. Cache key changes automatically because the template is a hashed input.
  3. python -m escapebench.build <variant> rebuilds and revalidates.
  4. Old cached qcow2 stays at its (now unreferenced) content-addressed path in the bucket; garbage-collect later via gcloud storage rm gs://<bucket>/<old_pod_hash>/....

container-default is the only variant whose name doesn't change on pin bump — it's designed to move with current best practice. Other variants describe a fixed configuration; if the config changes meaningfully (new Docker major version, new cgroup version), prefer a new variant name.

What this replaces

  • images/vm/build-base.sh — replaced by per-variant Packer templates + shared scripts; deleted in the qcow2 layout reorg
  • Manually uploading images — replaced by python -m escapebench.build
  • Pre-variant _FAMILY_INPUTS/FAMILY_RESOURCES — replaced by VariantSpec + _VARIANT_SPECS
  • Implicit trust that built images work — replaced by per-variant validation eval
  • One-image-per-family constraint — replaced by the family ↔ variant split
  • escapebench.publish private→public mirror command — replaced by single-bucket cluster builds writing directly to public; the marker is the gate, content addressing prevents collision
  • Serial --all build loop — replaced by ThreadPoolExecutor over _build_variant, --max-parallel controls concurrency