Contested-Environment Arbitration Runbook (#1493 / ADR-067)

On this page

How to tell "the component mishandles adversity" from "the test/fixture/envelope is bad" when a run flakes or fails. Mechanism detail lives in ADR-067 (the two-layer doctrine, the injectors, the required fault stage); the program and surface taxonomy live in the plan (epic &83). This runbook is the operator-facing decision procedure.

Why this exists

Battery failures used to cost afternoons because nothing separated a real defect from a bad test. This program gives every IN-class component a contested contract — deterministic fault legs (disconnect, reset, confirm-timeout, burst, pool-exhaustion, crash-and-resume) that force the adverse condition. That contract is the arbitration instrument: run it, and its verdict assigns the fix to an owner. It is the objective backstop for the no-environmental-blame rule — "every load/battery failure gets a fix with an owner", never a shrug at "the environment".

The arbitration ladder

When a battery test flakes or fails, do NOT rerun-until-green and move on. Walk the ladder:

  1. Name the component and the behavior. Which surface (from contested-surfaces.toml) and which claimed behavior (its oracle) is implicated? The failing test’s ID maps to a registry entry; the registry names the levers and the oracle.

  2. Run that component’s contested contract in isolation — the surface’s fault legs, off the full battery, on a healthy fault layer (cargo xtask fault-preflight must pass first; see Fault-layer preflight).

  3. Read the verdict:

    Contract verdict Diagnosis Owner + action

    GREEN (fault legs pass in isolation) + battery flaky

    The component handles the adversity correctly. The defect is in the test/fixture/envelope, or it is a capacity attribution (the box was genuinely saturated — see Capacity attribution (the honest "environment" case)).

    The test author. Fix the test/fixture — NOT by making it easier (no added serialization beyond semantic ordering, no envelope widening, no assertion loosening; see the never-easier checklist in testing-reference). If it is capacity, record the attribution.

    RED (a fault leg fails in isolation)

    A real component defect, with a reproducible case in hand.

    File a fix: issue with the exact fault leg + seed as the repro (see Reproducing a failure (bursts are seeded)), /relate it, fix on its own branch. This is the good outcome — the contract caught it deterministically.

    CANNOT RUN (fault-preflight fails)

    The fault layer is absent/unhealthy — the run should never have been treated as green (M9: a zero-fault green battery is a FAILURE).

    The fault-layer owner. Repair the layer (Toxiproxy up + loopback-bound, fault services healthy, lease table reachable). NEVER skip the fault stage to go green.

The ladder is deterministic: every failure exits with an owner and an action. "Transient" / "environmental" is never a terminal disposition — it is at most step 1 of walking the ladder.

Fault-layer preflight

The L2 fault sidecar (Toxiproxy) rides the opt-in fault compose profile. cargo xtask dev brings it up automatically (--profile fault), so a normal devstack always carries a healthy fault layer; a raw docker compose up (no flag) leaves it down (#1499).

cargo xtask fault-preflight gates the fault stage. It is wired into cargo xtask validate (after ensure_ready, before nextest — --required since C24 (#1519)), and is the contested CI job’s guard:

  • Toxiproxy /version healthy on the fault compose profile;

  • its control API published on the host loopback only (verified via docker port — a PROVEN non-loopback publish is a hard fail regardless of report-only mode; the control plane must never be reachable off-host);

  • the port-lease table (fault_port_leases, on the test-plane Postgres) reachable.

A missing layer is a hard failure, not a skip. The only sanctioned fault-optional modes are an explicitly-invoked raw cargo xtask fault-preflight (no --required) and validate --skip-devstack, whose CI-substitute battery (--lib --bins) never reaches this phase.

Reproducing a failure (bursts are seeded)

Burst/permutation floods print their seed and a replay command on every run:

  • the seed is CRAIG_BURST_SEED=<n> in the test output;

  • rerun with that env var set to replay the exact duplicate/shuffle/out-of-order permutation;

  • the retained artifacts (test-results/fault/) carry the armed-vs-fired ledger for the run.

Attach the seed + the failing leg ID to the fix: issue — that is the reproducible case the RED verdict promised.

The program-verification gate (C24)

The gate verifies a WORKING SYSTEM rather than a merged-MR count. It is a PAIR, and the pairing is what makes its central claim true: cargo xtask validate purges + stamps the executed-fault records immediately BEFORE the integration battery and verifies them immediately after, so "the battery executed faults" is a statement about THIS run rather than about the disk.

The six verification phases:

  1. the records belong to THIS run — every record’s mtime must postdate test-results/fault/.gate-stamp, and the stamp is SINGLE-USE: a SUCCESSFUL paired verification consumes it (a failed one leaves all evidence intact for triage; the standalone command never consumes it). Under validate a missing stamp is a hard failure — the pairing broke, or this verification already passed once;

  2. every record file was READ — the aggregation counts files it could not read or parse, and any such count refuses certification; a test killed mid-write leaves missing evidence, not a silent zero;

  3. every armed fault FIRED;

  4. the battery executed faults AT ALL — zero is a FAILURE, not a success;

  5. every recording class fired at least once;

  6. the program’s two self-tests still EXIST in the nextest inventory — a_panic_mid_fault_leaves_residue_that_the_reclaim_path_reaps (the ADR-067 §M2 two-stage cleanup contract) and dsn_guard_refuses_canonical_db (PgFaultArmer never arms a canonical database) — and the accounting is retained as test-results/fault/fault-report.json. This pins test EXISTENCE, not test behavior: it catches a delete or a rename, which is the failure mode that would otherwise pass unnoticed.

cargo xtask program-gate runs the same verification standalone, preceded by fault-preflight --required (so it needs a LIVE fault layer). Standalone it reads whatever records are on disk — a missing stamp is announced rather than fatal, and a present one is left in place — useful for triage after a battery, never a substitute for the paired run.

CAVEAT worth knowing before you trust phase 3: the burst legs record fired: true unconditionally, so today only the pool harness can ever report an unfired fault. Phase 3 has teeth for the recorders that observe firing; making the rest observe it is #1547.

Reading the executed-fault accounting

Every armed fault records to test-results/fault/*.json (crate-local for integration tests — the recorder writes CWD-relative, #1524), aggregated to fault-report.json. Two invariants the battery enforces (M9):

  • every armed fault fired — an armed-but-never-fired fault means the test set up an injection the codepath never reached (a false-negative test), and fails loudly;

  • every RECORDING class has >0 executed faults — a class that ran with zero faults did not exercise its contract, and the run is a FAILURE even if all assertions passed.

The second invariant is enforced for the classes that actually call the recorder (mq-consumer, pool). The other nine IN classes have real, passing legs that never emit a record, so instead of asserting something it cannot prove the gate NAMES them — in the summary of every run that reaches it, and in the retained report’s classes_with_zero_fired field — as the #1546 residue; a class joins RECORDING_CLASSES in xtask/src/cmd/program_gate.rs as it adopts the recorder (the same per-class ratchet shape C23 uses for the registry).

If the report shows a recording class at zero, treat it as a coverage regression: the surface either lost its fault leg or the registry drifted — reconcile against contested-surfaces.toml (the C23 ratchet catches this, but read the report when triaging a "green" run that feels too easy).

Capacity attribution (the honest "environment" case)

Sometimes the box really was saturated (a concurrent heavy job, disk pressure). This is the ONE legitimate "environment" disposition — and it is still not a shrug:

  • it is only reachable via a GREEN contract verdict (the component provably handles the adversity);

  • record the attribution (what else was running, the resource that was contended);

  • if it recurs, it becomes a test-isolation fix (own the fixture’s resources) or a sequencing decision — e.g. the standing one-battery-at-a-time rule that resolved the 2026-08-17 docker-devtools contention. Capacity attribution feeds a fix; it does not close the ticket.

The cluster rig (C20)

cargo xtask cluster-tests runs the out-of-band multi-replica rig (docker-compose.cluster.yml, project craig-cluster-{run}; ADR-067 § Amendment — #1515 records the L3 kill lever). It owns the whole lifecycle: stale-prefix sweep → build (rig-tagged cluster-local images — the devstack’s :local tag is never touched) → up -d --wait → the two kill legs, serialized (one shared rig) → artifacts → signal-safe down -v.

Reproducing / triaging a cluster failure:

  • artifacts land in test-results/cluster/ (ps.txt, logs.txt) on BOTH outcomes — a green run’s files are the baseline the next red run is compared against;

  • re-run locally with cargo xtask cluster-tests (add --keep to hold the rig up for inspection; the next run’s sweep reaps it);

  • a leaked craig-cluster-* project (crash, SIGKILL of the stage) is reaped by the next run’s sweep — never reap by hand with docker compose down against the devstack project;

  • the weekly CI job is allow_failure (devstack-class): a red weekly run needs a look, not a shrug — the local run is authoritative.

The degraded-UX stage (C22)

cargo xtask e2e --degraded runs the browser-level degraded matrix in its own serial, out-of-band stage. It recreates craig-web under docker-compose.fault.yml (backend URLs → standing craig-degraded-* toxiproxy pass-throughs), runs tests/e2e/specs/degraded-flows.spec.ts, then restores the base shape.

Operator notes:

  • the stage holds the devstack lifecycle lock end-to-end — a concurrent dev reseed waits rather than tearing the stack down mid-run;

  • the stage is OUT-OF-BAND: validate/mutants call ensure_ready lock-free by design, so never run one against a devstack a degraded stage is using — the stage’s matrix would fail loudly mid-run (the devstack itself still lands in the base shape);

  • if a run crashes or is killed, .devstack/degraded-overlay stays behind and the next dev start (or validate) restores craig-web to its base shape and clears it — the devstack never silently serves a rerouted BFF. dev reseed/dev clean reach the base shape by tearing down instead;

  • if the STACK is down when the marker is found, the marker is simply cleared — there is no overlaid container to restore, and recreating craig-web alone would build a one-container "devstack" that reports healthy (/readyz is process-up only);

  • a FAILED restore leaves the marker armed deliberately; re-run dev start to retry. Manual escape if it ever wedges: rm .devstack/degraded-overlay then cargo xtask dev start;

  • the specs toggle proxies through the in-network control API (http://toxiproxy:8474) and MUST send a non-browser User-Agent — toxiproxy answers browser UAs with 403 User agent not allowed.

See also

Edit this page · latest