Testing Reference (CRAIG)

On this page
Contents

Critical Rules

  1. Never dismiss test failures as transient — investigate root cause. Only classify as edge case after thorough analysis.

  2. All checks must pass before push — the pre-push hook runs the full test battery (validate + E2E + k6 smoke)

  3. Test results in test-results/ — check this directory for failure context before investigating

  4. Never weaken a test to make code pass — fix the code, not the test. Specifically:

    • Never delete or weaken a test to make CI pass

    • Never [ignore] a test to silence a failure. Distinct from [ignore = "requires devstack"] which is the mandated canonical pattern per Plan D F-024 — see Devstack-gated tests section below

    • Never loosen assertions (e.g., assert_eq! → assert!, widening tolerance, removing checks) to accommodate broken code

    • Never change expected values to match wrong output

    • The ONLY reason to modify a test is if the test itself is genuinely incorrect — and that must be explained in the commit message

  5. All test types produce JUnit XML — unit, integration, and E2E tests all output to test-results/

Audit platform invariants before code-level smells

A code-quality or "incomplete work" audit runs in two passes, in this order — pass-1 findings outrank pass-2 findings even when pass-2 has more items:

  1. Platform invariants. Does encryption fail closed? Does auth validate audience and scope? Are events transactional with the DB write? Are DLQs actually wired? Do health checks distinguish ready / live / degraded? Is idempotency persistent? Are cross-service references reconciled?

  2. Code-level smells. TODO / FIXME, dead code, test-skips, dependency duplicates — the standard greps.

Storage shape is not verification. Reading "the table exists, the worker is spawned, the helper is exposed" confirms scaffolding, not semantic correctness under concurrency and failure. For each invariant, name the failure mode you would have to test — "if two replicas run this at once, what should happen, and is there a test that fires that scenario?" If the test does not exist, the invariant is unverified. Recurring examples where the shape passed but the semantics failed:

  • Idempotency — (middleware-era, historical) a real table and middleware, but ON CONFLICT DO NOTHING on the response write after the handler ran was not an atomic claim before it; two concurrent same-key POSTs both missed the cache and both executed the handler. The ADR-062 §B request_claims substrate (claim-first inside the domain tx, B2 #1194) is the current answer; its battery pins the one-execution property directly.

  • Outbox — a spawned worker with SELECT-then-UPDATE, but no FOR UPDATE SKIP LOCKED / lease column means multi-replica double-publish.

  • Inbox — ON CONFLICT DO NOTHING claims the row before the handler runs, so on handler failure the redelivery hits the existing row, short-circuits, and the work never retries.

Distinguish layers explicitly (fail-closed on cipher error is a different layer from fail-closed when the key is not configured), and when triangulating with multiple agents, give each a different verification mechanism — one reads source, one greps, one writes a hypothetical failing test — so they do not verify the same surface three ways. A test-framework gap that structurally cannot fire the failure mode is itself a debt item: absence of a failing test is not evidence of correctness. This is the storage-shape-vs-semantics lesson the Process rule below embodies.

Mandatory Rules

  • NEVER dismiss test failures as transient. Investigate root cause. Only classify as edge case after thorough analysis.

  • Test results location: All test output (unit, integration, E2E) is saved to test-results/ at the repo root. Check test-results/e2e/artifacts/ for failure screenshots and error context, test-results/e2e/junit.xml for structured results. Always check these files when investigating failures.

  • Full test battery REQUIRED after code changes:

    1. cargo nextest run --workspace --lib

    2. cargo xtask dev restart (schema change) or cargo xtask dev reload (code-only)

    3. cargo nextest run --workspace

    4. cargo xtask e2e

Diagnosing load-dependent flakes

When a test passes in isolation but fails under cargo nextest run --workspace, do not accept "infra flake" as the answer — there is almost always a real bug. Reject the infra-flake hypothesis unless you can prove there is zero code-level mechanism; hardware capacity is not the bottleneck on a modern dev machine. Instead:

  • Instrument what the failure needs. Add reviewer-spec’d fields at the suspected callsites (username / attempt / elapsed / status; cache hit-miss-negative; upstream elapsed) — never raw secrets — so a captured failure ties symptom to mechanism.

  • Capture per-service logs from an actual failed run. Bracket each repro with timestamps and pull docker logs craig-<svc>-1 --since <start> --until <end> for every service; --since/--until retrieve whatever the daemon buffered even if a background tailer died. Look for cross-service signal — one bad pattern (rollback storm, lock contention, retry storm) touching multiple services is the smoking gun, not the single service whose test failed.

  • A shared precise duration is a lead, not noise. If every failure takes the same wall-clock time (e.g. exactly 30.1 s), that is a hard timeout being hit, not flakiness — find what hangs upstream.

  • Reproduce at the source before fixing. Reproduce the failure at the Postgres prompt (docker exec … psql … -c "…") — that is stronger evidence than reading the code and inferring the cause.

  • Ship the fix as a clean diff. Revert the diagnostic instrumentation before shipping; the diagnostic pass and the fix have different audit surfaces.

Multi-agent test-failure diagnosis

For a real (non-infra) test failure, a single diagnostic pass plus a hand-written fix is brittle — independent diagnoses frequently surface different real findings, and a synthesis over all of them can land a cleaner fix than any single diagnosis would (e.g. adopting a sibling handler’s established pattern instead of the literal fix each diagnosis proposed). The proven shape:

  1. Diagnose — N parallel agents with different framing angles (test-setup shape / handler-logic order / shared-fixture), each returning a structured diagnosis (root cause, proposed fix, fix target, risk notes).

  2. Fix — one synthesis agent receives all diagnoses, applies the chosen fix, and runs the verification commands.

  3. Verify — M parallel adversarial verifiers, each independently trying to refute the fix; an M-of-N majority passes.

Do not dispatch this for infra failures (CI runner disk OOM, devstack port collision, an e2e flake on admin pages) or for a single-handler fix whose failure mode is obvious from the error message.

Reconcile test-count deltas

When the test count changes between two runs, surface the delta explicitly — do not wave it away with "all green" if the numbers do not reconcile:

  1. List the tests removed (by name).

  2. List the tests added (by name).

  3. State the arithmetic: "removed N, added M, net +X — matches the observed delta of +X."

  4. If the numbers do not reconcile, find the discrepancy (skipped tests? #[ignore]? dependency gating?) before declaring done.

The same discipline applies to any quantitative claim — LOC counts, file counts, MR counts.

The never-easier checklist (contested-environment program)

Standing maintainer constraint (2026-08-17): tests never get easier. "The application needs to be able to handle a contested environment. Life is never 0% system usage." Every J-review in the contested-environment program (epic &83, ADR-067; disposition procedure in the arbitration runbook) runs this checklist, and it is the bar for any fix that touches a load-dependent or fault test anywhere:

  • Zero .config/nextest.toml serialization/envelope hunks — unless a test is semantically order-dependent AND the ordering is called out in the review. Adding a [test-groups] max-threads = 1 to make a flake pass is weakening the battery, not fixing it.

  • No timeout/envelope widening to make a leg pass. A per-leg setup ceiling (≤90s in this program) is a fixture ceiling — the budget for arranging the scenario — never an assertion-widening budget. If the assertion needs more time, the component is slow: that is the finding.

  • No assertion loosening, no [ignore] to dodge a real failure. The only legitimate [ignore] here is fault-layer gating (#[ignore = "requires devstack+fault"]), enforced by cargo xtask fault-preflight — it is a hard gate, not a skip (M9: a zero-fault green battery is a FAILURE).

  • Tests stay exposed to ambient load. A leg must not quiesce the box (kill neighbors, pin to a reserved core, sleep the world) to pass. It must be honest under the same contention the real system faces.

  • A ceiling breach or an added serialization is a blocking review finding — split the fixture or fix the slow path; a genuine exception needs a maintainer-approved amendment recorded in the plan.

The rationale is the arbitration instrument: a test made easier can no longer tell a real component defect from a bad fixture, which is the entire value the program exists to create.

Test Categories

Eight test categories. Most existing tests are Functional; the goal of the test-framework-hardening + platform-stab-2 work is to make Concurrency, Fault-injection, Recovery, and Invariant the second-most- common classes. The taxonomy describes how tests are written, not a quota per service.

Category Description

Functional

Single-fire request → single response → assert status + body. The dominant class. Catches "does this endpoint exist and return the right shape."

Invariant

Asserts a property that holds across many states or many calls. "Every successful POST /v1/cases/persons produces exactly one person.created outbox event with the same UUID as the response body." A failing invariant test usually points at a missing transactional bracket.

Concurrency

N parallel calls against the same endpoint or store function; assert no-double / no-drop / consistent-state. Use craig_test_lib::concurrent::*.

Fault-injection

Wraps a dependency in a wrapper that errors on demand, asserts the error path is reached. Use craig_test_lib::fault::*.

Recovery

Continues past a fault and asserts the system recovers — orphans reaped, in-flight jobs reclaimed, retries succeed.

Contract

End-to-end typed-DTO round-trip. Test client uses the same DTO type the service handler imports. Schema regressions fail compilation, not in production.

Property-based

proptest generators produce inputs; the test asserts a property holds for all of them. Useful for matchers, encoders, sanitizers. Available; not required.

Mutation

cargo mutants --smoke against craig-test-lib-tested modules; surviving mutations indicate uncaught real-world bug shapes. Available via cargo xtask mutants (Phase B). Not enforced as pre-push gate.

File-naming convention

The convention applies to net-new test files; existing tests are not mass-relocated.

Category Path convention

Functional

tests/api/<endpoint>.rs (status quo) or tests/<feature>.rs for cross-endpoint.

Invariant

tests/invariants/<invariant>.rs.

Concurrency

tests/concurrency/<scenario>.rs — filename names the property: idempotency_no_double_handler.rs, outbox_no_double_publish.rs.

Fault-injection

tests/fault/<failure_mode>.rs — filename names the upstream fault: rmq_publish_unavailable.rs, db_pool_timeout.rs.

Recovery

tests/recovery/<scenario>.rs — filename names what is recovered: attachment_orphan_reaped.rs.

Contract

typed-client tests live under crates/craig-test-lib/tests/typed/<svc>_client.rs.

Property-based

tests/properties/<property>.rs.

Mutation

No per-test file; configured at workspace level via .cargo/mutants.toml and run via cargo xtask mutants.

Process rule

Process rule (epic &21 / &22 onward): every concurrency-relevant fix, fault-tolerance fix, or platform-invariant fix MUST ship with at least one Concurrency, Fault-injection, Invariant, or Recovery test — in any combination — in the same MR as the production-code change. Failing-test-first (commit failing test, then commit fix in the same MR) is preferred but not mandated.

Reviewer responsibility: when reviewing an MR labeled concurrency, platform-invariant, durability, or any sub-label of those, the reviewer must explicitly confirm in CR that a test of one of the four categories exists. The MR is blocked until either the test is added or the author argues — and the reviewer accepts — that the change is structurally incapable of regressing the invariant (e.g. comment-only edits).

Linkage: this rule is the embodiment of the two-pass platform-invariant audit storage-shape-vs-semantics lesson. An audit that verifies "table exists" without the matching Concurrency / Recovery test does not constitute platform-invariant verification.

Quality budgets

Style rules in the coding-conventions standard §Style are hard rules — enforced via clippy / xtask lints from Plans H/I, not via declining-baseline budgets. The budget table below lists items that remain in budget form (numeric thresholds without a clippy rule today).

Item Threshold Enforcement

Max route module length

500 LOC

cargo xtask budget --metric route-loc (Phase B)

Max function length

40 LOC (hard)

Clippy too_many_lines + clippy.toml max_lines=40 per Plan H F-031; subagent triages "important function" overrides per the coding-conventions standard §Style <10% exception. The legacy 80-LOC budget is superseded.

Struct / impl block method count

≤16 (hard, excl. getters/setters/builders)

cargo xtask lints struct-method-count via syn::visit::Visit per Plan I F-054.

serde_json::Value in CRAIG-business-logic code

0 (hard) — partner-edge sites carry // PARTNER-EDGE-UNTYPED: see plans/partner-typed-schemas.adoc until Plan L F-064 retires the carve-out

Plan I F-037 sweep + rg 'serde_json::Value' services/ crates/

unwrap / expect / panic! / unreachable! / unimplemented! / todo! in non-test code

0 (hard) — only main may panic, and only via eprintln! + std::process::exit(1)

Clippy unwrap_used / expect_used / panic / todo / unimplemented / unreachable per Plan H F-031 + F-030 sweep.

let _ = result_expr / .ok(); / .unwrap_or_default() discards on Result types

0 silent (hard) — each site converts to ? propagation, tracing::warn!, or explicit // SILENT-OK: <reason>

Clippy let_underscore_must_use + ignored_unit_patterns per Plan H F-031 + F-051 sweep.

#[allow(…​)] count

reason-required (hard)

Clippy allow_attributes_without_reason = "warn" (already enforced workspace-wide).

New dependency duplicates

0

cargo tree -d

New untyped test clients

0 (post §D8 migration)

cargo xtask budget --metric untyped-clients

Mutation-test surviving-mutant ratio on craig-mq, craig-api/request_claims.rs, craig-store/validation.rs

0

cargo xtask mutants --smoke

Migration Naming Convention

  • Format: YYYYMMDDHHMMSS_descriptive_name.sql (sqlx timestamp-based)

  • Initial tables: YYYYMM01000000_create_<service>_tables.sql

  • Subsequent changes: YYYYMMDDHHMMSS_<action>_<objects>.sql (e.g., rename_document_url_to_object_key)

  • Cross-service batch updates share the same date prefix (e.g., 20260305100000_uuid_v7_defaults.sql across all services)

  • Each service has its own migrations/ directory under services/<service>/migrations/

Integration Test Organization

  • Pattern: tests/api.rs (module index) + tests/api/*.rs (per-endpoint files)

  • Module index declares submodules: #[path = "api/auth.rs"] mod auth;

  • Each test file follows the pattern:

    • Devstack gate: #[ignore = "requires devstack"] on the test function (see Devstack-gated tests below) — the legacy if !devstack_available().await { return; } early-return pattern is being swept out by Plan D F-024 and is now an anti-pattern for new tests

    • Setup: TestHarness::new().await → typed client (e.g., caseworker_cases_client(); the harness_clients! matrix emits every client under all seven roles — admin_/supervisor_/caseworker_/readonly_/county_director_/regional_director_/ state_office_ — #1082 added the readonly principal carol.reader, #1094 the three office-authority principals dana.county/rita.regional/sam.state)

    • Action: client method call (e.g., client.create_case(&body).await)

    • Assert: status code + response body validation

  • Naming: tests/api/{endpoint}.rs — e.g., cases.rs, referrals.rs, workflow.rs, auth.rs

Devstack-gated tests

Tests that depend on a running devstack (Postgres, RabbitMQ, Keycloak, garage) use the #[ignore = "requires devstack"] attribute, NOT a runtime devstack_available() early-return. This is canonical per Plan D F-024.

#[tokio::test]
#[ignore = "requires devstack"]
async fn create_report_persists_typed_envelope() {
    let harness = TestHarness::new().await;
    // ...
}

cargo nextest run --workspace shows these as (skipped) in default mode. cargo nextest run --workspace — --ignored runs them when devstack is up. The pre-push hook + CI run the --ignored variant so coverage is unchanged.

Reason strings distinguish the prerequisite class:

Reason Meaning

"requires devstack"

Full devstack (Postgres + RabbitMQ + Keycloak + garage)

"requires devstack postgres"

Postgres only (no broker/IdP needed)

"requires identity multibackend"

Devstack + the multibackend profile (Keycloak + authentik + ZITADEL + Kanidm)

"requires smoke devstack"

Devstack + a freshly-seeded smoke fixture

Plan D F-024 ships cargo xtask validate-no-silent-skips (via syn::visit::Visit) which fails the build if any [test] / [tokio::test] function body begins with the legacy if !<helper>() { return; } early-return pattern. After F-024 lands, the silent-skip pattern is mechanically prevented from re-entering the tree.

.gitlab-ci.yml — Scans, CI Tests, Docker Promote

The pre-push hook handles the full test battery locally — it is the sole functional-correctness gate. CI carries what needs server-side or scheduled execution: security/supply-chain scans, the no-devstack ci-tests subset (--lib --bins), secrets/docs policy jobs, Docker image promotion, Pages, and the scheduled pentest/perf/cluster jobs (four stages: scan → promote → deploy → triage).

workflow:rules restricts pipelines to four contexts: MR head pipelines (gate merges), main-branch pipelines (post-merge), tag pipelines (releases), and scheduled pipelines (pentest/perf/cluster/triage). Feature-branch pushes without an MR produce no pipeline — CI is not a place for per-push feedback; the pre-push hook already runs the full local battery. See #187 for the rationale (suppressing the first-push-before-MR race that used to create duplicate branch + MR pipelines on the same SHA).

Stage 1: scan (all branches, all parallel)

  • cargo-audit: advisory database check on Cargo.lock changes (allow_failure)

  • sast, secret_detection, dependency_scanning (GitLab security templates)

  • All jobs run on dhs-aws-autoscaler-docker.small

Stage 2: promote (main + tags only)

  • docker-promote: builds + pushes 9 Docker images to GitLab Container Registry

    • Tags: $CI_COMMIT_TAG (for releases) or $CI_COMMIT_SHORT_SHA (for main), plus latest

    • Uses BuildKit inline caching (--cache-from + BUILDKIT_INLINE_CACHE=1)

    • Runs on dhs-aws-autoscaler-docker.medium

Stage 3: deploy

  • review: Antora docs + mock UI as review app (non-main branches)

    • stop_review: manual, allow_failure: true

  • pages: Antora docs + mock UI to GitLab Pages (main branch)

  • All deploy jobs run on dhs-aws-autoscaler-docker.small

Security Scanning (OWASP ZAP + ffuf)

Setup

  • No local install required — runs via Docker (ghcr.io/zaproxy/zaproxy:stable, ghcr.io/ffuf/ffuf)

  • Configs: tests/security/zap-*.yaml (ZAP Automation Framework), tests/security/ffuf-wordlist.txt

  • Auth hook: tests/security/zap-auth-hook.js (injects Bearer token via ROPC for authenticated scans)

Running

  • Full scan (all 7 phases): cargo xtask security

  • Skip ZAP: cargo xtask security --skip-zap

  • Skip ffuf: cargo xtask security --skip-fuzz

  • Regression only (pre-push): cargo xtask security --skip-zap --skip-fuzz

  • One or more phases: cargo xtask security --phase <name> (repeat the flag to subset; omit to run all 7). Phases: public-api|public-web|authenticated-api|auth|injection|infra|fuzz

  • Phases: 1a public-api (ZAP), 1b public-web (ZAP), 1c authenticated-api (ZAP), 2 auth boundary (reqwest), 3 injection (reqwest), 4 infrastructure (reqwest), 5 path fuzzing (ffuf)

Pre-push Integration

Security regression tests (phases 2-4) run automatically in pre-push after k6 smoke. Takes ~5 seconds. Catches regressions in auth boundaries, injection handling, and security headers/CORS.

CSP value-assertion tests (#248)

The xtask security probe verifies CSP presence. Value-assertion tests live in each service’s integration suite and pin the exact Content-Security-Policy value to the documented spec in Security:

  • 7 backend services (craig-{rules,cases,placement,exchange,financial,reporting,security}/tests/api/health.rs::healthz_carries_documented_csp) — assert BACKEND_API_CSP

  • craig-intake/tests/api/health.rs — two tests, one per mode (API + standalone embedded UI)

  • craig-web/tests/csp.rs::login_carries_documented_csp — asserts WEB_BFF_CSP

All 11 call helpers in crates/craig-test-lib/src/csp.rs, which holds the single-source-of-truth pub const`s + `assert_*_csp panicking helpers + a self-test verifying strict-mode values stay free of 'unsafe-inline'/'unsafe-eval'. Drift between code, helper constants, or Security § Content-Security-Policy fails CI on the spot — the panic message points at the offending directive and tells the contributor which three places to update.

Results

  • HTML reports: test-results/security/zap-*.html

  • JSON reports: test-results/security/ffuf-*.json

  • Summary: test-results/security/ffuf-summary.txt

CI

  • pentest job in scan stage (allow_failure: true) — runs ZAP (phases 1a/1b/1c) + ffuf (phase 5) against a devstack spun up from the post-promote registry images.

  • Triggers: main pushes ($CI_COMMIT_BRANCH == "main") and scheduled runs ($CI_PIPELINE_SOURCE == "schedule"). MR pipelines deliberately do not trigger pentest — DAST stays out of per-MR cycle time while velocity is the gate. This is temporary; flip to add MR triggers once the velocity constraint lifts.

  • Schedule: configure via CI/CD > Schedules with a cron that fits your cadence (e.g. 0 7 * * * for daily 07:00 UTC).

  • Image source: each promoted CRAIG service in docker-compose.yml carries an image: ${CI_REGISTRY_IMAGE:-craig}/<service>:${CRAIG_IMAGE_TAG:-local} line alongside its build: block. CI sets the env vars to pull pre-built images; local dev uses the defaults so build behavior is unchanged.

  • Artifacts: test-results/security/ (HTML + JSON) retained for 30 days.

Performance Testing

Profiles (4)

  • smoke — 1 VU, ~29 requests across all 9 services, ~5 seconds. Pre-push gate runs this on every push. Quiescence fence (#1315): because the smoke asserts p95 thresholds and runs immediately after the Playwright e2e stage in the battery, cargo xtask perf first establishes quiescence for smoke-profile runs — (1) a bounded drain fence polling every stateful DB until unpublished event_outbox + pending event_inbox rows reach zero (90 s cap; a timeout WARNS and proceeds — the fence protects the measurement, it is not a gate), (2) ONE cluster-wide CHECKPOINT so the e2e write burst flushes inside the fence instead of under the smoke’s own writes (the #1319 checkpoint-fsync storm class produced 60× write-path transients pre-fence), (3) a 2 s settle. Root cause recorded on #1319; the fence is the accepted-devstack-behavior guard.

  • load — sustained realistic concurrency for ~5 minutes; codified SLO thresholds in tests/k6/helpers/thresholds.js:4-11 (READ p95 < 50ms, WRITE p95 < 200ms, error < 1%). CI gates this on main + schedule.

  • stress — ramps beyond expected production load to identify breaking points; 10–30 minutes. Manual / scheduled invocation only.

  • soak — moderate load sustained for 30+ minutes for memory-leak / connection-pool drift detection. Manual invocation only.

Running locally

  • Smoke (~5s): cargo xtask perf --profile smoke

  • Load (~5min): cargo xtask perf --profile load --save-baseline (writes JSON to test-results/k6/)

  • Single service: cargo xtask perf --service craig-rules

  • Named scenario: cargo xtask perf --scenario person-suggestions-bench

CI (#258)

  • perf-load job in scan stage (allow_failure: true initially) — runs cargo xtask perf --profile load --save-baseline against a devstack spun up from the post-promote registry images. k6 exits non-zero on threshold breach, so failed thresholds = failed job.

  • Triggers: main pushes + scheduled runs only. MR pipelines deliberately don’t trigger — load-test runtime (~5 min) doesn’t belong on per-MR cycle time while pre-push smoke is the local gate. Flip to add MR triggers after a clean week of baselines.

  • perf-stress job — same wiring with --profile stress. Manual invocation only (or via a dedicated schedule); not triggered on every main push.

  • Artifacts: test-results/k6/ JSON retained for 30 days for trend tracking.

Pre-push behavior unchanged

The pre-push hook runs only cargo xtask perf --profile smoke (~5s). Load/stress/soak stay out of pre-push because they need minutes of runtime that the local cycle can’t absorb.

Runner Tags

All CI jobs use DHS AWS autoscaler runners — no shared runners:

  • .small: security scans, fmt, docs generation

  • .medium: Docker builds

  • .large: (reserved for future use)

.githooks/pre-push

The pre-push hook is the primary quality gate. Sequential checks (all must pass):

  1. cargo xtask validate --skip-docker:

    • Repository visibility (Kerckhoffs enforcement)

    • Commit signing configuration

    • Mandatory docs (Tier 3) exist and have content

    • Tier 1 doc integrity

    • SPDX headers on all .rs files

    • JDM ruleset schema validation — walks rulesets/{jurisdiction}/*.json, deserializes to zen_engine::DecisionContent, constructs Decision. Parameter files (no nodes array) skipped.

    • gitleaks secret scan (soft-fail if not installed)

    • cargo deny check — license + advisory DB

    • cargo fmt --check --all

    • cargo clippy --workspace --locked — -D warnings

    • cargo build --workspace --locked

    • cargo nextest run --workspace --locked --profile integration

  2. cargo xtask e2e:

    • Reseeds the devstack first (since 2026-06-18): cargo xtask validate leaves the DB polluted (its nextest integration battery mutates/deletes seeded rows), so e2e runs cargo xtask dev reseed to restore a clean, deterministic state before asserting. dev reseed waits for service health + verifies the seed before returning, so there is no post-restart bring-up race, and .ports.env is reconciled after the reseed (it reserves fresh ephemeral ports).

    • Then regenerates tests/e2e/lib/seed.ts from CRAIG_SEED/CRAIG_FAMILIES (defaults: 42/12) and runs the full Playwright suite against the devstack.

    • --no-refresh skips the reseed + manifest regen — pass it for an iterative spec-debugging loop, or when you have already reseeded, to run against the current data without wiping it (NOTE: plain cargo xtask e2e now wipes local DB state by default).

    • County specs consume single-owner fixtures (#1151): the three county-project specs mutate their seed fixtures irreversibly — subsidies-county consumes fixture D (its ERR create refuses on rerun), subsidies-guardianship fixture E, and subsidies-onestep fixture G. Every manual rerun of any of them needs cargo xtask dev reseed first, and the guardianship serial arc reproduces only as a whole file (each step’s precondition is the prior step’s postcondition — a line-filtered run of a later step fails from stale state by construction). Each spec’s header carries the local pointer.

    • Custom seeds: CRAIG_SEED=99 CRAIG_FAMILIES=15 cargo xtask e2e — both the container reseed (compose passes CRAIG_SEED/CRAIG_FAMILIES through) and the manifest regen read the same env, so they stay consistent in one command.

    • Minimum families: 8 (pagination tests need ≥10 referrals)

  3. cargo xtask perf --profile smoke:

    • k6 smoke test via Docker container (grafana/k6)

    • 1 VU, 29 requests across all 9 services

    • Threshold assertions (p95 latency, error rate)

    • ~5 seconds total

Uses set -euo pipefail.

E2E Tests (Playwright)

Setup

  • Dockerfile: tests/e2e/Dockerfile (FROM playwright, COPY tests, npm ci)

  • Config: tests/e2e/playwright.config.ts

  • Run: cargo xtask e2e (checks health, starts if needed, auto-rebuilds Playwright container, runs tests)

Projects

The project matrix lives in tests/e2e/playwright.config.ts — it is the source of truth and grows too often to enumerate here (16 projects defined at 2026-08-25, of which 14 materialize in a default run; screenshots and degraded are env-gated). The shape: a setup project logs in each per-role browsing principal (jane.doe, bob.smith, admin, dana.county, carol.reader) and saves per-role storageState files; the per-role browsing projects (caseworker, bobsmith, admin, county, readonly) each load their role’s state and own a testMatch alternation of spec files; auth runs unauthenticated; auth-cookies covers the ADR-013 session lifecycle; config-guard fails any spec file no project’s testMatch claims (an orphaned spec cannot silently never run); and the special-purpose projects (screenshots, degraded, accessibility, intake-ui, intake-ui-shines, intake-ui-shines-tls, intake-ui-integrated) carry their own fixtures.

Single-service restart (used by auth-cookies spec)

cargo xtask dev restart-service <name> restarts one CRAIG application service (data preserved) and polls its healthcheck until green. Service name is validated against docker::CRAIG_SERVICES. The E2E helper restartCraigWebAndWait() in tests/e2e/lib/helpers.ts uses this to prove that cookie-encrypted sessions survive a web-tier restart.

Spec files and counts

Spec files live under tests/e2e/specs/ — the directory is the source of truth, and counts drift too fast to pin here (ls tests/e2e/specs/*.spec.ts | wc -l for the file count, cargo xtask e2e --list or the Playwright report for the run count). The suite runs single-worker, sequential (workers: 1, fullyParallel: false), so serial order is usable as a state proof within a file.

Seed Data

  • Generated by craig-seed — deterministic seed generator (tools/craig-seed/)

  • Pinned defaults: CRAIG_SEED=42, CRAIG_FAMILIES=12 (pinned in docker-compose.yml; CI inherits 12 through compose’s :-12 fallback — there is no separate .gitlab-ci.yml pin). The generator binary’s OWN default stays --families 9 — the reproducibility baseline pinned by the seed42/families9 byte-identity fixture. The 12-vs-9 split is intentional two-tier layering (#1006): operational dataset vs baseline corpus; the compose env line carries the authoritative comment.

  • SQL: auto-generated at container startup, seeded via craig-seed binary in Docker

  • TypeScript manifest: tests/e2e/lib/seed.ts — auto-generated via craig-seed --manifest

    • Indexed naming: case0_floyd, ref0_fulton, inv0, home0_bibb, partner0, icpc0, plan0

    • Person keys: camelCase(firstName, lastName) — e.g. keiraRolfson

    • Cross-entity refs: contacts have caseId, placements have caseId/childId, agreements have partnerId, ICPC has childId

  • With seed=42, families=12: 50 persons, 15 referrals, 12 investigations, 10 cases, 6 plans, 6 contacts, 5 court orders, 3 foster homes, 12 placements, 5 partners, 6 agreements, 1 ICPC, 5 rate tables, ~36 payments, 1 claiming record

  • Subsidy fixture (#1068/#1081, pinned ids …5000_00xx — seed-independent): 2 agreements (SG + NRSG), each with rev-1 terms (dues AS-OF-relative: renewal +11 months, paper +5), a pending→active interval chain, 2 open cycle-1 review slots (renewal_12mo + paper_6mo, due on the terms anchors — the shape materialize_anchor_slots seeds at activation), and a 3-month unit-month payment history ending at the as-of month

  • Default pagination: per_page=25 (BFF handler). Use ?per_page=10 in tests to force pagination with 15 referrals.

  • Case number format: {COUNTY}-{DATE}-{HASH} (e.g. FULTON-20260305-7AE1)

Page Object Model

  • tests/e2e/pages/ — CasesListPage, CaseDetailPage, FosterHomeDetailPage, RulesListPage, PaymentDetailPage, etc.

  • Helpers: tests/e2e/lib/helpers.ts (scrollAndWaitForFragment)

Do not reuse a spec-targeted CSS class on a new element

When adding a UI element to a page an e2e spec covers, do not reuse a CSS class that a spec targets via a generic page.locator('.classname'). Playwright strict mode fails a locator that resolves to more than one element — and a node counts even when it is x-show-hidden, because Alpine x-show only toggles visibility and leaves the element in the DOM. (x-if is safe: it removes the node.) A hidden new element sharing an existing class therefore produces a two-match strict-mode failure that breaks unrelated tests — and it surfaces only at the pre-push gate, after a full validate cycle, not where the new element was added.

Before reusing an existing class on a new element, grep the specs first:

grep -rn "locator('.classname')" tests/e2e/

If any spec targets it generically, give the new element a dedicated class. Mirror styling via the existing design tokens (var(--danger-) / var(--success-)), never raw hex — the intake_public_css_has_no_raw_hex_colors unit test rejects raw hex in the public CSS.

The Playwright container reaches the intake edge over http://host.docker.internal:<port>;, which is not a secure context, so browser crypto.subtle / crypto.randomUUID are unavailable — in-browser signing cannot be exercised in e2e. Prove the crypto round-trip at the HTTP level and keep the browser e2e crypto-free.

axe-core accessibility-audit configuration

The accessibility-audit spec drives @axe-core/playwright (AxeBuilder). Three configuration gotchas produce silently-wrong audits — a run that passes while asserting less than intended:

  • options() OVERRIDES withTags() / withRules() — they do not compose. Passing options() discards any tags or rules set via the separate builder methods. Put all configuration in a single options({ runOnly: { type: 'tag', values }, rules }) call rather than mixing the two styles.

  • The target-size rule ships enabled: false by default and must be explicitly enabled to run — include its tag in runOnly and set rules: { 'target-size': { enabled: true } }.

  • Under forced-colors / high-contrast, axe returns color-contrast as incomplete by design. That is expected behavior, not a failure — allowlist the incomplete color-contrast result when auditing a forced-colors media emulation rather than treating it as a violation.

nextest Profiles (.config/nextest.toml)

Profile Use Case Slow Timeout JUnit Output

default

Local dev (cargo nextest run)

60s

unit/results.xml

integration

The pre-push battery’s main stage (cargo xtask validate: ONE workspace run with --run-ignored=all)

60s

integration/results.xml

ci

CI --lib --bins run (devstack-bound tests are `#[ignore]`d and never execute here)

default

unit/results.xml

ci-integration

Reserved for CI devstack runs (no CI job currently invokes it — pre-push is the functional gate)

120s

integration/results.xml

All four profiles use fail-fast = false and test-threads = 8 (bounded so integration tests don’t overwhelm the Docker devstack services). JUnit paths are relative to the store dir + profile: the battery’s report lands at test-results/integration/integration/results.xml. The integration profile deliberately has NO retries (#1165: a retry masked a real contention failure — first-attempt red must red the battery).

Serialization groups ([test-groups])

Contention pairs are serialized by group, never masked by retries. Both groups carry their full charters as comments in .config/nextest.toml; membership is re-checkable with cargo nextest show-config test-groups --profile <p> --run-ignored=all (the selection flag matters — without it, `#[ignore]`d members are invisible).

Group Charter (abridged)

subsidy-generation-lease

The subsidy lifecycle/sweep tests contend on the two deployment-global run leases whose exclusivity IS the feature under test (#1068/#1096).

security-detection-scan

The craig-security tests that mutate shared detection state (scan-invoking — every scan is global — plus ack-invoking and the 8-way volley). Tertiary defense behind the #1172 partial-unique arbiter and the devstack scheduler disable (#1167 D4); pure validation/authz/EXPLAIN tests and the hermetic template-clone tests stay parallel.

The subsidy-generation tick is a standing legitimate contender (#1108)

Unlike the security scan scheduler (disabled on the devstack, #1167 D4), the financial subsidy-generation scheduler stays ALWAYS ON — its hourly
on-restart tick against the current business month is deliberately part of the environment under test (ADR-053; #1108 recorded the posture decision). The serial group excludes test-vs-test contention but NOT the tick, so generation-suite assertions must be tick-tolerant:

  • Never assert global run counters (report.generated == 0 fleet-wide) — the tick, or a concurrent suite’s fresh seed swept up by your own global run, legitimately mints rows. Scope every oracle to a test-owned child. (Exception: report.errors == 0 on a global run is safe ONLY because the fault-arming suites share the same serial group — a child-gated insert fault can never be armed while another group member’s global run sweeps.)

  • Month-window row oracles. Fixture agreements are open-ended, so the tick may mint a current-real-month row for ANY test child between seed and assert. Bound row counts to the months the test exercises (subsidy_generation_lifecycle::live_rows(pool, child, from, until)).

  • Sum generated + already_existed when a test’s fixed month can collide with the current real month — either party may mint first; per-month uniqueness is the property.

  • hold_generation_lease fences multi-step seeding that must not interleave with a tick (the reconcile-drain suites' pattern); 409 AlreadyRunning from a test’s own run is retried as the documented client semantic (run_for_child).

Template pre-warm (#1275, epic &78 M4)

Scratch databases clone from migrated TEMPLATE databases (craig_test_lib::template_db, #1162). Before #1275 the templates were built LAZILY by whichever test hit ensure_template first — under 8-way nextest contention the first battery run after a migration-set change was a rebuild storm and waiters died at the 120 s kill. Now cargo xtask test and validate’s test phase run template_db::prewarm_all BEFORE nextest: every base in the explicit WARMABLE_BASES registry (8 bases over 4 unique migration dirs) is warmed through THE SAME ensure_template path the tests use, the security group arriving as 1 leader apply + 4 Postgres file copies (each unique set applies once). Bounds: 3 min cold total, 1 min per-base warn, per-base timings printed.

Completeness is RUNTIME-ENFORCED, grep-free: ensure_template refuses a base not in WARMABLE_BASES, so a new call site fails its own first run with the registration instruction. Fingerprint parity (runtime dir migrator == embedded sqlx::migrate!()) is pinned per unique dir by the four prewarm_fingerprint_parity_runtime_vs_embedded tests — the warm can never silently produce a template name the tests don’t consume.

The test-plane Postgres instance (#1403)

The checkpoint-storm class. Every DROP DATABASE (and the file-copy CREATE DATABASE … TEMPLATE) forces an IMMEDIATE full checkpoint — cluster-wide, unskippable, serializing all Postgres I/O while it syncs. Run in parallel under nextest, the scratch-DB lifecycle suites (the 8 ADR-063 gate proofs, the keyed-harness template clones, retention/lease/invariant scratch DBs) produced hundreds of forced checkpoints per battery on the ONE devstack instance — starving the LIVE services the api suites talk to (observed 2026-08-11: single-row UPDATEs at 25 s, pool acquires past the 5 s timeout, eight craig-cases::api collateral failures; the full forensics live on #1403). Serialization via nextest test-groups was REJECTED as the fix: bounding test parallelism masks the performance surface instead of removing the artificial aggressor (the no-retries charter’s doctrine — see also #1404, which owns the product-side degradation posture the storm exposed).

The guard. ALL database-lifecycle churn targets the dedicated postgres-test compose service via craig_test_lib::postgres_test_plane_db_url — same devstack/postgres build as the shared instance (init.sql role + pg_trgm parity), but volume-less: the data dir is a tmpfs and the durability knobs are the test trio (fsync=off, synchronous_commit=off, full_page_writes=off), so a forced checkpoint costs memory writes, not fleet I/O. Everything on it is throwaway BY CONTRACT — templates rebuild on the next pre-warm after any recreate. Live-instance READS (asserting what a running service wrote: postgres_test_db_url("craig_<svc>")) stay on the shared instance; the resolver doc-comment carries the boundary. When the test-plane port is absent (a stack from before the service existed), the resolver falls back to the shared instance with a once-per-process warning — shared fate degrades loudly, never silently dials a dead port. Budgets: the shared instance’s 250-connection derivation lost its ~80 scratch pools to postgres-test (its own floor: 150) — both pinned in crates/craig-db/tests/connection_budget.rs.

Paused-Time Tests (tokio test-util)

First used by the #784 eval-timeout suite (services/craig-rules/src/engine.rs mod tests). Technique for testing time-dependent behavior with zero sleeps and zero flake surface:

  • Dev-dep tokio = { workspace = true, features = ["test-util"] } (the workspace feature set deliberately omits test-util from production builds).

  • #[tokio::test(start_paused = true)] + tokio::time::advance(…​) drive a virtual clock. Spawn the future under test first and yield_now().await before advancing, so its timers register; a sequential await would silently rely on auto-advance.

  • prop_assert! stringifies its condition into a format string — bind matches! results with { .. } patterns to a bool first.

  • Counter-seam oracle: when asserting "work was NOT done" (e.g. the eval loop skipping dead requests), thread an &AtomicUsize through an _inner variant of the loop — a "the live path still works" assert alone passes even without the guard. Production wraps the inner fn with a write-only local counter (one delegation line, no cfg(test) code).

Failure-path testing helpers

craig_test_lib::concurrent provides spawn-N-await-K helpers for concurrency and race tests. Reach for them whenever a test asserts a property under contention (atomic claim, idempotent retry, capacity bound, transition matrix).

Helper When to use

concurrent_fire_collect(n, builder)

Drop-in spawn-all/await-all. Panics propagate to the test thread.

concurrent_fire_collect_ordered(n, builder)

Need per-task latency, ordered-by-start, or panic-as-data instead of test-thread panic.

concurrent_fire_synchronized(n, builder)

Recommended default for race tests. All N start within one scheduler tick; caller cannot leak the sync point.

concurrent_fire_with_barrier(n, builder)

Need a post-prep, pre-race sync point inside the builder body (e.g. all N pre-fetch state, barrier, then race to UPDATE).

concurrent_fire_until_first_success(n, builder)

Any one Ok suffices (transition-allowed tests, JWT-mutation parametric tests).

concurrent_fire_first_n(n, k, builder)

Exactly K of N succeed (concurrent same-key claim races, capacity races).

 
// Recommended default — N-way race with no leakable sync point.
// (Same-key convert-class races carry the shared client_request_id in
// the request BODY per ADR-062 §B — the request_claims era.)
let report = concurrent_fire_synchronized(8, |i| async move {
    client.post("/v1/cases/persons", payload_with_request_id(i, key)).await
}).await;
assert_eq!(report.ok_count(), 8, "winner creates; every loser replays 2xx");

// Caller-controlled sync point.
let report = concurrent_fire_with_barrier(4, |i, bar| async move {
    let snapshot = client.read_state().await;   // pre-prep
    bar.wait().await;                           // race start
    client.commit(snapshot, i).await
}).await;

The original concurrent_fire(n, builder) is a deprecated alias for concurrent_fire_collect; new tests should call the new name.

Fault injection

craig_test_lib::fault provides the harness for failure-path tests — the codepath worked under contention; now exercise it under timing, intermittent failure, or back-pressure. Reach for these whenever the test asserts that a recovery / retry / DLX / live-response path actually fires.

Wrapper When to use

LatencyInjector::new(inner).with_pre_delay(d)

Force a real wall-clock stall on the wrapped callsite (e.g. winner stalls 5s so loser ceiling-409s).

IntermittentFlapInjector::new(inner, pattern)

Test retry-with-backoff: pattern [true, true, false] fails twice then recovers.

ConsumerBackpressureInjector::with_ack_delay(d) / .drop_every(n)

Slow-consumer / hung-slot simulation for outbox-depth alarms, DLX surface, /readyz flips.

FaultInjector is the common shape; Attempt records each invocation; ScenarioGuard’s `Drop asserts the wrapped callsite was actually reached (otherwise the test silently "passes" while exercising nothing). Suppress with .disarm() only when the test deliberately skipped the codepath.

let injector = LatencyInjector::new(real_store)
    .with_pre_delay(Duration::from_secs(6));
let report = concurrent_fire_synchronized(2, |_| async {
    client.post("/v1/cases/persons", payload_with_request_id(0, key)).await
}).await;
// Winner stalls inside its claimed tx; the loser's claim statement
// waits on the row lock and classifies as Replay after the commit.

Multi-replica testing

craig_test_lib::multi_replica::MultiReplicaCluster is the C20 (epic &83 / #1515) dedicated cluster rig: it runs the SELF-CONTAINED docker-compose.cluster.yml (craig-financial + its own postgres/rabbitmq/keycloak + the ADR-063 migrate gate; zero fixed host-port publishes; resource limits) under a unique craig-cluster-{run} project — never the devstack compose pair the original design reused. Use it only when a test needs two real OS processes contending on shared infra: kill legs, lease-release-by-socket-death, competing-consumer failover.

The NORMAL entry point is the stage, not the builder:

$ cargo xtask cluster-tests    # sweep → build → up --wait → legs → artifacts → down -v

The stage owns the lifecycle (signal-safe: Ctrl-C tears the project down) and exports CRAIG_CLUSTER_PROJECT; the kill legs in crates/craig-test-lib/tests/multi_replica.rs attach to it via MultiReplicaCluster::attach and skip loudly when the env is absent. OUT-OF-BAND ONLY: a weekly discriminated CI schedule (CLUSTER_TESTS=true), the cluster-tests MR label, or manual — never the per-push battery (plan F1/L10). Artifacts land in test-results/cluster/.

The self-managing path remains for local smoke use (opt-in via CRAIG_MULTI_REPLICA_SMOKE=1), now with a best-effort Drop teardown so a panicking run no longer leaks the project; a stale craig-cluster-* leak from a killed run is reaped by the next stage run’s sweep:

let mut cluster = MultiReplicaCluster::builder()
    .replica("craig-financial", 2)
    .build();
cluster.start().await?;   // up -d --wait: returning Ok IS readiness
let urls = cluster.replica_urls("craig-financial", 8005).await?;
cluster.stop().await?;

Tests that don’t need multi-replica should continue using the existing TestHarness against the global single-instance devstack. (The middleware-era §D3.8 multi-replica idempotency verification retired with the middleware in B2 #1194; the three C19 worker kill-leg oracles land on this rig as #1539.)

Named scenarios live in craig_test_lib::fault::scenarios::* — prefer them over open-coding (bp_scenarios::steady_50ms() over BackpressureScenario::Steady { ack_delay: Duration::from_millis(50) }). The base injectors land per the ADR-067 §D7 disposition (superseding the plan §D2.0a roster): the statement-boundary faults are served by pg_armer::PgFaultArmer (§D5); the publish-fault split — a stage_event StageFault (craig_mq::fault::arm_stage_fault) + a Publisher pre-send injector (craig_mq::fault::PreSendInjector) — lives in craig-mq under its default-off fault-injection feature (§D6, epic &83 C7), re-exported as craig_test_lib::fault::mq; CipherErrorInjector (C8) and ObjectStoreErrorInjector (C9) land under their host crates' features. RabbitDownInjector was RETIRED, not built (§D7) — one abstraction cannot express both a graceful AMQP close (L1) and a TCP reset (L2), which are the Toxiproxy legs.

Mock-server harness (spawn_for_test)

craig_mock_server::spawn_for_test() is the canonical ephemeral-port mock-server harness for partner integration tests, at tools/craig-mock-server/src/lib.rs.

pub async fn spawn_for_test() -> Result<MockServerHandle, std::io::Error>;

impl MockServerHandle {
    pub fn base_url(&self) -> &str;  // e.g. "http://127.0.0.1:54321"
}
impl Drop for MockServerHandle { /* aborts the spawned task */ }

It binds a TcpListener to 127.0.0.1:0 (so parallel test runs never collide on a port), spawns axum::serve in a tokio task, and returns a handle whose Drop aborts that task — the server lives until the test drops the handle. Canonical consumer:

let server = spawn_for_test().await.expect("spawn mock-server");
let url = format!("{}/<name>", server.base_url());
let ack = adapter.send(&url, sample_payload()).await.expect("round-trip");

The consumer crate needs craig-mock-server = { workspace = true } plus tokio (rt, macros, rt-multi-thread) in [dev-dependencies]. Each per-partner module exposes a routes() → Router<MockState> with a typed POST / (Json<Adapter::Outbound> in, Json<Adapter::Inbound> out) and HEAD|GET /health for test_connectivity() probes. It is strictly a test harness — never spawn it in a production path; the cargo run -p craig-mock-server bin target is for devstack only.

Property tests

proptest is a workspace dev-dep; opt in per crate via [dev-dependencies] proptest = { workspace = true }. Properties live under <crate>/tests/properties/<property>.rs with a tests/properties.rs module-index file declaring submodules via #[path = …​] mod …​, mirroring the existing tests/api/*.rs pattern.

Convention Rule

Path

<crate>/tests/properties/<property>.rs

Test name

prop_<invariant> — e.g. prop_sanitize_filename_invariants.

Cases

Default 256; override per property in code (not env var) so CI and local match.

Fail-first comment

Each property carries a // FAIL-FIRST: …​ comment naming the one-character production bug that demonstrates the property bites + the expected shrunk counter-example.

Runtime budget

// PROPTEST_BUDGET_MS: <N> comment so reviewers spot regressions.

Run the full property suite with:

cargo nextest run --workspace -E 'binary(properties)'

Adding a new property: write the prop_<invariant> body, deliberately break the production code with a one-character flip, watch proptest shrink to a minimal counter-example, then revert and land. The shrunk-input log line goes in the MR description.

Hash-pinned regression tests for deterministic byte emitters

When refactoring code that emits deterministic bytes — SQL output, IaC YAML, manifest files, JSON renderers — bake a SHA256 of the pre-refactor output (against a known input) into a regression test. Mechanical rewrites (e.g. converting many INSERT builders into impl SqlRow blocks) fail in subtle ways a compile check and input-side unit tests miss: a dropped space, a swapped column order, an off-by-one comma. Only an end-to-end byte comparison catches them, and the hash pin makes that comparison cheap (~10 LOC) with an actionable failure (a SHA256 mismatch naming the file).

How to apply:

  1. Before refactoring, capture the current output’s SHA256 for each well-defined input.

  2. Refactor the code.

  3. Re-run the test — it must pass with the same hashes.

Bless flow: pair the pin with a guarded second test that prints the current hashes when an env var is set (e.g. BLESS_SEED_HASHES=1). For the inevitable case where output bytes should change (a deliberate schema tweak), re-run with the env var, copy the printed values into the constants, and ship the MR with the intentional hash change documented.

Caveat — the input must be deterministic. Never pin a hash against output that depends on Utc::now(), random values, or HashMap iteration order. Fix the non-determinism first (preferred), or skip the pin for those outputs with a docstring explaining why. This pattern is faster than diff-based comparison (no fixture files to commit) and its failure mode is clear.

Mutation testing

cargo-mutants answers a question coverage cannot: do the existing tests actually catch wrong code, or do they just execute it? Mutation testing flips comparisons / deletes statements / replaces returns and re-runs the test suite per mutant. A surviving mutant means the test suite has a real semantic gap on that line.

Where What

Workspace config

.cargo/mutants.toml — timeout, nextest runner, exclude_globs for tests/examples/benches/main.rs.

Baseline

xtask/mutants-baseline.toml — committed list of accepted surviving mutants. Ships empty; populate after a clean smoke run.

Install

cargo install cargo-mutants --version "^25.0" --locked.

Smoke run

cargo xtask mutants --smoke (xtask wrapper) or cargo mutants directly.

Gated run (#1425)

cargo xtask mutants --gated [--package <name>]… [--shard k/n] — the DEVSTACK context: stack ensured ready, .ports.env reconciled, keyed seed verified, scratch templates pre-warmed, and the #[ignore = "requires devstack"] suites opted in via the trailing -- --run-ignored=all (the flag is what un-gates them; the env injection only guards .ports.env staleness). Defaults to craig-mq — the crate whose missed set is dominated by broker suites. LONG-RUNNING (hundreds of mutants × the 300 s timeout floor): plan an unattended window with no conflicting devstack lifecycle operations. --shard k/n slices the mutant set for parallel/overnight splits.

Residue scan (#1425)

Every wrapper run snapshots git status --porcelain before/after and FAILS on new entries (report-and-bail, never auto-delete) — an --in-place run must restore the tree exactly, and e.g. a proptest-regressions seed shrunk against MUTATED code is meaningless for the real implementation.

Catch variance (#1472)

Consecutive gated runs over a large service crate disagree by ~25–30 entries on which WEAKLY-COVERED mutants get "caught": incidental in-process code paths catch region mutants nondeterministically across runs. A caught verdict is therefore evidence only where a DELIBERATE killing test exists; treat each run’s fresh tail as candidate gaps to triage (the #1470→#1472→#1474 chain), never as regressions. Corollary for craig-cases: black-box TestHarness suites exercise the devstack CONTAINER’s unmutated binary — kills come only from inline units or the in-process keyed harness (#1162).

Critical crates

craig-api, craig-mq, craig-auth, craig-store, services/craig-cases/src/transitions/. Highest-leverage, not most-touched.

Per-mutant timeout

300s (kills hangs).

Total wall budget

<30 min for the full smoke. "Expect a coffee." The gated craig-mq run is multi-hour.

Mutation testing is never a pre-push gate — runtime cost is too high for a per-push cycle. Run quarterly or when adding tests to a known-thin module.

A future plan can tighten the budget by requiring new MRs not to increase the missed count (delta = 0) and requiring each reason = "tracked-issue-#NNN" line in the baseline to reference a real backlog issue. The first iteration enforces neither — "available, not gated."

Mutation-testing fixture rules

Three fixture-shape rules from the #1475 gated-run triage. Each names a mutant class that survives a fixture whose shape happens to make the mutant’s wrong answer indistinguishable from the right one — the fix is the fixture, not more tests.

  1. Assert totals over cardinalities ≠ 1. A 1-row fixture cannot distinguish the real count from a mutant’s constant (count → Ok(1) survives against total == 1). Seed ≥ 2 rows — or 0 where the emptiness is the property — whenever the assertion is a count or total.

  2. Behind wire-equivalent paths, assert at-rest side effects via raw sqlx. Where two outcomes share a status code (the deleted-download trap: a soft-deleted attachment 404s whether the guard ran or the blob is simply gone), the wire cannot see the guard mutant — read the row/blob state directly with a raw sqlx query and assert the at-rest truth.

  3. Search fixtures carry a non-matching decoy row. A suite that seeds ONLY rows matching the filter under test returns the identical page when the filter is no-op’d (a deleted bind arm) — the mutant survives. Seed one row that does NOT match and assert its id is absent from the filtered page (id-based, never bare counts; order the decoy so a no-op’d filter would place it on the truncated page).

Restart/replay testing

Light helpers for asserting "service crashed at point X, restarted, drove the recovery sweep, ended in the documented post-recovery state." Plan test-framework-hardening.adoc §D16. A heavyweight process-spawning harness was rejected per §D16 — these helpers run entirely in the test thread and avoid PID-racing flakiness.

Module: crates/craig-test-lib/src/restart.rs

Item Purpose

CrashPoint enum

6 named variants (one per platform-stab-2 recovery sweep) + Custom("label") escape hatch.

CrashHarness

Façade for adoption tests: wraps TestHarness for client construction, captures diagnostic artifacts, exposes crash_now(msg) for manual aborts and arm(injector, scenario) for fault-injection.

simulate_crash_after(point, builder)

Bare entry — FnOnce() → Future<Result<Option<Value>>>. For unit-test harness use and integration tests that arm their own injector.

simulate_crash_after_with_harness(point, builder)

Full entry — FnOnce(CrashHarness) → Future<Result<()>>. Constructs a live CrashHarness (requires devstack); used for the §D16.4 platform-stab-2-paired tests.

restart_service(svc_main)

Spawn svc_main() under the test runtime; returns a SimulatedRestart join handle.

assert_recovers_to(state, timeout)

Poll a state predicate until Ok(true) or timeout (100 ms interval).

assert_crash_fired(&crash)

Test-side guard against the no-crash control case.

Convention

Every platform-invariant fix that has a recovery sweep ships a paired restart.rs test as the failing-test-first artifact. Reviewer responsibility: a fix without a paired test fails the reviewer-discipline gate from the coding-conventions standard.

Example

let crash = simulate_crash_after_with_harness(CrashPoint::AttachmentPostPut, |h| async move {
    let client = h.harness.caseworker_cases_client().await?;
    let resp = client.upload_contact_attachment(case_id(), b"hello").await?;
    h.record_artifact(serde_json::json!({"row_id": resp.id}));
    Err(h.crash_now("post-put status promote skipped"))
}).await?;
assert_crash_fired(&crash)?;

restart_service(|| async { craig_cases::main_inner(test_args(), test_env()).await });

assert_recovers_to(
    || async {
        let artifact = crash.artifact.as_ref()
            .ok_or_else(|| anyhow!("crash artifact missing — pre-crash builder did not record one"))?;
        let row = query_attachment_row(artifact).await?;
        Ok(row.object_status == "present")
    },
    Duration::from_secs(30),
).await?;

Three of the §D16.2 fault injectors ship today (Latency, IntermittentFlap, ConsumerBackpressure) and are armable through CrashHarness::arm. The remaining four (FaultyAttachmentStore, FaultyOutboxStore, FaultyInboxStore, FaultySendJobStore) are introduced by platform-stab-2’s §D2.0 store-seam refactor; per-fix adoption tests ship as those seams land per §D16.4 / §D16.5.

Evil Input Corpus

Single source of truth for "what does a hostile input look like." Plan test-framework-hardening.adoc §D15.

Module

crates/craig-test-lib/src/evil/ — 12 categories:

Category Coverage

jwt

Single-axis JWT mutations (wrapper over craig_test_lib::jwt_mutation).

upload

Magic-byte-spoofed uploads (wrapper over craig_test_lib::upload_fixtures).

string

Overlong (1 KB / 1 MB / 4 MB), invalid UTF-8, NUL/CRLF/RTLO/ZWSP injection, NUL-truncated.

uuid

Wrong version, wrong length, all-zeros, dash scrambled, non-hex, embedded NUL.

json

Depth bombs, ref cycle, integer overflow, schema-stuffing, type-swap, missing required.

path

../, double-encoded, NUL-truncated, UNC, alternate data streams, backslash separator.

html

Script tags, onerror, javascript: URLs, base64 data URLs, double-decode escapes.

unicode

NFC/NFD pairs, homoglyphs, BiDi overrides, UTF-8/UTF-16 BOMs.

enum_value

Unknown variant, case mismatch, adjacent typos, trailing whitespace.

date

Leap-second, year-9999, negative year, fractional overflow, no-offset, invalid Feb-30.

multipart

Missing boundary, malformed Content-Type, oversized field count, empty filename, content-type spoofing.

signature

Truncated JWS, empty signature segment, alg=none, malformed garbage, alg-confusion.

Public API

craig_test_lib::evil::all_evil_cases()              // every case across every category
craig_test_lib::evil::evil_cases_for(EvilCategory)  // filter by category

// The macro emits 12 separate #[tokio::test] blocks, one per
// category (evil_jwt, evil_upload, …, evil_signature). Wrap in
// a `mod` to qualify the test names.
mod cases_post_evil {
    use craig_common::error::problem_types;
    use craig_test_lib::evil::EvilCategory;
    use craig_test_lib::parametric_evil_test;

    parametric_evil_test! {
        // The closure returns (HTTP status, raw response body). The body lets the
        // harness honour ExpectedRejection::ProblemType by comparing the RFC 9457
        // `type` member (see craig_test_lib::evil::rejection_matches), not just a
        // generic 4xx.
        endpoint: |case| async move {
            match client.send_with(case).await {
                Ok(r) => (r.status.as_u16(), r.raw),
                Err(_) => (0, String::new()),
            }
        },
        skip_categories: [EvilCategory::Multipart],
        // #954: tighten specific categories to the exact RFC-9457 `type` this
        // endpoint guarantees (verify each against the running service first).
        // The shared corpus default stays AnyClientError for every other
        // endpoint — only this invocation asserts the specific type.
        problem_types: [(EvilCategory::EnumValue, problem_types::INVALID_ENUM_VALUE)],
    }
}

Each emitted test is nextest-addressable (cargo nextest run -E 'test(evil_jwt)' filters to the JWT category). A regression in one category does not hide regressions in the other 11.

The optional problem_types list (#954) turns a category from a generic-4xx check into an assertive one: every case in a named category must return that exact RFC-9457 type, not merely a 4xx. It is a per-endpoint override — the shared EvilCase::expected is untouched, so no cross-service cascade. Verify before asserting: only pin a type you have confirmed against the running handler for every case in the category. A category that isn’t uniform (e.g. craig-rules String, where the 4 MiB overlong case is rejected by the HTTP body-limit layer as a 413 plaintext, never reaching garde) must stay on the AnyClientError default rather than be force-pinned.

Contracts

Test Purpose

crates/craig-test-lib/tests/evil_corpus_self.rs

Self-test: every category contributes ≥1 case, names unique, deterministic iteration, ≥60 cases total.

crates/craig-test-lib/tests/evil_corpus_global_contract.rs

Global ingestion-rejection contract: every evil case rejected as 4xx at the public-intake endpoint. No 500s, no 200s, no silent acceptance.

crates/craig-test-lib/tests/evil_corpus_macro_demo.rs

Compile-time demonstration of parametric_evil_test! macro.

Adoption

Per-service tests/api/evil_corpus.rs files are tracked as backlog (plan §D15.6). Each service exposes ~5 public-ingestion endpoints; landing the macro per service is mechanical fan-out once the contract module proves the pattern in evil_corpus_global_contract.rs.

Adding a new attack class is a single file edit — append a new EvilCase to the relevant category module — and every test using parametric_evil_test! or all_evil_cases() picks it up automatically.

Constraint tests

Hand-curated harness for verifying API-boundary rejections (UNIQUE conflicts, FK lookups, enum validation, duplicate-create handling). Plan test-framework-hardening.adoc §D14.

Convention

File Purpose

crates/craig-test-lib/src/constraints.rs

Helper module: ExpectedRejection enum + two assertion helpers (assert_constraint_violation_status for status-only, assert_constraint_violation_returns when an RFC 9457 type URL is known to exist).

services/<svc>/tests/constraints.rs

Module-index file declaring #[path = "constraints/<file>.rs"] mod <file>; per constraint.

services/<svc>/tests/constraints/<constraint>.rs

One test per constraint. Each #[tokio::test] async fn <constraint>_evil() asserts against a real handler, named with the _evil suffix per the §D13 coverage-matrix tagging convention.

Running

cargo nextest run --workspace -E 'binary(constraints)'   # all constraint tests
cargo xtask reliability                                  # also runs them via the failure-path filter

Tests carry #[ignore = "requires devstack"] so they’re skipped by default and run when devstack is up via cargo nextest run — --ignored (or via the pre-push hook + CI). Plan D F-024 converted the legacy if !devstack_available().await { return; } early-return pattern; see Devstack-gated tests in this doc for the canonical shape.

Auto-generation rejected

Per §D14: auto-deriving negative tests from migrations was rejected. CHECK constraints are arbitrary boolean expressions, UNIQUE constraints are tuples, FK constraints have ON DELETE semantics that vary, NOT NULL is the only constraint where the negative space is trivial. Even when SQL parsing succeeds, the interesting assertion is at the API boundary, not at SQLSTATE. Hand-curation is mandatory.

Adding a new constraint

  1. Identify the API endpoint that exercises the constraint.

  2. Write a test in services/<svc>/tests/constraints/<name>.rs following the same template as the existing tests (TestHarness, devstack_available skip, typed/raw client call, assert_constraint_violation_status).

  3. Add #[path = "constraints/<name>.rs"] mod <name>; to services/<svc>/tests/constraints.rs.

  4. Run cargo nextest run -p craig-<svc> --test constraints --run-ignored=all against a live devstack (1179 correction: the constraint tests are [ignore = "requires devstack"], so WITHOUT --run-ignored=all the bare selector runs zero of them and exits green — a false all-clear).

The full enumeration of constraints is tracked in the backlog (issue filed alongside this plan).

Invariant tests

Post-test-run platform-invariant sweeper. Plan test-framework-hardening.adoc §D17. Runs SQL queries from a per-service catalog against each service’s Postgres DB; non-zero rows are invariant violations. The full sweep (cargo xtask invariants) is report-only; since #860 the pre-push cargo xtask validate battery ALSO enforces a curated BLOCKING subset (cargo xtask invariants --gate) — the list is BLOCKING_INVARIANTS in xtask/src/cmd/invariants.rs, entries that hold by construction (single-writer stores, GENERATED columns, triggers), so a violation always means a bypass write.

Catalog: crates/craig-test-lib/sql/invariants/

Bucket Applies to Entries

craig-mq/

every stateful service DB (cross-service)

outbox_unpublished_beyond_grace, inbox_unprocessed_beyond_grace, request_claims_retention_overrun, the retention/collision watchdogs

craig-cases/

craig-cases only

(none yet — platform-stab-2 §D4 will add orphan_pending_attachments)

craig-exchange/

craig-exchange only

stuck_pending_transactions + 1 more

craig-financial/

craig-financial only

27 entries: the payment-integrity pair (ffp_amount_stale, day_count_gross_consistent), the ADR-052 subsidy-ledger family (subsidy_projection_matches_head, window/revision contiguity, party rules, subsidy_coverage_overlap, …), the #1069/#1070 flow watchers (subsidy_perdiem_same_month_overlap, sg_transfer_matches_activation, sg_no_payment_before_boundary, sg_residence_floor, review-sweep queues, …), and the four #1071/ADR-057 conversion guards — subsidy_active_missing_review_slot, subsidy_terms_cover_active, subsidy_closed_cohort_qualifying_date, subsidy_cutover_iff_imported — ALL FOUR in the blocking subset

craig-placement/ / craig-reporting/ / craig-rules/

per-service

occupancy pair / 1 / 1 — see the catalog dir (this table names only the load-bearing families; the dir is the source of truth)

Each .sql file’s first comments are TOML-fragment metadata (name, service, grace, remediation, severity); the body is a SELECT that must return zero rows.

Running

cargo xtask dev start                    # populates .ports.env
cargo xtask invariants                   # sweep all stateful services
cargo xtask invariants --service craig-cases
cargo xtask invariants --catalog path/to/alt-catalog

Discovery: tries CRAIG_<SVC>__DATABASE_URL first; falls back to postgres://craig:craig@localhost:$CRAIG_PORT_POSTGRES_5432/craig_<svc> against the running devstack.

Outcomes

Outcome Meaning

OK

Query returned zero rows

FAIL

Query returned ≥1 row — invariant violated; details + remediation pointer printed

SKIP (table missing)

Postgres SQLSTATE 42P01 — the invariant references a table this service hasn’t shipped yet (e.g. cross-service event_inbox invariant against a service whose inbox migration is pending). Not a failure; surfaces as a feature-rollout-gap signal

ERROR

Query failed for any other reason; treated as a failure

Drift contract

A self-test in crates/craig-test-lib/tests/invariants_catalog.rs enforces:

  • Every .sql file parses; metadata header is well-formed.

  • Every name is unique across the catalog.

  • The grace header literal and the SQL body’s INTERVAL '<n> <unit>' literal both match the corresponding craig_common::constants::*_GRACE Duration.

Adding a new time-based invariant requires touching three places in lock-step: a constant in craig_common::constants, the .sql file, and the mapping table in the self-test. Drift fails CI.

Local subcommands

The xtask runner registers stable entry points for the failure-path, property, contract, invariant, and mutation suites. None are part of the pre-push gate — they exist so contributors can run a single test category locally and so CI dashboards have a deterministic command to pin to.

Command Filter / scope Plan §ref

cargo xtask reliability [--filter <expr>]

nextest test(::concurrency::) | test(::recovery::) | test(::fault::) | test(::invariants::) | test(::properties::) | binary(properties); --filter ANDs an extra expression onto the default

§D11.2

cargo xtask invariants [--service <name>] [--catalog <path>]

SQL platform-invariant sweeper over per-service Postgres DBs; SKIP on missing tables; bail on violations. Replaces the Step-12 nextest-filter stub. See "Invariant tests" subsection above.

§D17

cargo xtask contracts [--service <name>]

nextest test(::typed::) (or test(::typed::<service>::))

§D11.4

cargo xtask mutants --smoke | --full

cargo-mutants over the 5 critical crates (smoke) or workspace (full); reads .cargo/mutants.toml

§D11.3

cargo xtask coverage-matrix [--output <path>]

Walks OpenAPI specs + scans test sources, emits target/coverage-matrix.md with route × axis grid. Live + cached modes.

§D13

cargo xtask quality-budgets --report [--write-lock] [--fail-on-regression]

Reports the B1–B8 budget table against the xtask/quality-budgets.lock baseline (thresholds/globs/markers are constants in xtask/src/cmd/quality_budgets.rs — #1002 removed the unread quality-budgets.toml). --fail-on-regression is the BLOCKING pre-push + CI gate (#522).

§D12

Each command emits an "opt-in; not part of pre-push gate" banner so contributors are not misled into wiring them into hooks. The two stubs print a STUB: line pointing at the step that finishes them.

Ruleset Discovery Test

  • File: services/craig-rules/tests/ruleset_discovery.rs

  • Runs as part of cargo nextest run --workspace (no devstack needed — loads JSON from disk)

  • Scans rulesets/{jurisdiction}/*.json at runtime

  • Validates: JSON parse, structure (nodes/edges), zen-engine compilation, evaluation with synthetic default input

  • Adding a new jurisdiction: just create rulesets/{jurisdiction}/*.json files — the discovery test picks them up automatically

Conversion-import test surface (#1071 / ADR-057)

API suites — three devstack-gated files under services/craig-financial/tests/api/, all members of the subsidy-generation-lease nextest serial group (max-threads = 1 in .config/nextest.toml, all profiles — the deployment-global advisory leases subsidy-gen / subsidy-sweep / the per-batch finalize lease are exclusivity UNDER TEST, so serializing the group exercises their real semantics instead of masking contention with retries):

Suite Pins

subsidy_import.rs

The batch lifecycle end-to-end; the full §Replay-semantics table incl. replay-after-mutation (finalize → native lifecycle action → resend → 200 with the ORIGINAL facts) and canonical equivalence (money 450.5 ≡ "450.50"); manifest-gate 409s carrying the four numbers (and never burning the batch); abort claim-freeing
post-finalize abort refusal; the 22.9 grace-boundary split + finalize world-drift rejection looping the batch; cross-service reference blockers + the terminal payment assessment; the authority matrix (non-admin / office refusal naming state_office / wire refusals); the AC1 closed-cohort create re-pin. (The knob-OFF contract — typed 403 naming CRAIG_FINANCIALSUBSIDY_IMPORTENABLED, GETs open — is pinned at the UNIT level in subsidy_imports.rs::gate_tests: the devstack runs gate-ON stack-wide, the ERR F4 precedent.)

subsidy_import_concurrent.rs

Concurrent same-reference stages → one winner, the loser answers from a FRESH transaction per the replay table (never a 500); concurrent finalizes → the lease 409; a native create racing finalize → the record rejects typed and the batch loops

subsidy_import_finalize_fault.rs

Kill mid-finalize after K records → a re-POST resumes exactly-once under the SAME batch id with correct totals (per-record transactions + the live-reference unique + the same-tx record flip)

The raw-SQL fixture builder — crates/craig-test-lib/src/subsidy_fixtures.rs (D10): the store primitives now REFUSE shapes tests still need as raw material (import_agreement_history refuses open-pending heads and open programs; the native creates enforce the full witness battery), so the builder writes identity + interval
terms (+ party) rows directly, deliberately bypassing every store invariant — honest about being a fixture, never a production path. Rows built imported: true carry provenance (source_system = "test") AND a payment_cutover_month (the cutover-iff-imported CHECK demands the pair; the default cutover is the first interval’s month-start so generation semantics are unchanged), and imported-shaped rows keep the F4 belt exemption so a pending fixture can still drive the REAL public transition writer. The caller owns invariant compliance — a nonsensical chain trips cargo xtask invariants exactly like production corruption.

Seed family F — the conversion-ledger fixture (tools/craig-seed/src/datagen.rs): a FINALIZED shines batch + one MATERIALIZED record + the imported rcs agreement it landed (first Active 2013-06-01, strictly before the 2014-01-01 closure; a closed 2013 terms revision + the open revision with CURRENT future dues; cycle-1 renewal AND paper slots; state-office attribution; pinned ids, no stream draws). Its payment_cutover_month = as_of + 1 month is the PRE-CUTOVER pin: the scheduled generation tick judges BeforeCutover and counts the skip (skipped_before_cutover = 1) instead of paying, so the deterministic zero-generation seed contract holds with NO payment rows — pre-cutover months belong to the legacy ledger by machine rule. The canonical payload/hash on the record row are fixture literals (the fixture-C hash posture), deliberately not real digests.

Edit this page · latest