Local Development

Add it to your own (gitignored, outside the repo) ~/.cargo/config.toml:

This works cleanly (unlike the reverse case above) precisely because the project contributes no rustflags of its own to join with: your personal config is the only source. Add the musl triples too if you cross-build for Alpine targets locally.

Alternatively, per-invocation via the env var (which replaces config rustflags wholesale rather than joining):

The Alpine/musl Docker builder stage (Dockerfile) forces mold unconditionally: apk add mold plus ENV RUSTFLAGS="-C link-arg=-fuse-ld=mold" on the shared chef base stage, so both the cargo chef cook layer and the final build see identical rustflags (a mismatch would invalidate the ADR-039 dependency cache). No local install or opt-in is needed to build images. This is also where the win matters most: CI on RUNNER_SMALL was the original out-of-memory pressure that motivated #1371.

dev refresh (Plan K F-043 / Step 4) is the daily-driver third lifecycle command, alongside reload and restart. It reads the same .devstack/*.sha256 markers that dev start writes, works out what actually changed, and dispatches the cheapest compose action that covers the drift. reload and restart stay available as escape hatches.

Every lifecycle verb that converges the stack rewrites the markers: dev start, dev refresh, dev reload, dev restart, and dev reseed. The blessing lives in bring_up’s success arm, after the health wait and seed verification, so a written marker always describes the running stack (#1405). `dev status shows what refresh would do without acting, and dev start --force bypasses the staleness check entirely. When the markers are absent, dev refresh no longer aborts (#955); it degrades to the conservative cached rebuild described above rather than forcing you back to a heavyweight dev start, which would reseed and churn the ephemeral ports.

Source: xtask/src/devstack_guard.rs::auto_refresh (refresh dispatch), xtask/src/devstack.rs::ensure_ready (start dispatch).

Seed integrity works in two layers. First, cargo xtask dev start/refresh (and the pre-push validate battery) poll the craig-seed one-shot to a clean exit; the check is state-aware, so a still-running seed is never misread as success. Second, the battery runs cargo xtask verify-seed --expect keyed to assert the seeded PII is genuinely encrypted at rest. The devstack has run field encryption end-to-end since epic &66 C7: the devstack/field-key.env key is mounted into craig-cases and craig-seed (only those two), both with ENCRYPTION_MODE: required. A key change or a missing .devstack/field-key.kcv marker forces the full-volume wipe and keyed reseed automatically, and since ADR-064 the key itself is adopted from the repo-committed encrypted store for recipients, so a committed rotation drives the same wipe-and-reseed on the next mutating command. See the quality-gates "Field-encryption seed verification gate" for the gate detail and how to run verify-seed manually.

Opt-in profile for the IdP-neutral integration tests. Brings up Authentik 2026.2.2 + ZITADEL 4.15.0 + a Redis dep alongside the existing devstack. Default cargo xtask dev start is unchanged — these containers are heavy and only needed when verifying the cargo xtask identity render output against real backends.

Per coding-conventions.md § Docker & DevStack, always activate through cargo xtask dev — never invoke docker compose --profile identity-multibackend directly. See docs/modules/ROOT/pages/plans/idp-multibackend-tests.adoc for the plan + devstack/authentik/README.md + devstack/zitadel/README.md for backend-specific notes (Authentik applies a declarative blueprint at boot; ZITADEL requires a one-time operator-side Terraform apply with a PAT extracted from the Console).

The devstack postgres runs with synchronous_commit at its default (on), plus modest shared-buffer and checkpoint easing. This applies to the devstack and CI only; production deployments own their own postgres tuning.

The history behind that default is worth knowing. #1125 originally set synchronous_commit=off after measuring one WAL fdatasync at 1.1 ms idle but 5,068 ms under concurrent I/O (k6 create_referral p95: 8 ms idle rising to 9.5 s loaded). The #1129 root-cause analysis then reframed that measurement: fsync on this box is ~718 µs/op even at load average 25 (pg_test_fsync, 2026-07-26), and the multi-second events are episodic host page-cache transients from co-tenant memory pressure, not a chronic function of load. #1148 restored the default: muting a durability semantic in a system-of-record test environment to absorb a minutes-per-day transient was the wrong trade, and it masked exactly the commit-path behavior the battery exists to observe. The transient class is absorbed where it belongs, in the e2e retry policy (retries: 1, unconditional since the #1129 analysis).

Connection ceiling (#1160): the devstack postgres also raises max_connections to 250. The stock 100 was under water for this stack — 8 services with pool size 10 each, around 14 detached advisory-lease sessions outside pool accounting, plus the api suites' live-DB read pools — which surfaced as 5-second acquire-timeout cliffs during the pre-push battery. Since #1403 the battery’s scratch-DB churn pools live on the dedicated postgres-test instance instead (with its own 150-connection floor). The full arithmetic is recorded next to the knob in docker-compose.yml; the floor is pinned by crates/craig-db/tests/connection_budget.rs, and the production sizing formula lives in the deployment guide’s "Database connection budget" section.

Optional host-side mitigation for btrfs boxes: Docker named volumes live under /var/lib/docker/volumes and inherit COW + compression, which amplifies database fsync cost. The standard exemption (the same technique commonly applied to build-artifact dirs like a cargo target/): make volumes/ a dedicated subvolume marked chattr +C (NoCOW — applies to newly created files only, so recreate volumes after flipping it; note NoCOW also disables btrfs data checksums for those files). This is a per-machine choice, not repo state.

Healthchecks and resource limits (#1135): every devstack healthcheck carries a start_period (probes during it don’t count toward unhealthy) sized for build-saturated cold starts — keycloak’s is 120 s because --import-realm under concurrent image-build I/O routinely exceeds its JVM-boot window. Compose-level CPU/memory limits were considered and REJECTED for now: dev boxes vary too widely for one set of numbers, limits would slow the builds that share the box, and the async-commit change above removed the sensitivity that made contention visible. Revisit if healthcheck flakes return.

Source: xtask/src/field_key.rs (generate-once provisioning, hardened load), xtask/src/field_key_reconcile.rs (store adoption: warm marker, pending record, fail-closed matrix), xtask/src/devstack.rs::consume_store_transition (lifecycle wiring), xtask/src/cmd/secrets.rs (the operator command family), xtask/src/devstack_guard.rs (KCV staleness). Decision record: ADR-064.

Each stateful service depends on its ADR-063 migration gate (service_completed_successfully) plus postgres, rabbitmq, and keycloak (service_healthy). The rules, cases, exchange, reporting, and security services also depend on garage, and everything but craig-rules depends on craig-rules itself, whose authz engine must be up first. craig-intake is the exception: as the stateless edge (ADR-017) it has no postgres, rabbitmq, or garage edges, and depends only on keycloak, craig-cases, and craig-security. docker-compose.yml is authoritative.

docker-compose.yml declares 16 build: blocks across those 13 targets — one target, craig-intake, is built for FOUR services: craig-intake, craig-intake-standalone, and craig-intake-standalone-shines share identical build args and image tag (one real build satisfies all three); craig-intake-standalone-shines-tls builds the same target with a distinct build arg (CRAIG_INTAKE_FEATURES=self-signed-tls) and a distinct image tag, so it needs its own build. All of them share the SAME builder stage, which compiles every workspace binary in ONE cargo chef cook
cargo build invocation — so the layer that step produces is a pure function of its build args, not of which final target you asked for. Those args boil down to three distinct recipes across the whole compose file:

docker compose build and up --build schedule all requested targets CONCURRENTLY by default, and BuildKit only commits a layer to its cache once the build that produced it finishes. So when two same-recipe targets start close together, the second starts before the first’s cargo chef cook and cargo build layer has landed in the cache, sees nothing to reuse, and redundantly re-runs the full-workspace cargo build itself. In the worst case this stacks up to N concurrent full-workspace builds, each with its own set of rustc and linker processes, instead of 1 real build plus N-1 cache hits. That multiplication lands at the exact moment memory is already highest (musl static-linking ring/aws-lc-sys for rustls is a heavy step on its own), and it is a confirmed source of OOM-killed builds on memory-constrained dev machines and the smaller self-hosted CI runners (RUNNER_SMALL).

The fix: xtask/src/docker.rs’s `build_devstack_images_serial (backing BUILD_SPECS, the table above encoded in Rust) issues one docker build --target <X> per distinct (target, build-args, image-tag) tuple, ONE AT A TIME, ordered so every recipe’s builds run back-to-back — group 1’s targets first, then group 2’s, then group 3’s. Every xtask call site that could trigger a docker compose build/up --build goes through this instead and drops --build from its subsequent up: cargo xtask dev (start/reload/restart/reseed), dev refresh’s auto-rebuild path, `cargo xtask migrate apply, and cargo xtask validate’s Docker build phase (CI-covered). Building via `docker compose directly (as the raw-compose CI jobs still do — out of scope for #1347) reintroduces the race.

Adding a 14th image target: add its BuildSpec entry to the group matching its (CARGO_PROFILE, CRAIG_WEB_FEATURES, CRAIG_INTAKE_FEATURES) build args — inside that group’s contiguous run, not appended after a different group — and do NOT reach for docker compose build/up --build as a shortcut; that is exactly the scheduling this section fixed.

Infra images are a separate, disjoint set. BUILD_SPECS only covers the 13 targets sharing the root cargo-chef Dockerfile — it deliberately excludes postgres, rabbitmq, keycloak, garage, and garage-init, which each declare build: as a bare string pointing at their own standalone devstack/<x>/Dockerfile (no shared layer, so no race to serialize around). Dropping --build from up without also covering these left them frozen at whatever image was last built — editing devstack/postgres/init.sql (or any other infra Dockerfile/config) silently had no effect, since up alone never rebuilds a service’s image. xtask/src/docker.rs’s `infra_build_services discovers this set dynamically (services whose build: value is a plain string, not a mapping — the same signal build_specs_match_docker_compose_yml uses to rule them OUT of BUILD_SPECS, so the two lists can never silently overlap or gap) and build_infra_images hands them to a single docker compose build <services> call — concurrent scheduling is fine here since there’s no cargo-chef layer to race. Every call site above runs it alongside (never instead of) build_devstack_images_serial. A name-based filter (e.g. a craig- prefix) was considered and rejected: mock-server is a BUILD_SPECS target with no craig- prefix, and compose’s own default image naming makes the unrelated craig-postgres/craig-rabbitmq/etc. images ALSO start with craig- — the heuristic is wrong in both directions on this exact compose file.