ADR-064: Repo-Committed Encrypted Secrets (SOPS + age) for the Devstack/CI Loop

On this page

Status

Accepted (2026-08-10), implementation COMPLETE (epic &79 closed 2026-08-10; plan: Repo-Committed Encrypted Secrets). Drafted first (the draft-first path); the v1 program design was externally reviewed the same day (~40 findings, 13 stop-ship class) and this document records the redesigned architecture. Tool and CI-shape decisions were made by the maintainer on 2026-08-10 after a presented comparison.

Context

The secrets-management decision was deferred twice (canopy-xtask-backports §Out of scope; devstack-field-encryption §Out of scope) while per-secret CI variables accreted: CRAIG_FIELD_ENCRYPTION_KEY (required by three raw-compose scan jobs, never provisioned — #1031 sat open awaiting a maintainer), GITLAB_TOKEN (triage), and every future secret implied one more maintainer-provisioned variable. #1032 ordered one scheme before the pattern hardened.

The secret inventory is small and dev-shaped. The only committed real key material is devstack/devstack-actor-keys.env — a deliberately public devstack test fixture contributors' tests parse. The field-encryption key is per-box random (gitignored, generate-once, ADR-048) with a KCV staleness marker; kanidm/ZITADEL bootstrap outputs are gitignored and machine-regenerated. Nothing today needs cross-box sharing — the value of the scheme is the rails: one bootstrap secret, rotation as a git commit, and a home for every future dev/CI secret so variables stop accreting.

Two constraints shape everything:

  1. The public tree (AGPL, Kerckhoffs baseline): whatever is committed must derive its security exclusively from keys held outside the repo, and identity-less contributors must keep a fully working devstack.

  2. The root compose project hard-fails at load time while devstack/field-key.env is missing (required env_file; pinned by the CI comment at .gitlab-ci.yml:573-577 — even compose pull fails). The secrets engine therefore cannot live behind root compose, and reconciliation cannot be "decrypt produces the file compose needs to run."

Decision

CRAIG adopts SOPS + age with canopy’s containerized workflow ported and hardened. Committed artifacts: .sops.yaml (age recipients) + secrets/dev.yaml (per-leaf ENC[AES256_GCM…]). The engine is cargo xtask secrets driving pinned sops/age binaries in a dedicated hardened container — never host installs, never root compose. First migrant: the devstack field-encryption key. Production key delivery is explicitly out of scope (the orchestrator’s secret store, per ADR-048 and the deployment guide) — a public repo never carries production secrets, encrypted or not.

D1. Tool: SOPS-in-container (maintainer-chosen for canopy parity)

The maintainer weighted cross-project workflow parity (canopy runs the same SOPS+age workflow; one identity unlocks both stores; one muscle memory) above the pure-Rust alternative’s advantages. The runner-up and the honest trade-off are recorded in Alternatives.

D2. Engine isolation: no root compose, content-addressed hardened image

tools/craig-devtools/ — alpine base pinned by digest, sops/age apk versions pinned; built by direct docker build; the local tag is content-addressed (hash of the Dockerfile + inputs), so a stale or foreign :latest can never receive the key (canopy’s exists-only image check is deliberately not ported). Invocation is direct docker run: --network none, the identity mounted as an exact key-FILE read-only, workspace read-only (read-write only for encrypt targets), UID:GID, non-root. A considered-not-built alternative — a dedicated compose file — is recorded: it would reintroduce a second compose project for zero gain over docker run.

D3. Trust: multi-recipient age, offline recovery, honest revocation

Recipients: the maintainer dev key + the CI key + an offline recovery key (a discarded CI private line is not a recovery mechanism). Identity path ~/.config/sops/age/keys.txt is deliberately shared with canopy — accepted with its threat model stated: one compromised dev key opens both projects' stores, and rotation/recovery couple across projects (ratified at gate G-A on #1031).

Two properties are stated because they are commonly misclaimed:

  • Git history is forever. Anyone who ever held a recipient key can decrypt every store revision they could see. Removing a recipient protects only future values — value rotation is the real revocation, and for devstack values it is cheap (the field key rotates via the automatic wipe + reseed).

  • sops updatekeys is not revocation. It rewraps the same data key to the new recipient set. remove-recipient therefore runs sops rotate (a NEW data key) and forces rotation of every plaintext the removed recipient could read.

D4. Store schema: exact and fail-closed

A neutral xtask/src/secrets_store.rs module (consumed by cmd/secrets and the field-key reconciler) enforces the v1 schema exactly: shared.field_encryption_key, a string containing valid base64 for 32 bytes — nothing else. Unknown sections, arrays, non-string leaves, duplicates, and normalized-env-name collisions are typed errors. .sops.yaml recipients must exactly equal the ciphertext’s sops.age[].recipient set (parity is linted and checked). Future store entries are explicit schema and trust-boundary changes, never drive-by additions.

Once the store exists: an identity-holding caller treats any failure — missing entry, malformed YAML, MAC failure, truncation, unsupported SOPS version, recipient/config drift, unknown sops error — as a hard failure. Only a positively classified no-identity case (and sops’s specific not-a-recipient error, for the joining-maintainer window) permits the contributor fallback; identity-present with store-absent is an integrity error.

D5. Field-key lifecycle: resolution first, adoption as a two-phase transition

ensure_field_key() resolves the desired key source FIRST (identity? store? local file? marker?), then materializes once. A Docker-free provision-only guard (generate-once, today’s behavior — this IS the permanent contributor path) remains the universal pre-compose step for every root-compose caller.

Store reconciliation is a separate, lifecycle-only concern returning a typed ReconcileOutcome { Unchanged, GeneratedFallback, PendingAdoption, Adopted }. Adopting a store key is a locked, durable, two-phase transition: write a pending record (.devstack/pending-adoption.v1.json — format version, store snapshot hashes, the new key’s KCV, never key material) → stop containers and wipe the encrypted volumes (the existing VolumeAction::Wipe machinery, driven by the outcome — never "overwrite the file and let the KCV marker notice") → atomically publish the new field-key.env → write the final marker → clear pending → bring-up + reseed. A crash in any window resumes idempotently from the pending record; the key value is re-derived from the store.

Only mutating lifecycle entry points consume the transition (start/reseed/refresh/ restart-with-recreate/e2e/validate’s ensure-ready); read-only commands report drift and change nothing. The warm path is gated by a versioned marker (.devstack/secrets-adoption.v1.json: format version + secrets/dev.yaml and .sops.yaml hashes + the adopted key’s KCV) — all three must match, so a replaced or regenerated local key is never silently blessed. Rotation is thereby a git commit: edit the store, commit; every identity-holding box adopts (wipe + reseed) on its next mutating command, CI picks the new value up next run.

D6. CI: one bootstrap variable, pinned blocking least-privilege decryption

One masked + protected variable CRAIG_CI_AGE_KEY (a single AGE-SECRET-KEY-1… line — GitLab masked variables must be single-line) replaces CRAIG_FIELD_ENCRYPTION_KEY outright. It is consumed only inside the pinned-digest container: a blocking secrets-policy job on protected refs runs the non-disclosing secrets check (decrypt
schema + recipient parity, prints no plaintext) — the only valid proof the CI keypair works — and devstack/ci/write-field-key.sh decrypts the field key the same way, with the output validated (single line, base64, 32 bytes), written atomically without following symlinks, the prior file retained on failure, and the variable unset before any unrelated tooling. ADR-048 §D5’s fail-loud property is preserved: an unset variable is a hard error naming the fix. The old variable stays inert for a bounded rollback window and is deleted only at gate G-C after one observed real scan-job success.

D7. Security-baseline interaction (escalated, not reinterpreted)

The synced security baseline says "no hardcoded secrets, API keys, or credentials in source code or configuration files." A SOPS ENC[…] value derives its security exclusively from the age key — the Kerckhoffs posture the baseline’s own rationale demands — but a downstream ADR cannot silently supersede a synced standard. The wording clarification is escalated upstream (claude-quickstart#27, filed 2026-08-10), and committing the store is gated on the maintainer ratifying the amendment/exception (G-A on #1031). The archived plan’s stricter reading (devstack-field-encryption §standing corrections) is superseded by that ratification, not by this document alone.

Alternatives considered

Alternative Verdict Why

SOPS + age, containerized (canopy’s workflow, hardened)

Chosen

Cross-project parity: identical command family and identity with canopy, one muscle memory, one key. Per-value encryption keeps YAML key names greppable. Costs accepted with eyes open: docker in the decrypt loop (mitigated by the D5 warm-path marker — zero container runs when nothing changed), a Go-binary supply chain outside cargo’s audit gates (mitigated by digest + version pinning), and ~100 lines of ported .sops.yaml editing (hardened to transactional).

age crate inside xtask (pure Rust, no external binaries)

Runner-up

Zero binaries, one supply chain (cargo-audit/deny), in-process battery tests, ~4 net-new deny-clean crates on an already-present substrate. Lost on parity: a second workflow diverging from canopy’s, whole-file granularity, and no sops edit/updatekeys/rotate ecosystem. The formats stay interoperable (age identities are shared), so this remains the recorded exit path if the container loop ever becomes the bottleneck.

Host-installed rage/sops binaries

Rejected

Host prerequisite creep on every dev box — against the container-ops doctrine (cargo
docker + git are the only host prerequisites).

git-crypt

Rejected

Transparent-filter model makes plaintext-commit accidents easy (an unconfigured filter commits plaintext silently); effectively unmaintained; no per-value structure; no multi-recipient rotation story comparable to age.

Vault / OpenBao / cloud KMS

Rejected

An online service dependency for an offline dev/CI need; contradicts "rotation is a git commit" and offline local dev; real operational cost. These remain exactly right for production — which is why production stays with the orchestrator’s secret store and out of this ADR.

Consequences

Positive:

  • One bootstrap secret (CRAIG_CI_AGE_KEY) instead of a variable per secret; #1031 collapses to provisioning gates.

  • Rotation is a git commit with a crash-consistent, automatically-executed devstack transition; CI follows on its next run.

  • Every future dev/CI secret has a schema-gated home, lint-covered from the moment it lands (secrets/** sops-mode lint; strict .sops.yaml lint; blocking secrets-policy CI job).

  • Contributors are unaffected: no identity → the permanent generate-once local path, regression-pinned.

Negative / accepted:

  • Docker sits in the decrypt loop (warm-path-gated to changes only); the sops Go binary is a second supply chain, pinned by digest and version rather than audited by cargo.

  • Shared identity with canopy couples key compromise and rotation across projects (threat model ratified at G-A).

  • Git history retains every store revision forever; recipient removal without value rotation protects nothing (D3 — remove-recipient forces both).

  • Forks cannot rewrap the upstream store: cargo xtask secrets fork-rebootstrap (destructive, explicit, refuses accidental overwrite) generates a fresh store and recipients — a pre-1.0 documented obligation, not a compatibility shim.

Open questions

  • ADR-048’s deferred production items (envelope-encryption/wrapped-DEK rotation, the compose-secrets: production loader, lineage bootstrap onto an already-encrypted production database) stay deferred and are NOT claimed by this program — they remain named for a future production secrets phase.

  • Whether the claude-quickstart template adopts this workflow (or the age-crate shape) is the template’s decision; the macro-feedback suggestion is filed either way (U8).

Edit this page · latest