Plan: Repo-Committed Encrypted Secrets (SOPS + age)

On this page

Status

Step Description Status

U0

Docs first: this plan page + ADR-064 + nav entries + epic &79 with children #1378–#1386 + the #1031 re-scope to the G-A/G-B/G-C maintainer gates + the security-baseline amendment escalation (claude-quickstart#27). No code changes. Docs-size exception named (migration-gate M0 precedent). Relates #1032.

Done (2026-08-10) — this MR

G-A

Maintainer gate (#1031): dev + CI + offline-recovery keypairs; CRAIG_CI_AGE_KEY (masked + protected, single AGE-SECRET-KEY-1… line); the three PUBLIC keys posted; baseline amendment/exception ratified; shared-identity threat model acknowledged. Blocks U4. (The keygen commands land in U2, so the keypair boxes open after U2 merges; the ratification boxes are open immediately.)

Done (2026-08-10) — executed AGENT-side under explicit maintainer authorization (#1031 note 3668427845: devstack-only exercise of the mechanism; a production deployment would never let agent context near key material). Dev = the pre-existing canopy-shared identity (the detect-existing secrets init path exercised live); CI private line piped DIRECTLY to the masked+protected variable (never entered the transcript); recovery private line rests LOCAL-0600 outside the repo — the offline placement is the recorded production deviation. Ratification boxes ticked on the maintainer directive. The dev-decrypt proof box rides the U4 merge.

U1

Hardened devtools runner (tools/craig-devtools/ + xtask/src/devtools.rs): content-addressed image (digest-pinned alpine, version-pinned sops/age), direct docker run — never root compose (the root project hard-fails at load while devstack/field-key.env is missing), --network none, exact key-file ro mount, UID:GID; extra_env test injection; serialized docker-secrets nextest group; cache-invalidation test. Closes #1379.

Done (2026-08-10) — this MR (image pins: alpine:3.23 by digest, age 1.2.1-r15
sops 3.11.0-r6; the full flag contract asserted by unit tests over the docker-free argv builder, and network-none/uid/ro-workspace/tmpfs additionally probed live; the remaining live probes — ReadWriteDir write path, exact-file ro mount — ride U2’s secrets init tests, noted on 1380; the module carries a self-retiring [expect(dead_code)] until U2’s cmd::secrets consumes it)

U2

xtask/src/secrets_store.rs (exact v1 schema, typed errors, recipient parity, snapshot hashing) + core commands: init, init --for-ci (single-line paste instruction), decrypt, check (non-disclosing), bootstrap-store (stdin/tmpfs plaintext, overwrite refusal, recipient floor, atomic publish). Proptests for the parsers. Closes #1380.

Done (2026-08-10) — this MR (as-built deltas: the bootstrap plaintext rides STDIN only — no tmpfs file at all; encryption is POLICY-driven via sops --filename-override secrets/dev.yaml because sops refuses a config-less stdin path ("no matching creation rules"), with parity still pinned post-encrypt; all five banked U1 advisories folded — the rw-dir write probe and exact-file mount probe are live docker tests, real material is file-mount-only with extra_env documented tests-only, the interactive guard is now symmetric, and the context-dir inventory is pinned)

U3

Recipient management: edit (interactive), transactional add-recipient (staged both files, bech32 validation, decrypt-check, atomic commit, rollback), remove-recipient (= sops rotate + forced field-key rotation — updatekeys is not revocation), fork-rebootstrap. Closes #1381.

Done (2026-08-10) — this MR (as-built deltas: remove-recipient re-encrypts from scratch — a fresh encryption mints a fresh sops data key, so sops rotate is SUBSUMED while the content rotates in the same pass; staging is in-place-with-byte-exact-rollback rather than dual-rename because sops updatekeys matches creation rules against the real file path — and the staged POLICY must sit at the workspace root because sops resolves path_regex relative to the config file’s directory (bit live in tests); self-removal refused; the two U1 self-retiring expects retired on schedule)

U4

Commit .sops.yaml + secrets/dev.yaml (fresh key via bootstrap-store; public keys only) + yaml-rust2 sops-mode lint for secrets/ + strict .sops.yaml lint + the blocking secrets-policy CI job + CI path-rules + .dockerignore. Gated on G-A.** Closes #1382.

Done (2026-08-10) — this MR (as-built deltas: the two planned lints shipped as ONE sops-policy action — grammar, policy, and parity are interdependent, so one gate
one remediation surface; a parser-differential tripwire added beyond plan — the lint’s YAML view and the store tooling’s own recipient parser must agree or the gate fails closed; the committed .sops.yaml uses the inline age form the docker suite drove through real sops; the CI job is two-mode — keyed decrypt proof where the protected variable arrives, structural double-checks on MR refs, missing-key-on-protected-ref a hard error per the write-field-key.sh doctrine; the store walk is FILESYSTEM-side so decrypt residue under secrets/ fails; violation output names keys only, never leaf values)

G-B

CI-keypair proof (#1031): the secrets-policy job green on a protected ref with the real CRAIG_CI_AGE_KEY — the only valid proof (maintainer-side decrypt proves only the dev key). Blocks U7.

Done (2026-08-10) — job 15817105321 on main pipeline 2748471196 (the !1328 merge): "keyed decrypt proof OK: the CI identity decrypts the store; 32-byte key verified, value undisclosed" (37 s). Recorded on #1031.

U5

Field-key store reconciliation: resolution-first ensure_field_key, typed ReconcileOutcome, versioned marker (store hashes + adopted KCV), durable pending-record layer, fail-closed matrix, lifecycle-lock + snapshot semantics. Closes #1383.

Done (2026-08-10) — this MR (as-built deltas: the pending record lives at the WORKSPACE ROOT (.craig-pending-adoption.v1.json), not the plan’s .devstack/ path — recon showed tear_down wipes .devstack/ wholesale, which would destroy the record inside the very transition it sequences; the adoption marker DOES stay under .devstack/ because dying with a teardown is correct for a warm-path cache. The provisioning guard (ensure_field_key) is UNTOUCHED — the reconciler is a sibling module (field_key_reconcile), lock-free-inner per the verify-seed split pattern, that U6 wires into mutating entry points; GeneratedFallback defers to the existing generate-once guard rather than generating itself. Marker reads are stale-on-corrupt (trust cache); pending reads are refuse-on-corrupt (destructive-transition anchor). The member/outsider disambiguation is pinned LIVE against real sops output — a corrupted recipient envelope hard-fails while a non-recipient identity falls back)

U6

Two-phase adoption across every mutating lifecycle entry point (teardown before publish; crash-resume per window; read-only commands report drift only) + the BLOCKING in-battery rotation lifecycle test. Merge serialized after U5. Closes #1384.

Done (2026-08-10) — this MR (as-built deltas: the consult (consume_store_transition) sits at the ENTRY commands — ensure_ready, reseed, reload, restart, restart-service, refresh_preflight — never inside bring_up (the adoption executor’s own path; recursion-free by construction). The rotation proof is TWO-LAYERED rather than one scripted lifecycle run: (1) the executor takes an INJECTED teardown closure, so the in-battery blocking tests pin the load-bearing order (old key observed live at teardown time), the crash-resume window, and the converged end state directly; (2) the full live composition executes ON THIS BOX during the U6 battery itself — the local generated key predates the committed store, so validate’s ensure_ready consumes a REAL adoption (teardown → publish → cold start → reseed) mid-battery. restart-service refuses with a remedy post-adoption instead of proceeding (single-service bounce on a torn-down stack would be a false success); dev status reports drift docker-free)

U7

CI switchover: write-field-key.sh store-mode via CRAIG_CI_AGE_KEY in the pinned container (validated output, prior-file retention, input+output test seams, protected-ref constraint) + ADR-048 §D5 amendment + CHANGELOG pre-1.0 entry. Gated on G-B. Closes #1385.

Done (2026-08-10) — this MR (G-B was proven before pickup: the secrets-policy job decrypted the committed store with the real CRAIG_CI_AGE_KEY on main’s pipeline, #1031 note. As-built deltas: FOUR seams not two — CRAIG_SOPS_CONFIG and CRAIG_DEVTOOLS_CONTEXT joined the planned input/output pair, because the script mounts the policy explicitly and the docker proof drives a temp store from a repo checkout; the container runs --user "$(id -u):$(id -g)" — the devtools image’s default user cannot read the host’s 0600 mounts (bit live in the proof); the key temp file lives under $PWD, NOT /tmp — the J-review caught that the three real call sites run docker-in-docker, where -v sources resolve on the DAEMON side and the job container’s /tmp is invisible to the dind service (only /builds is shared), so a /tmp mount would have silently failed at every production call site while the local single-daemon proof passed; the dind-topology composition itself is exactly what G-C box 1 (one real scan-job store-mode success) exists to prove; the fail-closed inventory (store-missing / non-sops store / wrong-length value / symlinked target, each with byte-exact prior-file retention) is pinned by a second docker proof; the protected-ref constraint for schedules is STRUCTURAL — schedules target maintainer-configured refs and a manual unprotected run hard-errors in-script, unchanged doctrine, verified rather than re-implemented)

G-C

Old-variable retirement (#1031): one real scan-job success observed → delete CRAIG_FIELD_ENCRYPTION_KEY (expected no-op — never provisioned) → close #1031 with evidence.

Not started

U8

Close-out docs sweep (inventory below) + plan archive + epic close + macro-feedback filings. Closes #1386 and #1032.

Done (2026-08-10) — this MR (the full inventory: local-dev Secrets section with the workflow + hygiene runbooks, developer-guide command rows, configuration-reference, quality-gates sops-policy section, security.adoc, ADR-020 supersession note, .env.example, the secrets-yaml lint remediation now naming the real store workflow, the .dockerignore canary extended with the store surfaces, plan archived, epic closed; follow-up issues filed for the three recorded residuals)

Epic: &79
Issues: #1378 (U0), #1031 (G-A/G-B/G-C gates), #1379 (U1), #1380 (U2), #1381 (U3), #1382 (U4), #1383 (U5), #1384 (U6), #1385 (U7), #1386 (U8); program anchor #1032
Branches: feature/1032-u0-plan-adr, feature/1032-u1-devtools, feature/1032-u2-store-module, feature/1032-u3-recipient-mgmt, feature/1032-u4-store-commit, feature/1032-u5-reconcile, feature/1032-u6-lifecycle, feature/1032-u7-ci-switchover, feature/1032-u8-closeout
ADR: ADR-064

Context

The secrets-management decision was deferred twice (canopy-xtask-backports §Out of scope; devstack-field-encryption §Out of scope) while per-secret CI variables accreted; #1032 orders one scheme now. Target state, scope-honest: for authorized maintainers + CI, secrets live encrypted in the public tree, ONE bootstrap secret unlocks them, and rotation is a git commit. Identity-less contributors keep the permanent generate-once local path — that is the open-source contract, not a transition aid.

The maintainer ratified the two design forks on 2026-08-10: SOPS + age with canopy’s containerized workflow (parity weighted over the pure-Rust age-crate runner-up — comparison in ADR-064), and one CRAIG_CI_AGE_KEY variable with pinned-container decryption in CI. The v1 program design was externally reviewed the same day (~40 findings, 13 stop-ship class) and redesigned around the reviewer’s architecture: the engine runs independently of root compose; the key source is resolved before anything is materialized; adoption of a store key is a locked, durable, two-phase transition consumed only by mutating lifecycle entry points; every store failure under an identity is fail-closed; CI decryption is pinned, blocking, and least-privilege.

Two load-bearing facts (verified against HEAD b811d90e):

  1. The root compose project hard-fails at load time while devstack/field-key.env is missing (docker-compose.yml:369 cases + :1220 seed, required env_file; .gitlab-ci.yml:573-577 pins that even compose pull fails) — so the secrets engine must never sit behind root compose, and reconciliation cannot be "decrypt produces the file compose needs to load."

  2. write_key_file_hardened (xtask/src/field_key.rs:240-260) is noclobber-only — adoption needs an atomic overwrite sibling, and values (never file bytes) are what get compared.

Scope

In scope:

  • The hardened devtools runner + cargo xtask secrets family (init / init --for-ci / decrypt / check / edit / add-recipient / remove-recipient / bootstrap-store / fork-rebootstrap).

  • The committed store (.sops.yaml + secrets/dev.yaml) with exact v1 schema (shared.field_encryption_key only) + sops-mode and .sops.yaml lints + the blocking secrets-policy CI job.

  • Field-key migration end-to-end: resolution-first reconciliation, versioned marker, two-phase lifecycle adoption, CI write-field-key.sh switchover to CRAIG_CI_AGE_KEY.

  • ADR-064; the #1031 re-scope to gates; the baseline-amendment escalation; close-out docs.

Out of scope:

  • Production key delivery/rotation (orchestrator secret store — ADR-048’s deferred items stay deferred, named in ADR-064 §Open questions).

  • Migrating devstack-actor-keys.env (deliberately public devstack fixture — contributors' tests parse it), kanidm/ZITADEL bootstrap outputs (gitignored, machine-regenerated), or the Keycloak devstack-secret-* fixtures. Rationale: ADR-064 §Context.

  • The claude-quickstart template’s own adoption decision (suggestion filed; template’s call).

Design

The architecture is ADR-064 §Decision (D1–D7); this section carries only the implementation-shaping detail the ADR does not.

Reconcile decision matrix (U5 — pure StoreDecision, table-tested)

Identity Store Decrypt result Decision

absent

any

keep local, silent (contributor regression pin)

present

absent

integrity error once U4 lands (identity-holders must have a store)

present

present

value == local

keep local; write the versioned marker

present

present

value ≠ local

PendingAdoption (two-phase; loud log naming the wipe + reseed)

present

present

entry absent / malformed YAML / MAC failure / truncation / unsupported version / recipient drift / unknown error

HARD ERROR (fail-closed; a corrupt or partial store is never masked by a fallback)

present

present

sops not-a-recipient

WARN + keep local (the joining-maintainer window; no marker, retries next run)

HOME/XDG unset

present

treat as identity-absent here; hard error inside secrets subcommands

Warm path (zero container runs): marker version ok AND store hashes match AND local-key KCV == marker KCV. Any miss → full reconcile. The marker is written only on successful reconcile, so failures self-heal by retrying.

Two-phase adoption (U6)

PendingAdoption → durable .devstack/pending-adoption.v1.json (format version, store snapshot hashes, new-key KCV — never key material) → stop containers + wipe encrypted volumes (existing VolumeAction::Wipe machinery, driven by the outcome) → atomic overwrite-publish of field-key.env (new banner: "materialized from secrets/dev.yaml — edit the store, not this file") → final marker → clear pending → bring-up + reseed. Crash in any window resumes from the pending record (value re-derived from the store). Consumers = the mutating entry points only, each verified against its real path: decide_action ordering (xtask/src/devstack.rs:65-96), force + cold-start arms, reload (xtask/src/cmd/dev.rs:471), restart-service (xtask/src/devstack.rs:249), refresh marker-loss (xtask/src/devstack_guard.rs:699), containers-down. dev status reports drift only.

CI shape (U4 + U7)

  • secrets-policy (blocking, protected refs, pinned-digest container): the non-disclosing check with CRAIG_CI_AGE_KEY; path-rules ensure .sops.yaml
    secrets/** changes always run it.

  • write-field-key.sh store-mode: decrypt only inside the pinned container; output validated (single line, base64, 32 bytes); atomic no-symlink-follow write; prior file retained on failure; unset CRAIG_CI_AGE_KEY after the step; CRAIG_SECRETS_FILE input seam AND an output-path seam so local docker:27 proofs can never clobber a real dev key; scheduled scan jobs constrained to protected refs.

Test harness rules (all units)

Throwaway age keypairs only (generated via the runner, captured from stdout); temp stores under workspace-relative dirs; injected identity/store paths — no HOME/XDG mutation in-process (subprocess or path injection); the docker-secrets nextest group is serialized with isolated image tags/workspaces; docker-dependent tests are #[ignore = "requires docker"] and run under validate’s --run-ignored=all. No unit’s battery ever needs the maintainer identity or the real store.

Verification

  1. Per MR: full gate sweep + battery (pre-push authoritative) + J1–J8 review; this Status table updated in the same MR.

  2. Blocking proofs: the U6 in-battery rotation lifecycle test (store edit → mutating command → old volumes wiped → new key live → reseeded) and the G-B secrets-policy run with the real CI key. Nothing store-critical rests on allow_failure jobs or manual checkboxes except G-C’s deliberately-human variable retirement.

  3. The external reviewer’s test inventory is distributed per-unit (see the issue ACs): key replacement/deletion under a stale marker (U5); force / reload / restart-service / marker-loss / containers-down / crash-per-phase (U6); malformed-YAML / MAC / truncation / version / missing-entry / not-recipient classification (U2/U5); .sops.yaml-only change + recipient parity (U4); concurrent reconcile-vs-edit (U5); add-recipient rollback + rotate-on-remove (U3); bootstrap overwrite/crash-residue
    fork-rebootstrap (U2/U3); CI invalid/empty/wrong-length extraction with prior-file retention (U7); image cache invalidation (U1).

Documentation Updates

Rides each unit where noted; the U8 sweep covers the rest:

Edit this page · latest