ADR-064: Repo-Committed Encrypted Secrets (SOPS + age) for the Devstack/CI Loop
On this page
Status
Accepted (2026-08-10), implementation COMPLETE (epic &79 closed 2026-08-10; plan: Repo-Committed Encrypted Secrets). Drafted first (the draft-first path); the v1 program design was externally reviewed the same day (~40 findings, 13 stop-ship class) and this document records the redesigned architecture. Tool and CI-shape decisions were made by the maintainer on 2026-08-10 after a presented comparison.
Context
The secrets-management decision was deferred twice
(canopy-xtask-backports §Out of scope;
devstack-field-encryption §Out of
scope) while per-secret CI variables accreted: CRAIG_FIELD_ENCRYPTION_KEY (required by
three raw-compose scan jobs, never provisioned — #1031 sat open awaiting a maintainer),
GITLAB_TOKEN (triage), and every future secret implied one more maintainer-provisioned
variable. #1032 ordered one scheme before the pattern hardened.
The secret inventory is small and dev-shaped. The only committed real key material is
devstack/devstack-actor-keys.env — a deliberately public devstack test fixture
contributors' tests parse. The field-encryption key is per-box random (gitignored,
generate-once, ADR-048) with a KCV staleness marker; kanidm/ZITADEL bootstrap outputs are
gitignored and machine-regenerated. Nothing today needs cross-box sharing — the value of
the scheme is the rails: one bootstrap secret, rotation as a git commit, and a home for
every future dev/CI secret so variables stop accreting.
Two constraints shape everything:
-
The public tree (AGPL, Kerckhoffs baseline): whatever is committed must derive its security exclusively from keys held outside the repo, and identity-less contributors must keep a fully working devstack.
-
The root compose project hard-fails at load time while
devstack/field-key.envis missing (requiredenv_file; pinned by the CI comment at.gitlab-ci.yml:573-577— evencompose pullfails). The secrets engine therefore cannot live behind root compose, and reconciliation cannot be "decrypt produces the file compose needs to run."
Decision
CRAIG adopts SOPS + age with canopy’s containerized workflow ported and hardened.
Committed artifacts: .sops.yaml (age recipients) + secrets/dev.yaml (per-leaf
ENC[AES256_GCM…]). The engine is cargo xtask secrets driving pinned sops/age
binaries in a dedicated hardened container — never host installs, never root compose.
First migrant: the devstack field-encryption key. Production key delivery is explicitly
out of scope (the orchestrator’s secret store, per ADR-048 and the deployment guide) — a
public repo never carries production secrets, encrypted or not.
D1. Tool: SOPS-in-container (maintainer-chosen for canopy parity)
The maintainer weighted cross-project workflow parity (canopy runs the same SOPS+age workflow; one identity unlocks both stores; one muscle memory) above the pure-Rust alternative’s advantages. The runner-up and the honest trade-off are recorded in Alternatives.
D2. Engine isolation: no root compose, content-addressed hardened image
tools/craig-devtools/ — alpine base pinned by digest, sops/age apk versions
pinned; built by direct docker build; the local tag is content-addressed (hash of
the Dockerfile + inputs), so a stale or foreign :latest can never receive the key
(canopy’s exists-only image check is deliberately not ported). Invocation is direct
docker run: --network none, the identity mounted as an exact key-FILE read-only,
workspace read-only (read-write only for encrypt targets), UID:GID, non-root. A
considered-not-built alternative — a dedicated compose file — is recorded: it would
reintroduce a second compose project for zero gain over docker run.
D3. Trust: multi-recipient age, offline recovery, honest revocation
Recipients: the maintainer dev key + the CI key + an offline recovery key (a discarded
CI private line is not a recovery mechanism). Identity path ~/.config/sops/age/keys.txt
is deliberately shared with canopy — accepted with its threat model stated: one
compromised dev key opens both projects' stores, and rotation/recovery couple across
projects (ratified at gate G-A on #1031).
Two properties are stated because they are commonly misclaimed:
-
Git history is forever. Anyone who ever held a recipient key can decrypt every store revision they could see. Removing a recipient protects only future values — value rotation is the real revocation, and for devstack values it is cheap (the field key rotates via the automatic wipe + reseed).
-
sops updatekeysis not revocation. It rewraps the same data key to the new recipient set.remove-recipienttherefore runssops rotate(a NEW data key) and forces rotation of every plaintext the removed recipient could read.
D4. Store schema: exact and fail-closed
A neutral xtask/src/secrets_store.rs module (consumed by cmd/secrets and the
field-key reconciler) enforces the v1 schema exactly: shared.field_encryption_key, a
string containing valid base64 for 32 bytes — nothing else. Unknown sections, arrays,
non-string leaves, duplicates, and normalized-env-name collisions are typed errors.
.sops.yaml recipients must exactly equal the ciphertext’s sops.age[].recipient set
(parity is linted and checked). Future store entries are explicit schema and
trust-boundary changes, never drive-by additions.
Once the store exists: an identity-holding caller treats any failure — missing entry, malformed YAML, MAC failure, truncation, unsupported SOPS version, recipient/config drift, unknown sops error — as a hard failure. Only a positively classified no-identity case (and sops’s specific not-a-recipient error, for the joining-maintainer window) permits the contributor fallback; identity-present with store-absent is an integrity error.
D5. Field-key lifecycle: resolution first, adoption as a two-phase transition
ensure_field_key() resolves the desired key source FIRST (identity? store? local file?
marker?), then materializes once. A Docker-free provision-only guard (generate-once,
today’s behavior — this IS the permanent contributor path) remains the universal
pre-compose step for every root-compose caller.
Store reconciliation is a separate, lifecycle-only concern returning a typed
ReconcileOutcome { Unchanged, GeneratedFallback, PendingAdoption, Adopted }. Adopting a
store key is a locked, durable, two-phase transition: write a pending record
(.devstack/pending-adoption.v1.json — format version, store snapshot hashes, the new
key’s KCV, never key material) → stop containers and wipe the encrypted volumes (the
existing VolumeAction::Wipe machinery, driven by the outcome — never "overwrite the
file and let the KCV marker notice") → atomically publish the new field-key.env → write
the final marker → clear pending → bring-up + reseed. A crash in any window resumes
idempotently from the pending record; the key value is re-derived from the store.
Only mutating lifecycle entry points consume the transition (start/reseed/refresh/
restart-with-recreate/e2e/validate’s ensure-ready); read-only commands report drift and
change nothing. The warm path is gated by a versioned marker
(.devstack/secrets-adoption.v1.json: format version + secrets/dev.yaml and
.sops.yaml hashes + the adopted key’s KCV) — all three must match, so a replaced or
regenerated local key is never silently blessed. Rotation is thereby a git commit: edit
the store, commit; every identity-holding box adopts (wipe + reseed) on its next mutating
command, CI picks the new value up next run.
D6. CI: one bootstrap variable, pinned blocking least-privilege decryption
One masked + protected variable CRAIG_CI_AGE_KEY (a single AGE-SECRET-KEY-1… line —
GitLab masked variables must be single-line) replaces CRAIG_FIELD_ENCRYPTION_KEY
outright. It is consumed only inside the pinned-digest container: a blocking
secrets-policy job on protected refs runs the non-disclosing secrets check (decrypt
schema + recipient parity, prints no plaintext) — the only valid proof the CI keypair
works — and devstack/ci/write-field-key.sh decrypts the field key the same way, with
the output validated (single line, base64, 32 bytes), written atomically without
following symlinks, the prior file retained on failure, and the variable unset before any
unrelated tooling. ADR-048 §D5’s fail-loud property is preserved: an unset variable is a
hard error naming the fix. The old variable stays inert for a bounded rollback window and
is deleted only at gate G-C after one observed real scan-job success.
D7. Security-baseline interaction (escalated, not reinterpreted)
The synced security baseline says "no hardcoded secrets, API keys, or credentials in
source code or configuration files." A SOPS ENC[…] value derives its security
exclusively from the age key — the Kerckhoffs posture the baseline’s own rationale
demands — but a downstream ADR cannot silently supersede a synced standard. The wording
clarification is escalated upstream (claude-quickstart#27, filed 2026-08-10), and
committing the store is gated on the maintainer ratifying the amendment/exception (G-A on
#1031). The archived plan’s stricter reading
(devstack-field-encryption
§standing corrections) is superseded by that ratification, not by this document alone.
Alternatives considered
| Alternative | Verdict | Why |
|---|---|---|
SOPS + age, containerized (canopy’s workflow, hardened) |
Chosen |
Cross-project parity: identical command family and identity with canopy, one muscle
memory, one key. Per-value encryption keeps YAML key names greppable. Costs accepted with
eyes open: docker in the decrypt loop (mitigated by the D5 warm-path marker — zero
container runs when nothing changed), a Go-binary supply chain outside cargo’s audit
gates (mitigated by digest + version pinning), and ~100 lines of ported |
|
Runner-up |
Zero binaries, one supply chain (cargo-audit/deny), in-process battery tests, ~4
net-new deny-clean crates on an already-present substrate. Lost on parity: a second
workflow diverging from canopy’s, whole-file granularity, and no |
Host-installed rage/sops binaries |
Rejected |
Host prerequisite creep on every dev box — against the container-ops doctrine (cargo |
git-crypt |
Rejected |
Transparent-filter model makes plaintext-commit accidents easy (an unconfigured filter commits plaintext silently); effectively unmaintained; no per-value structure; no multi-recipient rotation story comparable to age. |
Vault / OpenBao / cloud KMS |
Rejected |
An online service dependency for an offline dev/CI need; contradicts "rotation is a git commit" and offline local dev; real operational cost. These remain exactly right for production — which is why production stays with the orchestrator’s secret store and out of this ADR. |
Consequences
Positive:
-
One bootstrap secret (
CRAIG_CI_AGE_KEY) instead of a variable per secret; #1031 collapses to provisioning gates. -
Rotation is a git commit with a crash-consistent, automatically-executed devstack transition; CI follows on its next run.
-
Every future dev/CI secret has a schema-gated home, lint-covered from the moment it lands (
secrets/**sops-mode lint; strict.sops.yamllint; blockingsecrets-policyCI job). -
Contributors are unaffected: no identity → the permanent generate-once local path, regression-pinned.
Negative / accepted:
-
Docker sits in the decrypt loop (warm-path-gated to changes only); the sops Go binary is a second supply chain, pinned by digest and version rather than audited by cargo.
-
Shared identity with canopy couples key compromise and rotation across projects (threat model ratified at G-A).
-
Git history retains every store revision forever; recipient removal without value rotation protects nothing (D3 —
remove-recipientforces both). -
Forks cannot rewrap the upstream store:
cargo xtask secrets fork-rebootstrap(destructive, explicit, refuses accidental overwrite) generates a fresh store and recipients — a pre-1.0 documented obligation, not a compatibility shim.
Open questions
-
ADR-048’s deferred production items (envelope-encryption/wrapped-DEK rotation, the compose-
secrets:production loader, lineage bootstrap onto an already-encrypted production database) stay deferred and are NOT claimed by this program — they remain named for a future production secrets phase. -
Whether the claude-quickstart template adopts this workflow (or the age-crate shape) is the template’s decision; the macro-feedback suggestion is filed either way (U8).