Plan E: craig-identity Contract + Service Identity + On-Behalf-Of
On this page
- Status
- Context
- Scope
- Design
- D1. The craig-identity contract (operator-facing)
- D1.5. OIDC Discovery client (Step 2)
- D2. Wire shape (post-cutover)
- D3. Trust topology
- D4.
ClaimsAPI additions - D5. Receiving-side validation (auth middleware)
- D6. Outbound helpers
- D7. ActorTokenIssuer
- D8. Audit-log enrichment
- D9. xtask identity verify (conformance gate)
- D10. xtask identity render (reference IaC)
- Files Touched (aggregate)
- Verification
- Documentation Updates
- Risks
- Errata
- Step 14a deviations (2026-05-11)
- Step 13 deviations (2026-05-11)
- Step 12 deviations (2026-05-11)
- Step 11 deviations (2026-05-11)
- Step 10 deviations (2026-05-11)
- Step 9b deviations (2026-05-11)
- Step 9a deviations (2026-05-11)
- Step 8 deviations (2026-05-11)
- Step 7 deviations (2026-05-11)
- Step 6 deviations (2026-05-11)
- Step 5 deviations (2026-05-11)
- Step 4 deviations (2026-05-11)
- Step 3 deviations (2026-05-10)
- Step 2 deviations (2026-05-10)
- After this plan lands
Status
| Step | Description | Status |
|---|---|---|
1 |
Plan filing + ADR-028 (Service Identity + On-Behalf-Of, modeled on canopy ADR-019, 2026-05-08) in Proposed status. GitLab epic + 15 step issues. nav.adoc Active entry. CHANGELOG entry. 0 production code changes. |
Done (2026-05-09) — MR !235; Step 2 added 2026-05-09 per external review |
2 |
OIDC Discovery convergence in craig-auth + craig-web (added 2026-05-09 per external canopy/craig comparative review). Currently |
Done (pre-ADR-030) — Step 2: !248 — discovery client + JwksProvider + craig-web flows; Step 2b: this MR — destructive rename |
3 |
ADR-028 acceptance + |
Done (pre-ADR-030) — deviation: configurable |
4 |
|
Done (pre-ADR-030) — deviations: TOKEN_REFRESH_LEAD_TIME 30s reused from |
5 |
|
Done (pre-ADR-030) — deviations: module placed in |
6 |
Auth middleware: actor extraction. Update |
Done (pre-ADR-030) — deviations: per-CRAIG-service JWKS via |
7 |
Outbound helpers in * Internal services that don’t need user identity call |
Done (pre-ADR-030) — deviations: |
8 |
Bootstrap wiring. |
Done (pre-ADR-030) — deviations: no |
9 |
craig-web BFF flip. |
Done (pre-ADR-030) — Step 9a (provisioning + craig-web bootstrap wiring). Step 9b complete (ApiClient flip + caller updates). 9b deviations: the actual surface was ~91 call sites (not 25 — |
10 |
craig-intake flip. |
Done (pre-ADR-030) — deviation: partner-as-actor attribution deferred to Step 11. The actor JWT contract Step 5 designed ( |
11 |
Audit-log enrichment. |
Done (pre-ADR-030) — deviation: only audit insert path in craig-security today is the wildcard subscriber |
12 |
|
Done (pre-ADR-030) — deviations: (1 — authorization_code-with-PKCE worker probe deferred — the plan body called for an optional second probe verifying worker tokens, but that needs an interactive browser callback dance that doesn’t fit a CI/deploy-gate context; the |
s |
!s.is_empty())` so empty strings count as absent (the recurring shell-export-with-no-value foot-gun). (4) 9 inline unit tests cover the full matrix: discovery + JWKS + token success; no-test-principal-skips-token-probe; discovery unreachable; required endpoint missing; empty JWKS; empty access_token; non-Bearer token_type; case-insensitive |
13 |
* Backends without a render adapter (Okta, Entra, ForgeRock, PingFederate, custom) require operator-side configuration in whatever tooling the operator already uses. The contract definition above + the conformance test give them the spec they need. Templates use |
Done (pre-ADR-030) — deviations: (1 — module reshaped into a directory ( |
14 |
Devstack provisioning. |
Done (pre-ADR-030) — Step 14a (deviations: scope split into 14a + 14b. Step 14a (this MR — production-readiness MVP) shipped: (1) File-path env support: |
15 |
Issue: epic &28
Issues: #369–#382 (Steps 1, 3–15) + new issue for Step 2 (filed by the scope-expansion MR)
Branch prefix: feat/service-identity- / fix/service-identity- / chore/service-identity-
*Milestone: TBD
Context
Origin: canopy ADR-019 (2026-05-08, canopy) — sibling CCWIS-adjacent project — established canopy-identity as a contract (per-service client_credentials + X-Canopy-Actor on-behalf-of header signed by canopy-signing). Canopy ADR-019 explicitly cites craig ADR-026 as the worker-identity model and adopts it verbatim; craig adopts canopy’s service-identity layer in return to close the symmetric gap craig has on the receiving side.
craig’s specific gaps (audited 2026-05-09):
-
JWT pass-through in craig-web BFF (6 call sites) —
services/craig-web/src/api_client.rs:28,44,61,72,98,114forward the worker’s bearer to backend services. The audit_log row at the receiving service can’t distinguish "worker hit cases via web BFF" from "worker hit cases directly via CLI" — same anti-pattern canopy is fixing in canopy-web. -
Audit log lacks caller-service attribution —
services/craig-security/migrations/20260305100000_create_security_tables.sqlaudit_log hasuser_id, user_role, servicebut theservicecolumn is the target service, not the caller. Noactor_service, noactor_user_sub. Same gap canopy fixes in their ADR-019 Step 15. -
KeycloakServiceTokenis IdP-coupled —services/craig-intake/src/api/service_token.rs:55— the type name itself violates ADR-026’s IdP-neutrality stance, and it’s ROPC-based (username + password) rather thanclient_credentialsgrant. Embarrassing for a project whose ADR-026 just landed.
Why per-service client_credentials (not a shared service credential) — confirmed during plan-shaping (2026-05-09):
-
Auditability. The bearer JWT’s
azpclaim tells the receiving service which service called. With a single shared craig-internal credential, every receiver sees the sameazp— you can’t distinguish craig-web from craig-cli from craig-intake at the audit layer. Even WITH actor JWTs (telling you the worker), you still need per-serviceazpto distinguish callers. -
Blast radius on credential leak. Rotating one service’s secret vs. fleet rotation.
-
Permission scoping. craig-intake clearly needs to talk to cases + security only, not financial. Shared creds can’t express that.
-
Onboarding clarity.
xtask identity renderproduces N predictable client definitions; operators script it.
Why ship authentik + dex render adapters in Plan E (not just keycloak) — ADR-026’s IdP-neutrality stance demands it. Shipping with Keycloak-only xtask identity render defeats the ADR’s whole point. The marginal cost of two more tera templates is bounded; the operational signal of "we genuinely support multiple IdP backends" is not.
Differences from canopy ADR-019
-
Soft-cutover, not hard. Canopy’s plan is a hard cutover — internal services stop accepting worker JWTs. Craig’s call graph is smaller (~3 sync-call gaps vs. canopy’s ~30) and many craig endpoints are dual-purpose (BFF caller + direct CLI/API caller). Internal services keep accepting BOTH worker JWTs (so direct integration tests + CLI continue to work) AND service-identity tokens; craig-web + craig-intake start sending service-identity + X-Craig-Actor onward; audit log captures both. No breaking change to integration test harness.
-
9 service-account clients, not 13. Canopy needs 13 (more programs); craig has 8 backends + 1 BFF.
-
Reuse craig-signing keypair, distinct
audnamespace. Same as canopy. Theaud: craig-internal-actornamespace separates actor JWTs from the existing partner-JWS signing (aud: craig-intake).
Relationship to other plans
-
Plan A (multi-juris-authz): Steps 1-5 already shipped. Plan E builds on Plan A’s
ResourceType×Action+Claimsshape. Plan A Steps 6-14 continue in parallel; Plan E does not block them. -
Plan B (PII + races): Plan B Step 3 (reports-encryption) lands on the audit-log shape Plan E enriches. Plan E Step 10 must merge before Plan B Step 3.
-
Plan C (partner edge): Independent. Plan C’s partner-flow tightening (JWS replay, signer-key expiry, etc.) is orthogonal to internal service-identity.
-
Plan D (code-quality discipline): Independent.
Scope
In scope:
-
craig-identity contract definition (env vars, OIDC discovery requirements, token shape, claim path conventions).
-
ClaimsAPI extensions (is_service,service_id,require_service_caller,actor, configurableroles()lookup). -
OidcServiceToken(craig-auth) — OAuth2 client_credentials wrapper with refresh; replaces ROPC-basedKeycloakServiceToken. -
ActorTokenIssuer(craig-signing) — service-signed actor JWTs. -
Auth middleware actor-header extraction + craig-signing JWKS validation path.
-
Outbound helpers (
with_service_identity,with_actor). -
craig-web BFF + craig-intake flip from JWT pass-through to service identity.
-
Audit-log enrichment with caller-service + on-behalf-of-user.
-
cargo xtask identity verify— read-only conformance test. -
cargo xtask identity render --backend {keycloak,authentik,dex}— reference IaC fragment emission. -
cargo xtask dev identity provision— devstack-only Keycloak realm provisioning. -
devstack
craig-realm.jsonextension + devstack secrets entries. -
Documentation of the contract for deployers (
idp-integration.adocor equivalent).
Out of scope:
-
Production identity-backend lifecycle tooling. CRAIG does not own provisioning, secret rotation, or admin operations against deployer-owned IAM backends. Deployers use their existing IaC.
-
A craig-identity service container. craig-identity is a contract, not a service we ship. Operators deploy any compliant OIDC issuer.
-
mTLS between services (transport-layer; could layer later).
-
Token-binding (RFC 8473).
-
Per-call audience scoping.
-
Removing worker JWTs from worker-facing entry points (BFFs, partner intake).
-
Hard cutover (rejecting worker JWTs at internal endpoints) — soft-cutover allows direct CLI / integration test access.
Design
D1. The craig-identity contract (operator-facing)
| Variable | Meaning |
|---|---|
|
OIDC issuer URL. |
|
Optional in-cluster network locator for the issuer. |
|
Audience for service tokens. Default |
|
JSON path to roles array. Default |
|
Service-role marker. Default |
|
Per-service OAuth2 client_id (e.g. |
|
Per-service OAuth2 client_secret. |
Required OIDC discovery endpoints: authorization_endpoint, token_endpoint, jwks_uri. Optional: end_session_endpoint.
Required service token shape (issued via client_credentials): iss matches CRAIG_IDENTITY_ISSUER, stable sub, azp identifies the calling service, aud includes craig-internal-service, exp + iat standard, roles claim contains an entry starting with CRAIG_IDENTITY_SERVICE_ROLE_PREFIX (default service:craig-<name>).
Worker tokens (issued via authorization_code to a BFF client): iss matches CRAIG_IDENTITY_ISSUER, sub is the worker’s stable identifier, aud is the requesting BFF client (craig-ui, craig-api, etc.), preferred_username, email standard OIDC, roles claim contains worker roles (no service:* entries).
Claims deserialization tolerates either string or array aud (per ADR-021’s aud_or_vec deserializer). Roles are looked up via the configured path with a default-Keycloak fallback.
D1.5. OIDC Discovery client (Step 2)
Per the 2026-05-09 external review, ADR-026 IdP-neutrality is currently violated by code paths that build Keycloak URLs directly:
-
crates/craig-auth/src/jwks.rs:81constructs<issuer>/protocol/openid-connect/certsinstead of fetching the discoveredjwks_uri. -
services/craig-web/src/auth.rs:198constructs Keycloak token-endpoint paths the same way.
Step 2 introduces a discovery client in craig-auth:
// crates/craig-auth/src/oidc_discovery.rs (new)
pub struct OidcDiscovery {
pub issuer: String,
pub jwks_uri: String,
pub authorization_endpoint: String,
pub token_endpoint: String,
pub end_session_endpoint: Option<String>,
fetched_at: Instant,
}
impl OidcDiscovery {
pub async fn fetch(http: &reqwest::Client, issuer: &str) -> Result<Self> {
let url = format!("{issuer}/.well-known/openid-configuration");
let doc: DiscoveryDocument = http.get(&url).send().await?.error_for_status()?.json().await?;
// Validate required endpoints, normalize trailing slashes, etc.
...
}
pub fn is_stale(&self, ttl: Duration) -> bool { ... }
}
Cached at boot; refreshed on TTL expiry (default 1h) and on kid cache miss against the JWKS. Existing Keycloak-construction sites in jwks.rs:81 and auth.rs:198 switch to consume the discovered URLs.
Settings rename: oidc_issuer → oidc_issuer with one-release backwards-compat env-var alias CRAIG_<SVC>__OIDC_ISSUER. Documentation updates under docs/modules/ROOT/pages/idp-integration.adoc describe the migration for operators.
Bring craig’s stricter JWT validation along (aud/azp/scope/typ/nbf checks per crates/craig-auth/src/claims.rs:19 and jwks.rs:146); canopy’s parallel review noted these as missing on canopy’s side and recommended adoption.
D2. Wire shape (post-cutover)
Worker → craig-web (worker JWT issued by craig-identity):
POST /cases/{id}/actions/file-appeal HTTP/1.1
Authorization: Bearer eyJ... (worker JWT, iss=craig-identity-issuer, aud=craig-ui)
craig-web → craig-cases (service token issued by craig-identity + actor JWT signed by craig-web):
POST /v1/cases HTTP/1.1
Authorization: Bearer eyJ... (service token, iss=craig-identity-issuer, azp=craig-web, aud=craig-internal-service, roles=[service:craig-web])
X-Craig-Actor: eyJ... (craig-web-signed actor JWT, sub=jane.doe.uuid, aud=craig-internal-actor, exp=now+10m)
craig-cases audit row (post-Step 10):
INSERT INTO audit_log (..., actor_service, actor_user_sub, ...)
VALUES (..., 'craig-web', 'jane.doe.uuid', ...);
D3. Trust topology
-
craig-identity issuer (whatever the operator deploys) publishes JWKS for worker + service token validation.
-
craig-signing publishes its own JWKS for actor JWT validation. Distinct from craig-identity JWKS.
-
Each craig-* service trusts:
-
craig-identity JWKS for bearer-token validation (workers AND services, same JWKS, different
aud). -
craig-signing JWKS for actor-JWT validation (X-Craig-Actor header).
-
-
Cross-stack: each jurisdiction’s stack uses its own craig-identity issuer + craig-signing keys. Cross-stack tokens fail signature verification.
D4. Claims API additions
impl Claims {
pub fn is_service(&self) -> bool {
self.roles().iter().any(|r| r.starts_with(SERVICE_ROLE_PREFIX))
}
pub fn service_id(&self) -> Option<&str> {
self.roles().iter()
.find_map(|r| r.strip_prefix(SERVICE_ROLE_PREFIX))
.or(self.azp.as_deref())
}
pub fn require_service_caller(&self) -> Result<(), ApiError> {
if self.is_service() { Ok(()) } else { Err(ApiError::Forbidden) }
}
pub fn actor(&self) -> Option<&Claims> { self.actor.as_deref() }
pub fn roles(&self) -> &[String] { /* honors CRAIG_IDENTITY_ROLES_CLAIM_PATH */ }
}
Claims::actor is Option<Box<Claims>> with #[serde(skip)] so it never round-trips through JWT serialization — middleware lifts it in after validating X-Craig-Actor.
D5. Receiving-side validation (auth middleware)
-
Extract
Authorization: Bearer <token>. Validate against craig-identity’s JWKS (cached at startup, refreshed onkidcache miss). Validateaudmatches per-endpoint expectation:craig-internal-servicefor internal endpoints, the BFF client for worker-facing entry points. Validateexp,iss. -
Look up the configured roles claim path. If any entry starts with
CRAIG_IDENTITY_SERVICE_ROLE_PREFIX, markClaims::is_service() == true. -
If the request also carries
X-Craig-Actor, validate the actor JWT against craig-signing’s JWKS, checkaud == "craig-internal-actor", setclaims.actor = Some(Box::new(actor_claims)). -
Reject if either validation fails. Never silently drop.
D6. Outbound helpers
// crates/craig-auth/src/client_ext.rs
pub trait CraigClientExt {
fn with_service_identity(self, source: &OidcServiceToken) -> Self;
fn with_actor(self, actor: Option<&Claims>) -> Self;
}
impl CraigClientExt for reqwest::RequestBuilder {
fn with_service_identity(self, source: &OidcServiceToken) -> Self {
// Block on token fetch (cached; ~free in steady state).
let token = source.current_blocking().expect("service token unavailable");
self.bearer_auth(token)
}
fn with_actor(self, actor: Option<&Claims>) -> Self {
match actor {
Some(claims) => {
let actor_jwt = ACTOR_ISSUER.mint(claims).expect("actor JWT mint");
self.header("X-Craig-Actor", actor_jwt)
}
None => self,
}
}
}
Internal services that don’t need user identity call .with_service_identity(…) only; drainer publishes + scheduled jobs that don’t carry worker context skip .with_actor(…).
D7. ActorTokenIssuer
// crates/craig-signing/src/actor_token.rs
pub struct ActorTokenIssuer {
keypair: Arc<SigningKeypair>, // existing craig-signing keypair
service_name: String,
}
impl ActorTokenIssuer {
pub fn mint(&self, worker_claims: &Claims) -> Result<String, ActorTokenError> {
let now = chrono::Utc::now().timestamp();
let payload = json!({
"iss": &self.service_name,
"sub": worker_claims.sub,
"preferred_username": worker_claims.preferred_username,
"realm_access": { "roles": worker_claims.realm_access.roles },
"aud": "craig-internal-actor",
"exp": now + 600,
"iat": now,
"act_for": &self.service_name,
});
// Sign with ES256 using the existing craig-signing keypair.
let jwt = sign_es256(&self.keypair, &payload)?;
Ok(jwt)
}
}
The 10-minute TTL bounds replay risk; if a service-call chain takes longer than 10 minutes (rare), the calling service mints a new actor JWT at the next hop.
D8. Audit-log enrichment
Migration:
-- services/craig-security/migrations/<TS>_actor_columns.sql
ALTER TABLE audit_log
ADD COLUMN actor_service TEXT NOT NULL DEFAULT 'unknown',
ADD COLUMN actor_user_sub UUID;
Insert path (in services/craig-security/src/store/audit.rs::insert_audit_entry):
let actor_service = claims.service_id().unwrap_or("unknown");
let actor_user_sub = claims.actor.as_ref()
.map(|a| Uuid::parse_str(&a.sub).ok())
.flatten();
sqlx::query(
"INSERT INTO audit_log (..., actor_service, actor_user_sub, ...) \
VALUES (..., $N, $M, ...)"
)
.bind(actor_service)
.bind(actor_user_sub)
...
Pre-1.0 destructive migration is acceptable per the project’s pre-launch posture — no production audit data to preserve.
D9. xtask identity verify (conformance gate)
Behavior per Plan E Step 11:
-
Hits
<issuer>/.well-known/openid-configuration. Verifies the four required endpoints are present. -
Fetches
jwks_uri. Confirms it parses, contains usable signing keys. -
Performs a
client_credentialsgrant using a test service principal (configured in the same env vars as a real craig service). Validates the response token against the contract:iss,aud,exp, role claim shape, audience. -
(If test worker creds available) Performs an
authorization_codeflow with PKCE against a test worker principal. Validates the resulting token similarly. -
Reports per-check pass/fail with diagnostic detail.
cargo xtask identity verify --issuer <url> is safe to run anywhere — dev, CI, production deployment-gating. It does not mutate the backend.
D10. xtask identity render (reference IaC)
Pure code generation. Emits IaC fragments for backends craig provides adapters for. Templates use tera (already a transitive workspace dep).
-
cargo xtask identity render --backend keycloak [--out realm.json]— Keycloak realm definition with worker realm + 9 service-account clients (8 backends + craig-web BFF) + audience mappers +service:craig-*realm roles. -
cargo xtask identity render --backend authentik [--out blueprint.yaml]— Authentik blueprint covering the same shape. -
cargo xtask identity render --backend dex [--out dex.yaml]— Dex static-clients + connector-config template.
Per-backend snapshot tests pin the rendered output against committed fixtures so refactors of the template file surface as visible test diffs rather than silent drift.
Files Touched (aggregate)
| File | Step |
|---|---|
|
1 (new), 3 (Accepted) |
|
1 (this plan) |
|
2 (new — OIDC discovery client) |
|
2 (consume discovered jwks_uri instead of building Keycloak path) |
|
2 (consume discovered authorization/token endpoints) |
|
3 (extensions: is_service, service_id, require_service_caller, actor, roles) |
|
4 (new module — OidcServiceToken; replaces craig-intake’s KeycloakServiceToken AND craig-cases’s authz_bootstrap.rs ROPC) |
|
7 (new — with_service_identity, with_actor) |
|
6 (actor extraction + craig-signing JWKS path) |
|
4 (oauth2 = "5.0") |
|
5 (new — ActorTokenIssuer) |
|
8 (per-service OidcServiceToken + ActorTokenIssuer) |
|
4 (DELETE; moved to craig-auth) |
|
4 (DELETE; replaced by OidcServiceToken) |
|
10 (with_service_identity + with_actor) |
|
10 (same) |
|
9 (6 call sites flipped) |
|
11 (new — destructive ALTER) |
|
11 (insert path populates actor_service + actor_user_sub) |
|
12 (verify), 13 (render), 14 (dev provision) |
|
13 (new) |
|
13 (new) |
|
13 (new) |
|
14 (9 service-account clients + audience mappers + |
|
14 (9 new |
|
1, 2 (oidc_issuer rename + alias), 8 (env-var schema additions) |
|
2 (OIDC discovery contract for operators), 15 (per-backend setup notes; xtask verify/render/dev-provision usage) |
|
15 (new "craig-identity contract" section) |
|
15 (request-flow narrative update) |
|
per step |
Verification
After every step:
-
cargo xtask validate --skip-docker -
cargo nextest run --workspace— full suite green -
cargo xtask check-docs— Tier 1 docs untouched
Plan-wide:
-
OIDC discovery smoke (Step 2): boot any craig-* service against the devstack Keycloak; assert
oidc_discovery::fetchpopulates the cache + subsequent JWT validation uses the discoveredjwks_uri. Negative test: invalid issuer URL → boot bails with diagnostic. -
Service-token-acquisition smoke (devstack-gated, post-Step 4): each craig-* service successfully exchanges client_credentials for a token at startup.
-
Actor-propagation regression (devstack-gated, Steps 9-11 wired): post
/cases/{id}/…as workerjane.doevia craig-web; assert craig-cases’s audit row hasactor_service = "craig-web"ANDactor_user_sub = jane.doe.sub. -
xtask identity verifysmoke: runs against the devstack Keycloak realm (post-Step 14); asserts pass. -
xtask identity rendersnapshot tests: per-backend rendered output matches committed fixtures. -
Cross-stack token rejection: cross-stack tokens fail signature verification (negative test in Step 6).
-
End-to-end Playwright (Step 9): worker → craig-web → craig-cases flow succeeds; audit row carries correct
actor_service+actor_user_sub.
Documentation Updates
-
Per-step CHANGELOG entries (per step)
-
docs/modules/ROOT/pages/idp-integration.adoc— OIDC discovery contract (Step 2), conformance test usage (Step 12), reference IaC + per-backend setup notes (Step 13), Service Identity Keypair Provisioning + rotation procedure (Step 14a), tense sweep (Step 15) -
.claude/docs/security.md— service-identity keypair env-var modes added to Secret Management; audit-log attribution mention (Steps 11, 14a) -
.claude/docs/services.md— wildcard subscriber writesactor_service+actor_user_sub(Step 11) -
.claude/CLAUDE.mdPhase Status row — flipped to Complete with 14 step-MR refs + 1954 Rust tests (Step 15) -
CHANGELOG.adocper-step entries (Steps 2 through 14a) + Step 15 wrap-up -
Plan archive —
nav.adocPlan E moved Active → "(none)",archive.adocPlan E row added under Security & Compliance,roadmap.adocPlan E section flipped to past-tense (Step 15) -
ADR-028 status flipped Proposed → Accepted (Step 3)
-
ADR-027 forward-tense reference to "pending Plan E" updated (Step 15)
Risks
| Risk | Mitigation |
|---|---|
Step 8 craig-web flip introduces Authorization-header issues that integration tests don’t catch |
Soft-cutover: backends still accept worker JWTs, so direct CLI tests continue to pass; Playwright e2e validates the BFF flow in browser; phased rollout per service rather than big-bang flip |
Step 10 audit-log migration breaks existing audit-row reads |
Pre-1.0; reseed regenerates devstack data; production deploy hasn’t happened yet so no real audit data to migrate |
Step 11 |
Soft-fail in |
Step 13 dev provision touches the devstack Keycloak admin API; admin password leak risk |
Devstack admin creds remain in |
Per-service |
|
Plan B Step 3 (reports encryption) accidentally lands before Plan E Step 10 (audit columns) |
Plan B explicitly cross-references Plan E Step 10 in its sequencing constraints; both plans' status tables track the dependency |
Errata
Step-by-step deviations from the original plan that landed in their respective MRs.
Step 14a deviations (2026-05-11)
Step 14 split into Step 14a (production-readiness MVP) and Step 14b (Keycloak admin-API automation, deferred). Step 14a ships the production blockers; 14b is tracked as a successor if and when a concrete devstack-reproducibility need surfaces.
-
Step 14a delivers three sub-tasks:
-
File-path env support in
crates/craig-auth/src/keypair_env.rs. Two new env-var families:CRAIG_<SVC>__SIGNING_JWK_FILE(per-service signing-key file path) andCRAIG_PEER_JWKS_JSON_FILE(deployment-wide peer-JWKS file path). Loaders try inline first, fall back to file path, normalize empty strings to absent. Production deployments mount JWKs as Docker / Kubernetes secrets at a path so the private key never appears in/proc/<pid>/environor container introspection output. The deferred-from-Step-8 production hook closes. -
cargo xtask gen-actor-keys --kid-suffix <suffix> [--out <path>]— promotes the throwaway one-off Rust recipe (previously documented only in agent memory, now Service Identity keypair provisioning) to a tested + supported xtask subcommand. 9 ES256 keypairs + the peer-JWKS array, output matches the env-file shapekeypair_env.rsalready consumes. Production deployments run it, then write the same content to their secret-mount paths rather than reading the inline env-file. -
Rotation procedure docs in
docs/modules/ROOT/pages/idp-integration.adoc § Service Identity Keypair Provisioning. 6-step flow leveraging the peer-JWKS map’s ability to carry multiple entries periss(pre-rotation → advertise both kids → rolling-restart each service → drain old TTL → retire old kid → audit). Emergency-rotation variant skips the drain window for compromised-key scenarios. References Step 11’s audit-log columns as the verification surface.
-
-
Step 14b deferred:
cargo xtask dev identity provision(Keycloak admin-API integration) was on the original plan but doesn’t ship today. Rationale: (a) devstack already works via the committeddevstack/keycloak/craig-realm.json+devstack-actor-keys.env; (b) Step 13’scargo xtask identity render --backend keycloakproduces a fresh realm.json on demand, so realm provisioning is fundamentally an "import the JSON" step that’s a singlekc.sh importcommand rather than meaningful automation; (c) ADR-026 (IdP-neutrality) makes a Keycloak admin-API client a controlled exception that’s hard to justify when it’s only used for one-shot devstack bootstrap; (d) production deploys are explicitly NOT meant to run this — the plan body itself says "Production deployers do NOT run this command — they provision via their own IaC". The net is: Step 14b is convenience tooling for devstack-rotation drills, not a production-readiness blocker. If a concrete need surfaces (CI test of full rotation flow against a real Keycloak; testing realm-config migrations), it lands as a successor MR. Not blocking Plan E archive (Step 15). -
Committed devstack env-file header updated:
devstack/devstack-actor-keys.envnow references the newcargo xtask gen-actor-keyssubcommand for regeneration and points at the file-mount production alternative. Keys themselves are unchanged — no devstack reseed required to merge this MR. -
Tests: 14 new (4 keypair_env file-path: round-trip via tempfile + empty-file detection + env-var constant pinning + peer-JWKS roundtrip; 10 gen_actor_keys: env-prefix conversion, multi-dash service-name handling, 9-services-unique-kids invariant, kid-format pinning, peer-JWKS array shape, private/public JWK round-trip, env-file structure, production-warning header check, empty-kid-suffix rejection, rotation-suffix-changes-kid). 1954/1954 workspace pass.
-
Workspace deps:
p256 = "0.13"+rand_core = "0.6"added toxtask/Cargo.toml. Same crates + versions already incraig-auth+craig-signing— net workspace size unchanged.tempfileadded tocraig-auth/Cargo.tomldev-deps (the file-mount round-trip test needs a tempfile; tempfile is already a workspace dep).
Step 13 deviations (2026-05-11)
Step 13 ships cargo xtask identity render --backend keycloak|authentik|dex — reference IaC generator. Pure code generation, no network, no mutation. Three backend renderers in xtask/src/cmd/identity/render/.
-
Module reshape:
xtask/src/cmd/identity.rs(single file from Step 12) becamextask/src/cmd/identity/(directory) with siblingsmod.rs,verify.rs(from Step 12),model.rs,render.rs,render/keycloak.rs,render/authentik.rs,render/dex.rs. The reshape is purely organizational — each renderer is independently testable, the canonicalIdentityModellives inmodel.rs, andvalidate.rs’s soft-check wiring is updated from `super::identity::*tosuper::identity::verify::*. No public surface change. -
No template engine: the plan body said "Templates use
tera(already a transitive workspace dep)". In practice, Keycloak realm.json is structured JSON best built viaserde_json::json!(); Authentik blueprints and Dex configs are small YAML shapes (~150 lines each) that hand-rolledwriteln!covers cleanly. Pulling tera in for ~450 lines of stable string-building would add unjustified surface area. Zero new workspace dependencies — reusesserde_json+std::fmt::Writealready inxtask/Cargo.toml. -
Canonical
IdentityModel: 9 services + 6 worker roles + 3 dev users are constants inmodel.rs. Renderers project from the model rather than each carrying its own service list — drift between backends is impossible because they share the same source. The 3 dev-user UUIDs are pinned to match `.claude/CLAUDE.md’s Keycloak table (00000…001 = jane.doe, 002 = bob.smith, 003 = admin); a unit test enforces this so a typo in a future model edit breaks the build instead of breaking devstack reseeds silently. -
Two output modes via
--productionflag: devstack-friendly (3 dev users + callback URIs filled withhttp://localhost:8080/auth/callback+ variants) vs production-clean (no dev users, empty redirect URIs with an inline operator hint like# operator: fill with your BFF callback URI). The same--backend X --productioninvocation slots into deploy IaC; no--productionproduces something the devstack scripts can consume verbatim. -
Authentik blueprint covers the structural pieces:
oauth2provider+application+core.group+core.userentries. Audience mapping is left as an operator merge step in inline YAML comments because Authentik handlesaudviaproperty_mappingon the linked application, which the blueprint can’t bundle into a single self-contained entry (Authentik doesn’t have Keycloak’s "service-account client" shorthand). Comment block in each provider entry documents what to wire. -
Dex is connector-agnostic on purpose: Dex doesn’t manage users itself — it federates an upstream IdP (LDAP / OIDC / GitHub / passwordDB). The Dex render covers just the static-clients block + header comments pointing the operator at where to wire
connectors:+issuer:+storage:from their existing IaC. Worker-role membership comes from the upstream’s groups claim; CRAIG’s middleware reads it viaCRAIG_IDENTITY_ROLES_CLAIM_PATH. Audience enforcement is documented as theaudience=<svc>query param convention since Dex doesn’t have native aud-mapper config. -
Per-backend snapshot tests via structural assertions, not committed fixture files: the plan body said "per-backend snapshot tests pin the rendered output".
insta(the canonical snapshot crate) isn’t in workspace, and committing literal fixture files invites brittle whitespace/order diffs every time a comment changes. The 30 inline structural tests (11 Keycloak + 10 Authentik + 9 Dex) pin the meaningful invariants — "all 9 service clients emitted", "each carries an audience mapper", "production mode omits dev users", "realm name carries through" — without coupling to byte-level output. If a future regression actually matters, it’ll fail a meaningful test rather than a whitespace snapshot. -
Empty-realm + whitespace-realm guards:
render::validate_realm()rejects both at the top of each renderer. Catches the--realm ""foot-gun before it produces broken IaC. -
Live CLI smoke confirmed all 3 backends:
cargo run -p xtask --quiet — identity render --backend keycloakproduces 459 lines of valid Keycloak JSON;--backend authentikproduces a parseable Authentik blueprint;--backend dexproduces a Dex config with header docs + 11 static clients. 43 inline tests + workspace 1940/1940 green.
Step 12 deviations (2026-05-11)
Step 12 ships cargo xtask identity verify as a read-only OIDC conformance gate. New module xtask/src/cmd/identity.rs (385 lines incl. 9 inline tests). Wired into cargo xtask validate at [6c/14] as a soft check.
-
3-variant outcome:
IdentityVerifyOutcome::{AllPass, Failures(Vec<String>), Inconclusive(String)}. The third variant exists sovalidate.rs::check_identity_softcan downgrade transient or local-dev unreachability ("CRAIG_IDENTITY_ISSUER not set", "discovery doc unreachable: connection refused") to a SKIP / warning, while real contract failures (missingtoken_endpoint, empty JWKS, non-Bearertoken_type) still surface as red FAIL lines. Without the distinction the pre-push gate either over-bails on offline devs or under-reports real misconfigurations. -
PKCE worker probe deferred: the plan body listed an optional 4th check ("(if test worker creds available) performs an authorization_code flow with PKCE against a test worker principal"). That probe requires either an interactive browser callback or a headless authorization-server adapter — neither fits a single-binary CI gate. Plan E’s core contract is the service-identity (
client_credentials) flow; the worker-token flow is already covered end-to-end by E2E Playwright. Successor MR can add--worker-onlyif a deploy-time worker-token regression slips past Playwright. -
Probe-injection shape:
run_with_probes(issuer, client_id, client_secret, discovery_probe, jwks_probe, token_probe)takes threeFnOnceclosures so tests drive each network step without making real HTTP calls. Mirrors the patterncheck_docs::check_docs_withestablished earlier in xtask. Public functionsfetch_discovery/fetch_jwks/request_client_credentials_tokenare the real-network probes; bothrun()(CLI) andcheck_identity_soft()(validate.rs) wire them in via the same client. -
Env-var precedence with empty-string normalization:
--flag → CRAIG_IDENTITY_*lookup chain plus.filter(|s| !s.is_empty()). Catches the recurringCRAIG_IDENTITY_ISSUER=shell-export-with-no-value foot-gun — empty string would otherwise pass theOption::Somecheck and feed an empty URL to the discovery probe. -
Soft-check wiring is silent when issuer absent: the pre-push
[6c/14]step printsSKIP (CRAIG_IDENTITY_ISSUER not set)and returns, never bails. Production deploy pipelines pass the env vars explicitly so the samecargo xtask validateinvocation becomes a hard gate by virtue of the env-var presence — no separate "production-mode" flag needed. -
Devstack smoke validates the failure path: running against the live devstack Keycloak surfaces a real
JWKS unreachablefailure (the issuer self-advertiseshost.docker.internalin the discovery doc, which the host can’t DNS-resolve). That’s the gate working as designed — production deployments with consistent intra-cluster DNS would pass. -
Workspace deps unchanged: reuses
reqwest,serde,serde_json,clap,anyhowalready inxtask/Cargo.toml. No new dependency.
Step 11 deviations (2026-05-11)
Step 11 adds caller-service + acting-worker columns to services/craig-security’s `audit_log and populates them from the wildcard subscriber. The migration is the destructive pre-1.0 variant the plan body called for: ALTER TABLE audit_log ADD COLUMN actor_service TEXT NOT NULL DEFAULT 'unknown', ADD COLUMN actor_user_sub UUID. Plus matching indexes (idx_audit_log_actor_service, idx_audit_log_actor_user_sub) so the new filters scale.
-
Insert path is param-driven, not Claims-driven: the plan body’s reference snippet showed
insert_audit_entryreadingclaims.service_id()+claims.actor.as_ref().map(\|a\| &a.sub)directly. In practice today’s only insert caller is the wildcard subscriberhandle_inbound_event, which gets anEventEnvelope— noClaimsin scope. Pushing&Claimsinto the store-level fn would change its API for one caller that doesn’t have one.InsertAuditEntryParamsinstead gainsactor_service: String+actor_user_sub: Option<Uuid>; the subscriber fills them from the envelope; future direct callers fill fromclaims.service_id().unwrap_or("unknown").to_string()+claims.actor.as_ref().and_then(|a| Uuid::parse_str(&a.sub).ok()). Same end shape, store stays decoupled from Claims. -
actor_service≡EventEnvelope.source_service: the envelope’ssource_serviceis exactly the caller-service signal the plan body asks for. For service-to-service flows where craig-cases is emitting on behalf of a worker authenticated via craig-web,source_serviceis stillcraig-cases— the outermost service producing the audit-relevant side effect, not the outer BFF. That matches the audit-attribution intent: the row reflects which service did the work, not which BFF was in the call chain. -
actor_user_subderives from the existing payload best-effort extraction: factored intoevent_parsing::extract_actor_user_sub(envelope) → Option<uuid::Uuid>so the priority order (created_by→approved_by→user_id→archived_by) is testable in isolation. ReturnsSome(uuid)when the first non-null attribution field parses as UUID;Nonefor system identifiers ("system"), legacy preferred_username-shaped values ("jane.doe"— should not appear post Plan A Step 4 but guarded), and events without any attribution field. 7 inline unit tests pin each branch. -
AuditQuery+ list/count store params gainactor_service: Option<String>+actor_user_sub: Option<Uuid>filters: pure additions (no rename). The validated sort whitelist also accepts the two new columns. Lets the BFF pivot the audit view on caller-service or acting-worker without a separate endpoint. -
2 devstack integration tests (
audit_rows_carry_actor_service_attribution,audit_filter_by_actor_service) verify the round-trip: every audit row surfaces a non-emptyactor_serviceafter reseed, and the filter narrows the result set to a single source service. 1888/1888 workspace tests pass post-reseed (up from 1879 — 9 new tests). -
Migration is timestamp
20260511100000_audit_log_actor_columns.sql: pre-1.0 destructive, no backfill machinery (no production data to preserve). New rows on a fresh reseed populate from the subscriber.
Step 10 deviations (2026-05-11)
Step 10 ships the bearer flip across services/craig-intake’s 4 outbound paths (partner_auth, signer_auth, cases_forwarder, attachments) — all now use Step 7’s `CraigClientExt::with_service_identity instead of manually building bearer_auth(&token.current().await?). Cosmetic refactor in spirit, but it brings intake’s outbound surface in line with the Step 7 helper contract (1 way to authenticate outbound, not 2). 1879/1879 workspace tests pass.
-
No X-Craig-Actor attached: the plan body called for partner-as-actor attribution on
/partners/verifyand/signer-keys/by-kidcalls. The actor JWT contract Step 5 designed targets workers (WorkerActor { sub, preferred_username, roles }); encoding partners would conflate principal types. Partner identity already flows in the request body (partner_idin the cases-forwarder report payload; the raw API key in the partner-verify body) and downstream audit rows read it from there — no signature attribution needed at the JWT level. If a future audit requirement needs partner-bound JWTs, extending the actor contract toPrincipal { Worker | Partner }can be done as a successor. -
partner_auth.verifycan’t have a partner actor: the verify call IS the partner-identity lookup — at call time, intake doesn’t yet know whose API key it’s holding. Service-only bearer is the only honest option. -
signer_auth.lookupsimilarly service-only: looking up a signer key bykidruns ahead of JWS verification; the partner association is downstream. -
CasesForwarderSink::service_token()accessor added: givesattachments.rsdirect access to theOidcServiceTokenso it can pass it towith_service_identity. The legacybearer_token() → Stringhelper (Step 4-era) is retired — there’s now a single way to send the bearer (through the trait).
Step 9b deviations (2026-05-11)
Step 9b ships the actual BFF flip: ApiClient methods + the shared helpers (fetch_page, fetch_page_or_empty, batch_lookup) now take &SessionUser instead of token: &str. Outbound calls send craig-web’s client_credentials token as Authorization: Bearer and attach a freshly-minted X-Craig-Actor JWT identifying the acting worker. Backend services (Step 6) verify the actor JWT and lift the worker identity into Claims::actor.
-
Surface was ~91 call sites, not the 25 the plan body called out: the plan body cited 6
bearer_authsites inapi_client.rs. In practice those 6 methods are called by ~91 sites inservices/craig-web/src/routes/(some viafetch_page/fetch_page_or_empty/batch_lookuphelpers). 29 route files updated. Mechanical regex pass + manual cleanup oflet token = &user.access_token;bindings that became unused. -
&SessionUserparameter, not&Claimsor&WebSession:SessionUseris craig-web’s projected session shape (already used everywhere as theExtension-injected handler param). It hassub,username,roles,access_token— exactly whatWorkerActorneeds plus the bearer for the legacy fallback. UsingClaimswould require synthesizing one from session data; usingWebSessionwould shift the handler-side signature contract. -
ApiClient::apply_identitymintsWorkerActordirectly, doesn’t go throughCraigClientExt::with_craig_identity: the Step 7 extension trait takesOption<&Claims>and projects viaWorkerActor::from_claims. Building a syntheticClaimsfrom aSessionUserjust to feed it back into a projection is needless. The ApiClient internal helper buildsWorkerActorfromSessionUserdirectly + callsissuer.mint(&worker)+ attaches the header. Same end result, less indirection. -
Legacy bearer-forward fallback kept: when
service_tokenoractor_issuerisNone(e.g., a devstack profile without provisioning),apply_identityforwardsuser.access_tokenas the bearer. Lets the flip land safely even if a deployer hasn’t provisioned the new principals yet. -
Devstack reload smoke verified: all 9 services log peer-JWKS-wired + per-service issuer-wired at boot; craig-web /healthz returns 200; backend services accept BFF calls with the new identity shape.
-
Authz had to scope on the acting worker, not the calling service: a P0 issue surfaced by E2E: with X-Craig-Actor flowing, backends saw
claims.is_service() == true(outer caller is craig-web) and admitted via the service-caller rule with scope=all — caseworkers saw all cases instead of their own, RBAC boundaries broke. Twocraig-authzfixes: (1)build_check_input/build_scope_inputnow readclaims.acting_worker()for the policy’s identity fields (sub, preferred_username, roles) and only treat the JDMis_serviceflag as true when no actor was lifted; (2)claims_sub_uuidreads the acting worker’s sub soListScope::AssignedWorker(sub)filters by the worker’s UUID, not the caller service’s. Service-caller-only paths (peer bulk-fetch, cold-boot fallback) still admit because the outer Rust-levelclaims.is_service()method is unchanged and craig-rules'allow_bootstrap_fallbackstill admits services. End-to-end verified: 203/203 E2E tests pass with the BFF flip active.
Step 9a deviations (2026-05-11)
Step 9 split into 9a (provisioning + craig-web bootstrap wiring; this MR) and 9b (ApiClient flip + ~25 caller-site updates; subsequent MR). Step 9a’s purpose: activate Step 6’s actor-JWT verifier on every backend service by provisioning the deployment-wide peer-JWKS map + per-service signing keys, plus extend craig-web’s AppState so the ApiClient methods can borrow the new primitives once Step 9b flips them.
-
Devstack provisioning of actor keys pulled into Step 9, not Step 14: plan body Step 14 originally owned devstack key provisioning. But Step 6’s verifier middleware is
fail-closedwhen anX-Craig-Actorheader arrives without a populated registry — so Step 9b’s flip can’t safely land until peer-JWKS provisioning is in place. Pulling provisioning into Step 9a removes the deploy-window where keys are missing while callers are sending the header. 9 ES256 keypairs generated once + committed atdevstack/devstack-actor-keys.env(devstack-only, with explicit production-warning header). docker-compose.ymlenv_file-references the file for all 9 services. Step 14 remains for the production-grade provisioning story (file-mounted secrets, IaC integration, rotation). -
craig-web’s
WebSettingsgainsclient_id+client_secret: craig-web doesn’t use the standardbootstrap()(it has its own settings shape since it’s not a stateful backend), so the Step 8 env-var contract had to be re-added insideservices/craig-web/src/config.rs. Optional fields —Nonefalls back to forwarding the worker bearer (legacy). -
ApiClient::with_outbound_identity()builder, not constructor: keeps the existingApiClient::new(http)call sites untouched in this MR.AppStatechains.with_outbound_identity(svc, issuer)at boot. Step 9b will use these fields inside the 6 method bodies. -
ApiClient methods + ~25 caller sites deferred to Step 9b: keeping Step 9a tightly scoped to "wire the primitives into place" lets it land safely. Step 9b is the actual semantic flip: method signatures change from
token: &strtosession: &WebSession, internals usewith_craig_identity, all caller sites inservices/craig-web/src/routes/update. -
Devstack reload smoke verified: all 9 services log
actor-JWT verifier wired with peer JWKS count=9at boot + their per-serviceactor-token issuer wiredline. jane.doe LIST cases returns 7 (caseworker + supervisor / Georgia realm-flat) — unchanged from pre-Step-9a behavior.
Step 8 deviations (2026-05-11)
Step 8 ships bootstrap wiring: new craig-auth env-loader helpers (load_signing_keypair_from_env, load_peer_jwks_from_env), BootstrapResult gains service_token: Option<OidcServiceToken> + actor_issuer: Option<ActorTokenIssuer>, and AuthLayer is wired with the deployment-wide peer-JWKS registry so Step 6’s actor verification activates as soon as keys are provisioned. All 6 standard services consume the bootstrap-provided primitives (deletes the duplicate OidcServiceToken construction Step 4 placed in each main.rs); craig-rules discards both fields with _ patterns. 5 inline unit tests cover the env loader paths.
-
No
ClientExt::craig_internal()pre-bundledreqwest::Clientwrapper: plan body’s last sentence sketched aClientextension that returns a builder pre-configured with both the token source + actor issuer. Step 7 already ships per-method extension helpers onRequestBuilder(with_service_identity,with_actor,with_craig_identity), which compose with arbitrary URL/method combinations. Adding a second pre-bundled wrapper at theClientlevel would either duplicate that surface or hide the explicit composition that makes the Step 9/10 flips legible. Skipped without loss of capability. -
Option<…>BootstrapResult fields: services boot without env-provisioned principals (service_token = None,actor_issuer = None) so the rollout doesn’t gate on devstack/CI provisioning. Outbound callers (Step 9/10) handle theNonecase as a config error at the call site rather than at boot — keeps Step 8 isolated to wiring without forcing operator action. -
Per-service keypair env vars are inline JWK (
<prefix>SIGNING_JWK) + explicit kid (<prefix>SIGNING_KID): no file-path variant yet. Production deployments will want a path / secret-mount path; deferred to a future iteration alongside the production secrets story. Pre-1.0 devstack uses inline. -
Peer JWKS env is a single deployment-wide var
CRAIG_PEER_JWKS_JSON: holds a JSON array of{iss, kid, jwk}entries. Same value for every CRAIG service. Allows multiple entries perissfor rotation windows. Devstack provisioning is deferred to Step 14. -
AuthLayer::with_actor_registrywired at bootstrap when the peer map is non-empty: when the map is empty (default until Step 14 provisions it), middleware fail-closes any inboundX-Craig-Actorper Step 6. When the map is populated, peer-service actor JWTs verify and lift intoClaims::actor. -
craig-rulesdiscardsservice_token+actor_issuerwith_patterns: craig-rules is the rules-engine source of truth; it makes no outbound CRAIG service calls and never mints actor JWTs. Other 5 standard services consume the new fields.
Step 7 deviations (2026-05-11)
Step 7 ships crates/craig-auth/src/client_ext.rs: outbound reqwest::RequestBuilder extension methods that attach the calling service’s client_credentials bearer (with_service_identity) and optionally mint + attach an actor JWT (with_actor). 6 inline unit tests use a localhost fake OIDC token endpoint for the bearer-attachment path; the actor-attachment path round-trips through Step 6’s verifier to prove the minted header is well-formed.
-
with_service_identityis async, not sync: plan body D6 showedcurrent_blocking()returning a string synchronously. Step 4 shipsOidcServiceToken::currentasasync fn(cache hit returns immediately; cache miss awaits areqwestPOST). A blocking wrapper would deadlock when invoked inside an async runtime context. The trait method isasync fnusing Rust 2024’s nativeasync fn in trait. ReturnsResult<Self, ServiceTokenError>so callers can.await?it inline. -
with_actortakes an explicit&ActorTokenIssuerparameter (not a global): plan body D6 referenced anACTOR_ISSUERconstant. Globals tangle dependency injection. The trait takes the issuer as a parameter; Step 8 will hold the per-serviceActorTokenIssuerinBootstrapResultso handlers can borrow it fromExtension. -
Combined helper
with_craig_identity: added a third trait method that runs both attachments in one await + result. Common BFF flip pattern from Step 9 looks much cleaner with the combined form than with chained.await?and?interleaved inside a builder chain. Tests cover both ergonomic surfaces. -
WorkerActor::from_claimslives onclient_ext: the projection method is defined as an inherent impl onWorkerActor(in this module’s file) socrate::actor_tokendoesn’t gain aClaimsdependency. Test covers that the projection strips extras (email,aud,azp) that the actor JWT contract doesn’t carry.
Step 6 deviations (2026-05-11)
Step 6 ships crates/craig-auth/src/actor_verifier.rs (the verifier primitive) and wires it into crates/craig-auth/src/middleware.rs::auth_middleware. 4 middleware-level integration tests + 8 inline verifier tests cover round-trip, signature failure, tampered payload, smuggled long TTL, wrong audience, malformed JWT, worker-token ignore path, and unconfigured-registry fail-closed.
-
Per-service JWKS via
ActorJwksRegistrytrait, not a single "craig-signing" JWKS: plan body D5 / D3 framed verification as "validate the actor JWT against craig-signing’s JWKS" as ifcraig-signingwere a service publishing keys. It’s a library crate, not a service. Each CRAIG service signs with its own keypair (Step 5 + Step 8); the verifier needs to look up the right key by(iss, kid). Step 6 introducesActorJwksRegistry(trait +StaticActorJwksRegistryHashMap-backed default impl). Step 8 will populate the registry per service from peer-service JWKS endpoints; Step 6 ships the verification primitive + abstractions. -
Workers carrying
X-Craig-Actorare log+ignored, not rejected: plan body D5 said "Reject if actor JWT validation fails — never silently drop." For invalid actor JWTs on service callers, Step 6 honors that contract (401). But for the orthogonal case of a worker JWT (noservice:*role) carryingX-Craig-Actor, that header is semantically meaningless — workers can’t assert on-behalf-of identity — and rejecting would block worker traffic if a stray middleware ever attached the header. The middleware logs the anomaly atdebugand ignores the header for worker callers. This is fail-open for a header the receiver doesn’t trust anyway; the actor identity is never lifted intoClaims::actor, so authorization logic continues to treat the request as worker-only. -
Header present + registry unconfigured → 401, not silent skip: between Step 6 landing and Step 8 wiring the registry into each service’s bootstrap, no service has an actor registry. The plan body’s "never silently drop" maps to: if a service-caller sends
X-Craig-Actorbut the receiver has no registry, reject 401. Callers won’t start sending the header until Step 7 ships outbound helpers + Step 9/10 flip the callers, so this fail-closed posture is invisible during the deploy gap. -
TTL ceiling + iat-in-future checks in the verifier, not just jsonwebtoken: jsonwebtoken’s
Validationenforcesexp > nowbut doesn’t cap(exp - iat). A malicious peer could mint a 24h-lived actor JWT and replay it. The verifier explicitly caps TTL atACTOR_TOKEN_TTL_SECONDS + 30sskew tolerance and rejectsiat > now + 30s. Both have dedicatedActorVerifyErrorvariants for clearer operator log lines.
Step 5 deviations (2026-05-11)
Step 5 ships the ActorTokenIssuer module: short-lived ES256-signed compact JWTs that carry worker-on-behalf-of identity through service-to-service hops, with a fixed aud = "craig-internal-actor" audience namespace and a 10-minute TTL. 7 inline unit tests cover header shape, payload claims, signature verification, tampered-payload rejection, and wrong-key rejection.
-
Module placed in
crates/craig-auth/src/actor_token.rs, notcraig-signing: the plan body called for the module to live incrates/craig-signing. craig-signing’s mandate is the cross-languagecanonicalize_json+ payload-hash contract that the Rust / TypeScript / Python / browser intake SDKs all consume (single source of truth for the partner JWS signing contract). Adding server-only ES256 JWT-minting would have grown craig-signing’s API surface (read by SDK consumers in three languages) and pulled inp256+base64ct+chronodeps. craig-auth is server-only, already houses related primitives (Claims,OidcServiceToken,OidcDiscovery,JwksProvider), and is the natural home for the verifier (Step 6) too. -
No "existing keypair" to reuse: the plan body said "Reuses each craig-* service’s existing canopy-signing-style keypair (already used for ADR-018 partner JWS — same key, distinct
audnamespace)". CRAIG services do not have per-service signing keys today — partner-JWS infrastructure (services/craig-{intake,security}for verify, in-browsercraig-sign.js+ SDK clients for sign) handles partner-supplied keys. CRAIG services themselves don’t sign anything yet. TheActorTokenIssuerconstructor takes ap256::ecdsa::SigningKeydirectly; Step 8 (bootstrap wiring) will provision per-service keypairs and load them at boot from aCRAIG_<SVC>__SIGNING_JWKenv var. -
WorkerActorprojection instead ofClaims: the issuer accepts a smallWorkerActor { sub, preferred_username, roles }struct rather than borrowing a fullClaimsinstance. This decouples craig-auth’s actor-token module from upstreamClaimsevolution (e.g., the deferred Step 3b configurable roles-claim path) and keeps the minted payload bounded to vetted fields — an inbound bearer’s rawaudarray can’t accidentally leak into the actor JWT.
Step 4 deviations (2026-05-11)
Step 4 ships the OidcServiceToken module and retires the 6 in-service Keycloak ROPC bootstraps. New per-service confidential OIDC clients (service:craig-<svc> realm roles) in the devstack realm; per-service CLIENT_ID / CLIENT_SECRET env vars in docker-compose.yml. 98 rulesets gained an i_service JDM input + service-caller admit rule. Claims::require_* OR-in is_service() so role-gated handlers admit service callers without rewriting them.
-
Cold-boot chicken-and-egg fallback extended to service callers: pre-Plan-E the seed authenticated as the
adminROPC user, soservices/craig-rules/src/api.rs::rules_authz_or_admin_fallbackadmitted admin to escape the empty-cache fail-closed window at boot. Post-Plan-E the peer-service bulk-fetch (each service’sZenAuthzEngine::bootcallingRulesClient::list_by_prefix) is now a service principal with no admin role. The existing fallback rejected it, the warmup retry budget exhausted before the seed completed inserting policies, and every authzLISTscope downstream came backDenied. Step 4 extracts a pureallow_bootstrap_fallback(&Claims)predicate and admitsclaims.is_service()alongside theadminrole. Real per-user authz still applies once the cache is populated. 4 inline unit tests cover the predicate. Verified end-to-end on acargo xtask dev reseedcold-boot: jane.doe (supervisor) sees 7 cases via realm-flat scope, bob.smith (caseworker) sees 2 via assigned-worker scope, admin sees 7.
Step 3 deviations (2026-05-10)
Step 3 ships the Claims API extensions and ADR-028 acceptance. 9 new unit tests in crates/craig-auth/src/claims.rs (42 total, was 33).
-
Configurable
CRAIG_IDENTITY_ROLES_CLAIM_PATHdeferred to Step 3b: the plan body called for a configurable JSON-path lookup so non-Keycloak IdPs (Oktagroups, Azure ADroles, etc.) can emit roles at their own claim path. Implementing this means changingClaimsto retain the raw JSON value (or use aserde_json::Valueaccessor) instead of the typedrealm_access.rolesfield. The refactor is invasive and orthogonal to the service-caller predicates this step shipped. Step 4 (OidcServiceToken) can land before Step 3b. TheCRAIG_IDENTITY_SERVICE_ROLE_PREFIXenv var (used by the service-role check) ships in Step 3 as planned. -
ADR-028 accepted in Step 3, not Step 2: the plan body Status row for Step 1 said acceptance lands in Step 2. Step 2 shipped OIDC discovery convergence (a runtime fix) without flipping ADR-028’s status. Step 3 ships the
Claimsextensions that the ADR’s contract defines — accepting it now is more accurate. ADR-028 Status updated 2026-05-10; nav.adoc entry updated to match. -
#![forbid(unsafe_code)]precludes in-crate env-var override tests: the env-var prefix override behavior would requirestd::env::set_varwhich isunsafein Rust 2024 edition. The crate forbids unsafe at the lib level. Coverage moved out-of-crate (deployment / integration tests / manual verification). The default-prefix path has full coverage; the override path is exercised by env config.
Step 2 deviations (2026-05-10)
Step 2 ships OIDC discovery convergence in crates/craig-auth (new OidcDiscovery module + JwksProvider rewired to use it) and services/craig-web/src/auth.rs (login/callback/logout flows resolve endpoints via the discovery doc). Five new unit tests cover the discovery client (fetch, TTL caching, on-expiry refresh, split-DNS host rewrite x2). All 1831 workspace tests pass; devstack reload verified end-to-end.
-
Split-DNS host rewriter: the plan body assumed the discovery doc’s endpoints could be consumed verbatim. In practice, Keycloak emits the public issuer URL (
http://host.docker.internal:43441/realms/craig/…;) but internal services need to reach it atkeycloak:8080. Step 2 adds arewrite_endpoint_hosthelper that swaps the host portion ofjwks_uriandtoken_endpointto the internalfetch_urlwhenissuer != fetch_url.authorization_endpointandend_session_endpointkeep the public URL since those land in HTTP redirects to the user-agent. This is the pattern every OIDC client library handles for split-DNS deployments. -
Rust-field rename split to Step 2b: the plan body called for renaming
keycloak_issuer→oidc_issuerwith a backwards-compat env var alias for one release. Step 2 ships the architectural fix only; Step 2b lands the rename as a separate, mechanically-large but content-trivial MR. Per pre-1.0 policy (no production data to migrate, no external consumers locked in), Step 2b is destructive: no alias, no shim.keycloak_issuer→oidc_issuer,keycloak_url→oidc_internal_url,CRAIG_<SVC>KEYCLOAK_ISSUER→CRAIG_<SVC>OIDC_ISSUER,CRAIG_<SVC>KEYCLOAK_URL→CRAIG_<SVC>OIDC_INTERNAL_URLacross all 25 affected files. -
Stricter JWT validation already in place: the plan body called for "bring craig’s stricter JWT validation along (aud/azp/scope/typ + nbf checks)". Those were already implemented pre-Plan-E in
crates/craig-auth/src/{claims.rs,jwks.rs}per ADR-021. No additional code needed in Step 2.
After this plan lands
-
Per-service
azpdistinguishes which craig service called any given endpoint — observable via audit_logactor_servicecolumn. -
Audit log captures both the calling service AND the acting worker, so reviewers can answer "did the worker invoke X directly or via the BFF?" with a per-row read.
-
KeycloakServiceTokenretired in favor of IdP-neutralOidcServiceToken+ OIDC-discovery +client_credentials— fully aligns service-to-service auth with ADR-026’s IdP-neutrality stance. -
Operators of any compliant OIDC backend (Keycloak, Authentik, Dex, Okta, Entra, ForgeRock, custom) can deploy craig without backend-specific code in craig itself.
-
xtask identity verifyprovides a deployer-facing conformance gate — operators run it post-provisioning to confirm their backend matches craig’s contract. -
xtask identity renderprovides reference IaC for the three most common open-source backends; non-supported backends have a clear contract spec to fulfill manually. -
Service-token lifetime decoupled from worker session — long-running drainer publishes + scheduled jobs no longer depend on a worker’s 8-hour session.