Plan: Introspection-Mode Token Validation

On this page

Status

Step Description Status

1

Plan body + ADR-029 (Proposed) + nav entry move (Planned → Active) + CHANGELOG. Docs-only; zero code changes.

Done (2026-05-12) — MR !269

2

Introspection primitives. OidcDiscoveryDoc gains introspection_endpoint; new crates/craig-auth/src/introspection.rs with IntrospectionClient + TokenClaimsCache (DashMap-backed, SHA-256-hashed keys, Instant-clocked TTL) + RFC 7662 → Claims normalization layer enforcing per-claim parity with the JWS path (aud / token_type / exp / nbf). xtask identity verify gains an introspection-endpoint probe arm. Inline wiremock tests. No middleware or bootstrap changes in this step.

Done (2026-05-13) — MR !270

3

AuthLayer refactor + Settings + bootstrap. trait ClaimsExtractor with JwsExtractor / IntrospectionExtractor / AutoExtractor (token-shape dispatch). ServiceSettings gains token_validation_mode (default Auto) + 5 introspection knobs (cache TTL, cache max entries, serve-on-outage, endpoint override, auth method). init_auth dispatches on mode + hard-fails missing prerequisites with operator-clear error messages. ADR-029 Proposed → Accepted.

Done (2026-05-13) — MR !271

4

Devstack Kanidm profile + xtask identity render --backend kanidm adapter. RenderBackend::Kanidm variant emits a CLI bootstrap script (Kanidm provisions via kanidm system oauth2 create; no admin API client per ADR-026). Devstack identity-multibackend profile gains Kanidm service + ports + idempotent admin-recovery bootstrap.

Done (2026-05-13) — MR !272

5

Integration tests. kanidm_status_and_token_endpoint_reachable (probes /status, asserts the token endpoint dispatches the grant (not unsupported_grant_type), asserts the /oauth2/token/introspect endpoint exists). identity_multibackend_available extended to probe Kanidm /status with the self-signed TLS cert. Full-mint round-trip deferred because it requires operator-side bootstrap.sh to be run after idm_admin password recovery — same operator-bootstrap gap the original ZITADEL test has.

Done (2026-05-13) — MR !273

6

Docs reframe + ADR amendments + plan completion audit + archive. idp-integration.adoc Tested-backends list rewritten as a "JWS or introspection" matrix. ADR-026 + ADR-028 gain Plan-F revision notes. Plan moves Active → Archive; .claude/CLAUDE.md Phase Status update.

Done (pre-ADR-030) — this MR

Epic: none (single-plan initiative under Security & Compliance)
Issues: filed per step at implementation start; Step 1 = #404
Branch: docs/introspection-validation-step1-plan-body (Step 1); successors follow feat/introspection-validation-step<N>-<slug>
Predecessor: Built on Plan E (Service Identity, archived 2026-05-11). Direct follow-up to the Dex strip (!267) + ZITADEL JWT-mode addition (!268) under idp-multibackend-tests.adoc.

Context

CRAIG’s craig-auth::middleware::auth_middleware validates inbound bearer JWTs locally. The middleware calls layer.provider.validate_token(token).await (crates/craig-auth/src/middleware.rs:78), which dispatches into JwksProvider::validate_token (crates/craig-auth/src/jwks.rs:163-221); that path calls jsonwebtoken::decode::<Claims>(token, &key, &validation) at crates/craig-auth/src/jwks.rs:197. The path is fast (~10µs decode + cached JWKS lookup), resilient to short IdP outages (JWKS refresh is on a 1h TTL background task), and the natural fit for self-contained signed JWTs (RFC 9068).

It is one of two mature halves of the OAuth design space. The other half — opaque or encrypted tokens validated via introspection (RFC 7662) — is the model that ZITADEL defaults to, that Kanidm uses unconditionally for client_credentials access tokens (JWE-encrypted, validated only via introspection), and that Keycloak ("lightweight access tokens"), authentik (encrypted-token mode), and several other backends offer as configurable alternatives.

Today CRAIG’s "IdP-neutral over OIDC backends with client_credentials`" claim (ADR-026 + ADR-028) is materially narrower than the OAuth ecosystem: it excludes any backend whose `client_credentials path defaults to or requires introspection. Backends affected:

  • Kanidm — client_credentials access tokens are unconditionally JWE-encrypted via jwe_a128gcm_encrypt at kanidmd/lib/src/idm/oauth2.rs:1861. Validation requires the introspection endpoint. Confirmed in Phase 0 sandbox (2026-05-12; see Phase 0 Findings below).

  • ZITADEL — defaults to opaque access tokens (introspection-mandated). Operators can pin access_token_type = ACCESS_TOKEN_TYPE_JWT per machine user to force JWS (which is what !268 does), but that’s working around the IdP’s default rather than supporting it natively.

  • Keycloak — supports "lightweight access tokens" (smaller, opaque format); deployers in regulated environments may pin this for log-volume / data-minimization reasons.

  • authentik — encrypted-token mode available for high-confidentiality deployments.

This plan adds an introspection-mode validation strategy to craig-auth so deployers can choose either path (or have it auto-detected) per their backend configuration. The architectural goal: CRAIG’s IdP-neutral claim becomes "any OIDC backend with client_credentials, validated locally (JWS) or via introspection (RFC 7662)."

The work was deferred from !268 (ZITADEL JWT-mode adapter) precisely because it deserves its own architectural review surface — it changes the per-request runtime cost model, introduces cache-correctness work, and lands ADR-029. Doing it standalone keeps each MR independently reviewable.

Phase 0 Findings (durable; source-confirmed 2026-05-12)

Kanidm v1.10.1 (Rust-aligned candidate)

  • client_credentials grant works. Verified by sandbox: created an OAuth2 "resource server" via POST /v1/oauth2/_basic with idm_admin token; added a scope-map binding idm_all_accounts → openid (the RS is itself a member of idm_all_accounts because it carries EntryClass::Account); HTTP basic auth + grant_type=client_credentials&scope=openid at /oauth2/token returned HTTP 200 + an access token.

  • Discovery URL pattern is per-resource-server: https://<host>/oauth2/openid/<rs_name>/.well-known/openid-configuration. Token endpoint is issuer-level: <host>/oauth2/token. Introspection endpoint: <host>/oauth2/token/introspect. JWKS exposes one ES256 (P-256) sig key per RS.

  • Introspection works and returns expected RFC 7662 claims (sub, aud, client_id, username, scope, exp, iat, nbf, jti, active, token_type).

  • Dealbreaker for JWS path: access tokens are JWE-encrypted (alg=A128KW, enc=A128GCM, 5-segment compact serialization). Source-confirmed at kanidmd/lib/src/idm/oauth2.rs:1861 — jwe_a128gcm_encrypt(&access_token_data, ct) is called unconditionally during access-token issuance. No configuration knob to switch to JWS for client_credentials. Same on master. The intended validation path is the introspection endpoint — which is exactly what Plan F adds.

ZITADEL v4.15.0 (Go-based; shipped in !268 with JWT-mode pin)

  • Standard tenant-level discovery URL at <host>/.well-known/openid-configuration. Token endpoint: /oauth/v2/token. Introspection endpoint: /oauth/v2/introspect. JWKS: /oauth/v2/keys — two RS256 sig keys (no enc keys).

  • client_credentials is advertised in grant_types_supported. Bad-creds probe returns invalid_client (grant parsed + dispatched, only auth lookup failed) — same shape Authentik shows; opposite of what Dex returned.

  • !268 pins access_token_type = ACCESS_TOKEN_TYPE_JWT per machine user so issued tokens validate locally. ZITADEL’s default is opaque tokens that require introspection. Plan F Step 5 adds a zitadel_opaque_mode_introspection integration test covering the default.

  • Bootstrap chicken-and-egg: Terraform provider needs an authenticated PAT. Initial-instance human admin is created via ZITADEL_FIRSTINSTANCE_ORG_HUMAN_* env vars (already wired in docker-compose.yml). Step 5 pre-bakes via ZITADEL_FIRSTINSTANCE_MACHINEKEYPATH for sandbox reproducibility.

Scope

In scope:

  • trait ClaimsExtractor abstraction in craig-auth with three impls — JwsExtractor (today’s path, behavior unchanged), IntrospectionExtractor (new), AutoExtractor (dispatches on token shape).

  • RFC 7662 introspection client + per-process DashMap-backed token claims cache (SHA-256 hashed keys, std::time::Instant-clocked TTL).

  • Claims normalization layer mapping per-IdP introspection response shapes into the canonical Claims struct, enforcing audience / token-type / expiry / nbf parity with the JWS path (per ADR-021).

  • New ServiceSettings field token_validation_mode (Jws / Introspect / Auto; default Auto) + 5 introspection knobs (cache TTL, cache max entries, serve-on-outage tunable, endpoint override, auth method).

  • Devstack identity-multibackend profile gains a Kanidm service.

  • cargo xtask identity render --backend kanidm adapter (emits CLI bootstrap script — no admin API client per ADR-026).

  • cargo xtask identity verify gains an introspection-endpoint probe arm (soft-check at validate [6c/14]).

  • Integration tests: Kanidm full round-trip + ZITADEL opaque-mode introspection.

  • ADR-029 Proposed → Accepted lifecycle.

  • idp-integration.adoc reframe; ADR-026 + ADR-028 revision notes.

  • Metrics for introspection requests, cache hits, request duration, cache size.

Out of scope:

  • Token-exchange grant (RFC 8693) — separate work if a deployer asks.

  • Switching CRAIG’s default validation mode away from JWS-first — Auto mode dispatches to JWS for 3-segment tokens (today’s behavior); introspection only fires for JWE/opaque.

  • Replacing Plan E’s per-service OidcServiceToken — outbound + inbound are decoupled; service-identity tokens stay how Plan E wired them.

  • X-Craig-Actor JWT verification path — actor JWTs are CRAIG’s own ES256-signed tokens (peer-JWKS); unaffected by IdP-side validation mode.

  • Cross-replica cache sharing (Redis or similar) — per-process cache only. Each replica builds its own independently.

  • Push-based revocation (CAEP / SSE token-revocation streams) — captured in ADR-029’s Open Questions for forward-compat consideration only.

  • DPoP (RFC 9449) / mTLS-bound (RFC 8705) extractors — the trait abstraction leaves room but Plan F doesn’t ship these.

Design

Architectural decisions (locked)

Decision Choice Rationale

Dispatch shape

trait ClaimsExtractor with JwsExtractor / IntrospectionExtractor / AutoExtractor impls

Future-extensible to DPoP (RFC 9449), mTLS-bound (RFC 8705), public-key JWE, CAEP push-revocation. Rust 2024 async-fn-in-trait (already used in Plan E Step 7 for CraigClientExt) makes this clean.

Default mode

Auto (detect from token shape) with explicit jws / introspect operator overrides

Zero-config compatibility across the candidate IdP matrix. Operators can pin for predictable cost.

Outage posture

Fail closed by default; opt-in serve_on_outage: true lets cache hits continue serving until TTL

Symmetric with JWS path’s fail-closed JWKS-refresh failure mode; safer default.

Cache layer

In-process dashmap::DashMap<TokenHash, CachedClaims> with lazy + periodic TTL eviction

dashmap is already at workspace level. Per-instance cache; no shared-state correctness concerns; horizontal scale fine because each replica builds its own cache.

Outbound auth for introspection call

Reuse OidcServiceToken (existing per-service client_credentials source from Plan E Step 4)

Already cached, refresh-aware, the right principal. CRAIG presents itself to the IdP as the service introspecting on a worker’s behalf.

Token hash algorithm

SHA-256 (32 bytes)

HMAC-SHA-256 evaluated in ADR-029 Alternatives Considered; cache is per-process in-memory so cross-node correlation isn’t a threat that warrants the extra key-rotation surface.

Cache clock

std::time::Instant (monotonic)

Wall-clock skew during NTP corrections would otherwise expire valid cache entries or retain expired ones. Convert claim exp (wall-clock seconds since epoch) to an Instant at insert time.

Logging discipline

Raw bearer never appears in logs; the first 8 bytes of the SHA-256 hex digest are used as a diagnostic correlator

Standard practice. Codified in ADR-029.

Actor JWT lifting (X-Craig-Actor)

Unchanged

Actor JWTs are CRAIG’s own inter-service ES256-signed tokens validated against the peer-JWKS map; not affected by IdP-side validation mode.

Trait sketch

pub trait ClaimsExtractor: Send + Sync {
    async fn extract(&self, token: &str) -> Result<Claims, AuthError>;
}

// 1) JwksProvider (existing) implements the trait — zero behavior change.
impl ClaimsExtractor for JwksProvider { /* delegates to validate_token */ }

// 2) Introspection path.
pub struct IntrospectionExtractor {
    client: Arc<IntrospectionClient>,
}
impl ClaimsExtractor for IntrospectionExtractor { /* delegates to client.extract */ }

// 3) Auto path — dispatches on token shape.
pub struct AutoExtractor {
    jws: Arc<JwksProvider>,
    introspection: Arc<IntrospectionClient>,
}
impl ClaimsExtractor for AutoExtractor {
    async fn extract(&self, token: &str) -> Result<Claims, AuthError> {
        match token_shape(token) {
            TokenShape::Jws => self.jws.extract(token).await,         // 3 segments
            TokenShape::Jwe | TokenShape::Opaque => {                  // 5 segments or no dots
                self.introspection.extract(token).await
            }
        }
    }
}

// AuthLayer becomes mode-agnostic.
pub struct AuthLayer {
    extractor: Arc<dyn ClaimsExtractor>,
    actor_registry: Option<Arc<dyn ActorJwksRegistry>>,
}

Introspection client flow

impl IntrospectionClient {
    pub async fn extract(&self, token: &str) -> Result<Claims, AuthError> {
        let hash = sha256(token);                       // never log/store the raw bearer

        if let Some(claims) = self.cache.get(&hash) {   // lazy TTL check inside get()
            return Ok(claims);
        }

        let endpoint = self.resolve_endpoint().await?;  // discovery + override + split-DNS
        let svc_token = self.service_token.current().await?;

        let resp = self.http
            .post(&endpoint)
            .bearer_auth(&svc_token)                    // or basic_auth(client_id, client_secret) per config
            .form(&[("token", token)])
            .send()
            .await
            .map_err(AuthError::IntrospectionUnreachable)?;
        let body: IntrospectionResponse = resp.json().await.map_err(AuthError::IntrospectionFailed)?;

        if !body.active {
            return Err(AuthError::TokenInactive);
        }

        let claims = self.normalize(body)?;             // RFC 7662 → Claims; per-backend variations live here

        // Per ADR-021 — JWS and introspection enforce identical token semantics.
        validate_audience(&claims, &self.expected_audience)?;
        validate_token_type(&claims)?;
        validate_exp_nbf(&claims, SystemTime::now())?;

        let ttl = compute_ttl(&claims, self.config.cache_ttl_max);
        self.cache.insert(hash, claims.clone(), ttl);
        Ok(claims)
    }
}

Cache semantics

  • Key: [u8; 32] SHA-256 of the raw bearer. Hashing happens before any introspection call so the raw token never sits in the cache as a key.

  • Value: CachedClaims { claims: Claims, expires_at: Instant }.

  • TTL: min(claims.exp - now, config.cache_ttl_max_seconds). Default cache_ttl_max_seconds = 60.

  • Eviction: lazy on get (compare Instant::now() to expires_at) + a background task that sweeps when entries exceed max_entries (default 10_000). Both are required — the background sweep prevents the cache from holding unbounded expired entries between gets; the lazy check prevents returning expired entries between sweeps.

  • Outage behavior: with serve_on_outage: false (default), a failed introspection call (any error from the IdP) returns AuthError::IntrospectionUnreachable and the auth middleware emits 401. With serve_on_outage: true, cache hits continue to serve until their TTL expires even when the IdP is unreachable.

  • Concurrent introspection of the same bearer: optional singleflight deduplication (Arc<DashMap<TokenHash, Arc<Notify>>> keyed on in-flight hashes) is captured as a Step 2 nice-to-have. Without it, N simultaneous requests for the same uncached token fan out to N introspection calls; with it, only the first hits the IdP and the rest wait on the result.

Boot-time wiring

bootstrap::init_auth in crates/craig-api/src/bootstrap.rs switches on ServiceSettings::token_validation_mode:

  • Jws → build JwksProvider only; wrap as Arc<dyn ClaimsExtractor>. Behavior identical to pre-Plan-F.

  • Introspect → build IntrospectionClient (requires OidcServiceToken from BootstrapResult); wrap as Arc<dyn ClaimsExtractor>.

  • Auto (default) → build both, wrap an AutoExtractor.

Hard-fail at boot in these cases with operator-actionable error messages:

  • Introspect or Auto mode but OidcServiceToken is None — points operators at CRAIG_<SVC>__CLIENT_ID/SECRET (Plan E env vars).

  • Introspect mode but neither discovery advertises introspection_endpoint nor CRAIG_<SVC>__INTROSPECTION_ENDPOINT is set — names the override env var.

Cross-cutting invariants

These must hold regardless of validation mode. Codify in inline tests across Steps 2 + 3.

  1. Claim semantics parity — JWS and Introspect paths surface identical Claims for the same logical bearer. Audience, type, expiry, nbf, sub, preferred_username, email, realm_access.roles, aud, azp all populate consistently.

  2. Actor JWT lifting symmetry — claims.actor is set via verify_actor_token() post-claims-extraction, regardless of how the outer bearer was extracted. Actor JWTs are always JWS (CRAIG signs with its own ES256 keypair).

  3. is_service() / service_id() predicate parity — both paths populate realm_access.roles (via the optional CRAIG_IDENTITY_ROLES_CLAIM_PATH mapping for non-Keycloak shapes) so service-caller predicates yield identical answers. The deferred follow-up tracked in idp-integration.adoc rolls into Step 2’s normalization layer.

  4. Audit log column population — actor_service + actor_user_sub columns (Plan E Step 11) populate from EventEnvelope.source_service + event_parsing::extract_actor_user_sub. Both fields derive from Claims, so introspection mode doesn’t alter audit attribution.

  5. Service-to-service flow symmetry — when craig-web makes an outbound client_credentials + X-Craig-Actor call to craig-cases, and craig-cases is in Introspect mode, craig-cases introspects the craig-web service token via its own OidcServiceToken (acting as craig-cases). The introspection response carries azp / role information identifying craig-web; normalization populates Claims.azp so claims.is_service() returns true.

  6. Outbound + inbound decoupling — OidcServiceToken (outbound) and IntrospectionExtractor (inbound) are independent. A service in Introspect inbound mode still emits outbound tokens via the same client_credentials path Plan E established.

Steps

Step 1: Plan body + ADR-029 (Proposed)

Files: docs/modules/ROOT/pages/plans/introspection-validation-mode.adoc, docs/modules/ROOT/pages/adrs/adr-029-introspection-validation.adoc (new), docs/modules/ROOT/nav.adoc, CHANGELOG.adoc

Docs-only MR. Rewrites the stub plan body into the full content (this file). Drafts ADR-029 in Proposed status with sections per writing-adrs.adoc: Status, Context, Decision, Consequences, Open questions, Alternatives considered. Moves Plan F from nav Planned → Active. Adds ADR-029 to nav ADRs index. CHANGELOG entry under == Unreleased.

Step 2: Introspection primitives (no service wiring)

Files: crates/craig-auth/src/oidc_discovery.rs, crates/craig-auth/src/introspection.rs (new), xtask/src/cmd/identity/verify.rs, crates/craig-auth/Cargo.toml (wiremock dev-dep)

  • OidcDiscoveryDoc: add introspection_endpoint: Option<String> with #[serde(default)]. Extend the split-DNS rewriter to include this endpoint alongside jwks_uri + token_endpoint.

  • New module introspection.rs: IntrospectionClient + TokenClaimsCache + RFC 7662 response normalization. Per-claim validation parity with the JWS path. SHA-256 token hashing. Instant-clocked TTL. Background eviction task.

  • xtask identity verify introspection probe arm: when the discovery doc advertises introspection_endpoint, sentinel-probe with token=00000000 and assert RFC 7662 response shape (active: false expected); reports Inconclusive when the endpoint isn’t advertised.

  • Inline #[cfg(test)] tests via wiremock = "0.6" (workspace dep): happy-path mapping, active: false, expired exp, cache hit avoids second HTTP call, eviction on max_entries overflow.

  • No changes to auth_middleware, AuthLayer, or service bootstraps.

Step 3: AuthLayer trait refactor + Settings + bootstrap wiring + ADR-029 Accepted

Files: crates/craig-auth/src/middleware.rs, crates/craig-auth/src/jwks.rs, crates/craig-auth/src/lib.rs, crates/craig-common/src/settings.rs, crates/craig-api/src/bootstrap.rs, crates/craig-common/src/metrics.rs, docs/modules/ROOT/pages/adrs/adr-029-introspection-validation.adoc

  • Define trait ClaimsExtractor in middleware.rs. Refactor AuthLayer from { provider: JwksProvider, …​ } to { extractor: Arc<dyn ClaimsExtractor>, actor_registry: …​ }.

  • Implement ClaimsExtractor for JwksProvider (delegate to existing validate_token).

  • Add IntrospectionExtractor impl wrapping IntrospectionClient.

  • Add AutoExtractor { jws, introspection } dispatching on token-shape (3 segments = JWS; 5 = JWE; no dots = opaque).

  • ServiceSettings gains token_validation_mode: TokenValidationMode { Jws | Introspect | Auto } (default Auto) + 5 introspection knobs: introspection_cache_ttl_seconds (default 60), introspection_cache_max_entries (default 10000), introspection_serve_on_outage (default false), introspection_endpoint: Option<String> (operator override), introspection_auth_method: IntrospectionAuthMethod { Basic | Bearer } (default Basic).

  • init_auth dispatches on mode; hard-fails the two operator-error cases described above.

  • craig_common::metrics gains 4 counters/histograms: introspection_requests_total{result}, introspection_cache_hits_total, introspection_request_duration_seconds, introspection_cache_size.

  • ADR-029 status: Proposed → Accepted.

  • Inline tests cover AutoExtractor dispatch matrix, Settings SerDe, bootstrap mode selection, hard-fail messages.

Step 4: Devstack Kanidm profile + xtask identity render --backend kanidm adapter

Files: xtask/src/cmd/identity/render/kanidm.rs (new), xtask/src/cmd/identity/render.rs, xtask/src/cmd/dev.rs, xtask/src/docker.rs, docker-compose.yml, devstack/kanidm/server.toml (new), devstack/kanidm/README.md (new), devstack/kanidm/bootstrap.sh (new)

  • New render::kanidm module mirrors the zitadel.rs / authentik.rs shape. Kanidm provisions via CLI (kanidm system oauth2 create …​); the rendered output is a shell script that operators run against a Kanidm instance authenticated with an admin token. No admin API client — per ADR-026, CRAIG renders the operator’s bootstrap recipe but never integrates with Kanidm’s native admin API at runtime.

  • RenderBackend::Kanidm variant + dispatch.

  • docker-compose.yml: kanidm service under identity-multibackend profile (single-container, embedded sqlite, no Postgres dep). Cert auto-gen via kanidmd cert-generate at compose-up.

  • PORT_MAPPINGS: add ("kanidm", 8443, 8443) + ("kanidm", 3636, 3636).

  • multibackend_up/down: include kanidm. New bootstrap_kanidm() helper recovers admin password idempotently via docker exec kanidmd recover-account admin.

Step 5: Integration tests + ZITADEL-opaque-mode coverage

Files: crates/craig-test-lib/src/lib.rs, crates/craig-test-lib/tests/identity_multibackend.rs

  • identity_multibackend_available(): extend probe to include Kanidm discovery URL; deadline raised to 120s.

  • New tests:

    • kanidm_full_introspection_round_trip — discovery → client_credentials → introspection-validate → assert active=true + claims shape. End-to-end proof the introspection path works against Kanidm v1.10.1.

    • zitadel_opaque_mode_introspection — provisions a Machine User with ACCESS_TOKEN_TYPE_BEARER (opaque), mints, introspects, validates. Confirms the architecture covers ZITADEL’s default token type, not just !268’s JWT-pinned path.

  • Coverage of cache hits, fail-closed on injected 5xx, serve-on-outage preserves cached claims.

Step 6: Docs reframe + ADR amendments + plan completion audit + archive

Files: docs/modules/ROOT/pages/idp-integration.adoc, docs/modules/ROOT/pages/adrs/adr-026-idp-neutral-identity.adoc, docs/modules/ROOT/pages/adrs/adr-028-service-identity.adoc, docs/modules/ROOT/pages/plans/introspection-validation-mode.adoc, docs/modules/ROOT/pages/plans/archive.adoc, docs/modules/ROOT/pages/test-coverage-scorecard.adoc, docs/modules/ROOT/nav.adoc, .claude/CLAUDE.md, CHANGELOG.adoc

  • idp-integration.adoc: Tested-backends list reframed as a JWS/introspection matrix. Kanidm row added (verified). ZITADEL-opaque-mode row added alongside JWT-mode. New env vars documented.

  • ADR-026 + ADR-028: revision notes extending the IdP-neutral claim to introspection-validating backends.

  • Plan body Status table: all rows Complete. Plan moves Active → Archive.

  • test-coverage-scorecard.adoc: Plan F row added.

  • .claude/CLAUDE.md Phase Status update (final test count delta + MR list).

  • Plan-completion-audit subagent per delivery-protocol.md.

Files Touched

File Step Action

docs/modules/ROOT/pages/plans/introspection-validation-mode.adoc

1, 6

Rewrite stub → full body; status table Complete at end

docs/modules/ROOT/pages/adrs/adr-029-introspection-validation.adoc

1, 3

New; Proposed → Accepted

docs/modules/ROOT/nav.adoc

1, 6

Plan F Planned→Active→Archive; ADR-029 entry

crates/craig-auth/src/oidc_discovery.rs

2

introspection_endpoint: Option<String> + split-DNS rewrite

crates/craig-auth/src/introspection.rs

2

New — client + cache + RFC 7662 → Claims normalization

xtask/src/cmd/identity/verify.rs

2

Introspection probe arm

crates/craig-auth/src/middleware.rs

3

trait ClaimsExtractor + AuthLayer refactor + 3 impls

crates/craig-auth/src/jwks.rs

3

JwksProvider implements ClaimsExtractor

crates/craig-common/src/settings.rs

3

token_validation_mode + 5 introspection knobs

crates/craig-api/src/bootstrap.rs

3

init_auth dispatches on mode; hard-fails missing prerequisites

crates/craig-common/src/metrics.rs

3

4 introspection metrics

xtask/src/cmd/identity/render/kanidm.rs

4

New — render adapter

xtask/src/cmd/identity/render.rs

4

RenderBackend::Kanidm variant + dispatch

xtask/src/cmd/dev.rs, xtask/src/docker.rs

4

Kanidm in multibackend up/down + ports

docker-compose.yml

4

Kanidm service under identity-multibackend

devstack/kanidm/

4

New — server.toml + README + bootstrap.sh

crates/craig-test-lib/src/lib.rs, …​/tests/identity_multibackend.rs

5

Kanidm + ZITADEL-opaque integration tests

docs/modules/ROOT/pages/idp-integration.adoc

6

Tested-backends matrix reframe

docs/modules/ROOT/pages/adrs/adr-026-idp-neutral-identity.adoc, …​/adr-028-service-identity.adoc

6

Revision notes

docs/modules/ROOT/pages/plans/archive.adoc

6

Plan F row added

docs/modules/ROOT/pages/test-coverage-scorecard.adoc

6

Plan F row added

.claude/CLAUDE.md

6

Phase Status update

CHANGELOG.adoc

every step

One entry per MR under == Unreleased

Verification

After Step 5 lands, the end-to-end verification is:

# Local battery
cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo build --workspace --all-features
cargo nextest run -p craig-auth        # introspection unit tests (wiremock)
cargo nextest run -p xtask             # render dispatcher + Kanidm adapter inline tests

# End-to-end across all four backend modes
cargo xtask dev multibackend-up         # Authentik + ZITADEL + Kanidm + Redis dep
# wait ~120s for warmups (Authentik ~90s, ZITADEL ~30s, Kanidm ~10s)
cargo nextest run -p craig-test-lib --test identity_multibackend
cargo xtask dev multibackend-down

# JWS mode unchanged (byte-identical to today)
CRAIG_CASES__TOKEN_VALIDATION_MODE=jws cargo xtask dev start

# Introspection mode against Kanidm
CRAIG_CASES__TOKEN_VALIDATION_MODE=introspect \
  CRAIG_CASES__OIDC_ISSUER=https://kanidm.local:8443/oauth2/openid/craig-cases \
  CRAIG_CASES__CLIENT_ID=craig-cases \
  CRAIG_CASES__CLIENT_SECRET=<from-kanidm-show-basic-secret> \
  cargo run -p craig-cases

Step-level verification is in each step’s section above.

Expected workspace test growth: ~40-50 new tests (15 in Step 2 wiremock + xtask verify probe, 10 in Step 3 dispatch + Settings + bootstrap, 10 in Step 4 Kanidm render inline, 3-5 in Step 5 integration). Today’s count is ~1962 (post-!268). Plan F should land at ~2000-2010. The Step 6 CHANGELOG entry reconciles the actual delta with arithmetic per Reconcile test-count deltas.

Risks

Risk Likelihood Mitigation

Trait refactor (Step 3) breaks downstream callers of AuthLayer

Low

AuthLayer is constructed in bootstrap.rs, consumed by auth_middleware. Grep workspace before refactor. The Arc<dyn ClaimsExtractor> surface change is internal to craig-auth + craig-api.

RFC 7662 response shape divergence across IdPs causes silent claims-mapping bugs

Medium

Step 2 normalization layer has explicit per-claim mapping with tests for each backend’s response shape. Step 5 integration tests catch live regressions.

Auto-mode misdetects token shape and dispatches wrong path

Low

JWS = 3 segments, JWE = 5, opaque = no dots; collision space is empty. Step 3 inline test matrix covers all three shapes plus malformed inputs (4 dots, empty, non-ASCII).

Performance regression when operators enable Introspect mode

High by construction

Cache amortizes (~95% hit rate at default TTL). Metrics surface the cost. ADR-029 Consequences documents the perf budget. Opt-in only — default Auto keeps JWS path for JWS issuers.

Cache TTL stale-revocation window leaks access

Medium

Document the window in ADR-029. Operators in compliance-strict deployments can lower TTL (1-5s) at the cost of cache hit rate. Future CAEP push-revocation closes this.

IdP outage in Introspect mode → CRAIG-wide auth failure

Medium

Default serve_on_outage: false matches Plan E’s posture. Operators can opt-in true for resilience at cache-correctness cost. ADR-029 names this tradeoff explicitly.

Auto-mode operator confusion ("why is my IdP slow some requests?")

Medium

At boot, log the detected token shape from the first 10 inbound requests so operators can correlate. Document detection rules in idp-integration.adoc.

CI infra failure on the 6-MR sequence (recurring 2xlarge disk-OOM)

High by historical pattern

Per-step force-merge fallback documented in feedback_force_merge_runbook.md. Pre-push hook is the authoritative quality gate.

Migration / rollout / rollback

  • Migration impact on existing deployers: zero. Default mode is Auto. Auto dispatches to JWS for 3-segment tokens (what Keycloak/authentik/ZITADEL-JWT-mode issue today). Existing deployments observe no behavioral change.

  • Opt-in path for adding Kanidm: set CRAIG_<SVC>OIDC_ISSUER to the Kanidm RS issuer URL + ensure CRAIG_<SVC>CLIENT_ID/SECRET are set (Plan E env vars). No code-side change.

  • Rollback path if a deployer hits an issue with Auto mode: set CRAIG_<SVC>__TOKEN_VALIDATION_MODE=jws to revert per service. Per-service granularity supports gradual rollback.

  • Forward path to opaque/encrypted-token IdPs: CRAIG_<SVC>__TOKEN_VALIDATION_MODE=introspect pins the path explicitly.

Documentation Updates

  • docs/modules/ROOT/pages/adrs/adr-029-introspection-validation.adoc — NEW (Step 1 Proposed; Step 3 Accepted)

  • docs/modules/ROOT/pages/idp-integration.adoc — Tested-backends list reframed; env vars added (Step 6)

  • docs/modules/ROOT/pages/adrs/adr-026-idp-neutral-identity.adoc — revision note (Step 6)

  • docs/modules/ROOT/pages/adrs/adr-028-service-identity.adoc — revision note (Step 6)

  • docs/modules/ROOT/pages/test-coverage-scorecard.adoc — Plan F row (Step 6)

  • docs/modules/ROOT/nav.adoc — Plan F nav move + ADR-029 entry (Steps 1, 6)

  • .claude/CLAUDE.md — Phase Status update (Step 6)

  • CHANGELOG.adoc — entry per MR under == Unreleased (every step)

  • .claude/docs/services.md — N/A (Plan F changes no endpoints)

Errata

Plan deviations documented per delivery-protocol.md:

  • E-01 (Step 3, 2026-05-13): Prometheus metrics deferred from Step 3. The plan body originally listed "Metrics counters + histograms emitted" as a Step 3 deliverable (introspection_requests_total{result}, introspection_cache_hits_total, introspection_request_duration_seconds, introspection_cache_size). Step 3 deferred these because: (a) they require separate craig-common::metrics plumbing on a different axis than the validation refactor, and (b) Step 5’s integration tests use wiremock::expect(N) to assert request-count behavior, giving us the same regression guarantee without the metrics pipeline. Captured for a follow-up MR after Plan F archives. Step 3 description in the Status table was updated accordingly.

    Resolved 2026-05-15 (closes #410). All 4 instruments shipped via the new craig_common::metrics::introspection module: introspection_requests_total{result=ok\|unreachable\|bad_status\|parse_error\|token_inactive\|expired\|not_yet_valid\|audience_mismatch\|wrong_token_type}, introspection_cache_hits_total, introspection_request_duration_seconds, and the observable introspection_cache_size gauge sampled from TokenClaimsCache::len() via a closure registered in IntrospectionClient::new. Feature-gated on craig-common’s `otel flag with no-op stubs for builds that drop OTel. Instrumented at every terminal outcome in IntrospectionClient::extract + duration recorded around the IdP round-trip (cache hits skip the histogram).

  • E-02 (Step 5, 2026-05-13): full Kanidm + ZITADEL-opaque mint round-trips deferred. The plan body originally listed two integration tests asserting end- to-end client_credentials → introspect against Kanidm and ZITADEL- with-opaque-tokens. Both backends require operator-side bootstrap before a Machine User / OAuth2 RS exists to mint against — Kanidm needs idm_admin password recovery + interactive login + the bootstrap.sh rendered by Step 4; ZITADEL needs Terraform-or-Console Machine User provisioning. Same operator-bootstrap gap !268’s original ZITADEL test has. Step 5 ships the structural assertion (kanidm_status_and_token_endpoint_reachable): /status healthy, token endpoint dispatches the grant (not unsupported_grant_type like Dex), /oauth2/token/introspect endpoint exists. Future MR can add a sidecar init container that runs bootstrap.sh automatically after compose-up, unlocking the full mint test for both backends.

    Partially resolved 2026-05-15 (closes #411). Two new xtask commands wrap each backend’s first-boot operator workflow into a non-interactive provisioning step:

    • cargo xtask dev kanidm-bootstrap — runs kanidmd recover-account idm_admin against the kanidm server container, parses the rotated password from stdout, then logs in + provisions the 9 CRAIG OAuth2 resource servers in a kanidm/tools:1.10.1 container. Captures the basic secrets to devstack/kanidm-secrets.env. Idempotent — re-runs skip already-existing RSes.

    • cargo xtask dev zitadel-bootstrap — reads the firstinstance Machine User PAT (provisioned via ZITADEL_FIRSTINSTANCE_ORG_MACHINE_*
      ZITADEL_FIRSTINSTANCE_PATPATH env vars at compose-up) from a bind- mounted host dir, then uses it to provision a craig-cases Machine User in opaque-token mode + a craig Project + a craig-cases- introspecter API application. Writes secrets to devstack/zitadel-secrets.env. Idempotent.

Integration tests:

  • kanidm_full_introspection_round_trip (NEW) — full mint → JWE shape assertion → introspect → active=true + RFC 7662 claims subset (sub / client_id / aud / token_type / exp). Asserts the complete Plan F invariant for Kanidm.

  • zitadel_opaque_mode_client_credentials_mint (NEW) — mints a client_credentials token + asserts JWE shape (proves opaque-mode round-trip is reachable from a cold multibackend-up). Introspection-returns-active=true is not yet asserted — ZITADEL’s introspect-by-API-app model requires additional Project-level role
    user-grant chain configuration; with the current bootstrap the API app authenticates against the introspection endpoint but the token’s audience is not yet routed, so introspect returns active=false. Tracked as a sub-issue follow-up to fully close E-02.

Plus a small docker-compose change (zitadel service gains ZITADEL_FIRSTINSTANCE_ORG_MACHINE_* + ZITADEL_FIRSTINSTANCE_PATPATH env vars + a bind-mount on ./devstack/zitadel-bootstrap) and a kanidm renderer fix (group create --description → group create # comment to match Kanidm 1.10.1 CLI syntax — drops the inline --description flag that was rejected as unexpected argument).

+ Resolved 2026-08-12 (closes #415). The 2026-05-15 hypothesis ("additional Project-level role + user-grant chain configuration") turned out WRONG — no provisioning step was missing at all. Settled live against ZITADEL v4.15.0: introspection by the API app answers active=true exactly when the token’s aud contains the app’s project, and that routing is requested at mint via the urn:zitadel:iam:org:project:id:<projectId>:aud scope. The pre-#415 test minted with bare scope=openid, so the audience was never routed; no user grant, projectRoleAssertion, or JWT access-token type is involved. As-built: cargo xtask dev zitadel-bootstrap now writes CRAIG_ZITADEL_PROJECT_ID + CRAIG_ZITADEL_MINT_SCOPE into devstack/zitadel-secrets.env, and zitadel_full_introspection_round_trip (renamed from zitadel_opaque_mode_client_credentials_mint) mints with that scope, keeps the NEGATIVE control in-test (a bare-openid mint must introspect active=false), then asserts the full round-trip — active=true + the RFC 7662 canonical claims subset (sub / client_id / aud-contains-project / token_type / exp) — the complete Plan F invariant for ZITADEL, mirroring kanidm_full_introspection_round_trip. E-02 is fully closed for both backends.

References

  • RFC 7662 — OAuth 2.0 Token Introspection

  • RFC 9068 — JWT Profile for OAuth 2.0 Access Tokens (CRAIG’s current model)

  • RFC 9449 — DPoP (forward-compat consideration; out of scope)

  • RFC 8705 — mTLS for OAuth (forward-compat consideration; out of scope)

  • ADR-021 — JWT Validation Policy (audience / type / expiry parity rules that Step 2 inherits)

  • ADR-026 — IdP-Neutral Identity

  • ADR-028 — Service Identity + On-Behalf-Of

  • ADR-029 — Introspection-Mode Token Validation (filed in Step 1)

  • Multi-Backend Identity Integration Tests — predecessor plan; Dex strip (!267) + ZITADEL JWT-mode (!268)

  • Plan E — Service Identity + On-Behalf-Of — JWS-only baseline this plan extends

Edit this page · latest