ADR-029: Introspection-Mode Token Validation

On this page

Status

Accepted (2026-05-13). Plan F Step 3 (!271) ships the runtime refactor that gives this ADR effect: trait ClaimsExtractor with three impls (JwsExtractor, IntrospectionExtractor, AutoExtractor); per-service token_validation_mode env-var dispatch in bootstrap; six new ServiceSettings fields; hard-fail boot conditions for missing prerequisites; classifier covering JWS / JWE / opaque / malformed token shapes.

Originally filed Proposed 2026-05-12 alongside Plan F Step 1 (!269).

Context

ADR-026 establishes CRAIG’s IdP-neutrality stance: any OIDC-compliant issuer can authenticate CRAIG users without code changes. ADR-028 tightens that to "any OIDC backend with `client_credentials`" — service identity (Plan E) made that grant load-bearing, and backends without it (Dex 2.45.1 being the worked example) can’t host CRAIG.

Two months of bringing up multi-backend testing (idp-multibackend-tests.adoc — Authentik + ZITADEL via !266 / !267 / !268) surfaced a second narrowing: CRAIG only validates JWS-signed access tokens. Today’s path is craig-auth::middleware::auth_middlewareJwksProvider::validate_tokenjsonwebtoken::decode::<Claims> at crates/craig-auth/src/jwks.rs:197. It expects a 3-segment JWS (header.payload.signature) decodable against the issuer’s JWKS.

That excludes a real and growing share of the OAuth ecosystem:

  • Kanidm v1.10.1 (Rust-aligned, would be the natural fit for CRAIG’s stack) issues client_credentials access tokens as JWE (alg=A128KW, enc=A128GCM, 5-segment). Source: kanidmd/lib/src/idm/oauth2.rs:1861 calls jwe_a128gcm_encrypt(&access_token_data, ct) unconditionally. The intended validation path is the introspection endpoint (RFC 7662). No configuration knob to switch to JWS. Sandbox-confirmed 2026-05-12.

  • ZITADEL v4.15.0 defaults to opaque (reference) tokens. Operators can pin access_token_type = ACCESS_TOKEN_TYPE_JWT per machine user (which !268 does), but that’s working around the IdP’s default. Tokens issued in default mode are opaque strings validated only via /oauth/v2/introspect.

  • Keycloak supports "lightweight access tokens" (a smaller, opaque format) — operators in regulated environments may pin this for log-volume / data-minimization reasons.

  • authentik offers an encrypted-token mode for high-confidentiality deployments.

Introspection isn’t a Kanidm-specific oddity. It’s the other mature half of the OAuth design space: tokens are deliberately opaque to relying parties; validation happens at the issuer; revocation is centrally controlled. RFC 7662 exists for this. Each named backend gives operators a real choice between the two modes.

CRAIG’s current architecture forces the JWS half — which is the right default (fast, locally-decoded, resilient to IdP outages) but excludes operators whose IdP doesn’t take that path. Plan F adds the introspection strategy as a peer of the JWS strategy, broadening CRAIG’s neutrality claim to "any OIDC backend with client_credentials, validated locally (JWS) or via introspection (RFC 7662)."

The decision space has three sub-questions: how to express the dispatch in code (trait vs enum vs either-pattern), what the default validation mode should be (JWS-only vs explicit-operator-pick vs auto-detect), and how the system should behave when the introspection endpoint is unreachable (fail-closed always vs cache-fallback vs fail-open).

Decision

Three coupled decisions:

  1. Dispatch as a trait ClaimsExtractor in craig-auth, with three impls today (JwsExtractor, IntrospectionExtractor, AutoExtractor) and room for future extractors (DPoP, mTLS-bound, public-key JWE, push-revocation).

  2. Default mode is Auto — at runtime, dispatch on token shape: 3 segments → JWS (decode locally); 5 segments → JWE (introspect); no dots → opaque reference (introspect). Operators can pin to Jws or Introspect explicitly via per-service env var CRAIG_<SVC>__TOKEN_VALIDATION_MODE for predictable cost.

  3. Fail-closed by default on introspection-endpoint failures; opt-in tunable CRAIG_<SVC>__INTROSPECTION_SERVE_ON_OUTAGE=true lets cache entries continue serving during an IdP outage for a bounded hard-grace window of one additional cache_ttl_max_seconds past their TTL (worst case 2 * cache_ttl_max_seconds since the last successful introspection, since the TTL itself is capped at the same knob — #909 corrected earlier wording that understated this exposure as "until their TTL").

Cache layer: per-process dashmap::DashMap<TokenHash, CachedClaims>. Key: [u8; 32] SHA-256 of the raw bearer (hashing happens before any introspection call, so the raw token never sits in memory as a key). Clock: std::time::Instant (monotonic). TTL: min(claims.exp - now, config.cache_ttl_max_seconds) with default cache_ttl_max_seconds=60. Default max_entries=10_000. Eviction is lazy on get plus opportunistic batch eviction on insert-at-cap (#1535 corrected earlier wording that claimed a background sweep task — none is spawned; bounded memory holds via the entry cap).

Outbound auth for the introspection call: reuse the per-service OidcServiceToken source from Plan E Step 4. CRAIG presents itself to the IdP as the service introspecting on a worker’s behalf.

Cross-cutting invariants (codified in Plan F’s design section + step-2/3 inline tests): claim-semantics parity between JWS and Introspect paths (audience, type, expiry, nbf, sub, preferred_username, email, realm_access.roles, aud, azp); actor JWT lifting (X-Craig-Actor) is unchanged — CRAIG’s own ES256-signed tokens are validated against the peer-JWKS map regardless of how the outer bearer was extracted.

Logging discipline: the raw bearer never appears in logs. Diagnostic correlator is the first 8 bytes of the SHA-256 hex digest. Wrong-credential errors quote the token hash, not the token itself.

Consequences

Unlocks:

  • Kanidm becomes a viable CRAIG backend (currently excluded by the JWE issue described above).

  • ZITADEL’s default opaque-token mode becomes supportable (today only the JWT-mode pin in !268 works).

  • Keycloak’s lightweight access tokens become supportable.

  • authentik’s encrypted-token mode becomes supportable.

  • Per-service per-deployment choice between cost-models: operators in low-latency contexts keep JWS; operators in high-confidentiality contexts can choose introspection (centralized revocation, opaque tokens in logs).

Locks in:

  • Arc<dyn ClaimsExtractor> indirection in AuthLayer. Trivial perf cost; gives forward extensibility for DPoP / mTLS / public-key JWE.

  • Runtime dependency on the IdP for every protected request in Introspect mode (amortized by cache; visible in metrics; tunable via cache knobs + serve_on_outage). Operators picking that mode see a different SLO than JWS-mode operators.

  • Per-process cache state. Plan F explicitly does not coordinate across replicas — each builds its own. Acceptable because the cache only affects performance, not correctness; revocation latency is bounded by TTL regardless.

Follow-on work implied:

  • Plan F Step 6 reframes idp-integration.adoc from a "JWS only" matrix to "JWS or introspection" matrix.

  • ADR-026 + ADR-028 gain revision notes in Step 6 sharpening the IdP-neutral claim to span both validation modes.

  • cargo xtask identity verify gains an introspection-endpoint probe arm in cargo xtask validate (Step 2 deliverable).

  • Operators monitoring p99 latency need the new metrics counters/histograms (introspection_requests_total{result}, introspection_cache_hits_total, introspection_request_duration_seconds, introspection_cache_size) added in Step 3.

Does not affect:

  • ADR-021's strict-validation rules. Both paths enforce identical audience / token-type / expiry / nbf semantics. The introspection path applies the same gates after parsing the RFC 7662 response.

  • X-Craig-Actor verification. Actor JWTs are CRAIG’s own ES256-signed tokens validated against the peer-JWKS map post-extraction. The IdP-side validation mode doesn’t affect this layer.

  • Plan E OidcServiceToken outbound. Outbound + inbound are decoupled; a service in Introspect inbound mode still emits outbound tokens via the same client_credentials path.

  • Audit log actor_service + actor_user_sub column population — both derive from Claims, which is mode-agnostic.

Open questions

  • Push-based revocation (CAEP / SSE token-revocation streams). The cache’s TTL-bounded revocation latency is a known tradeoff (default 60s window between revoke + cache eviction). Backends that support a revocation push (CAEP is an emerging standard) could close that window. Plan F doesn’t ship push support; future work may. Trait abstraction leaves room.

  • Singleflight deduplication for concurrent introspection of the same bearer. Step 2 lists this as a nice-to-have (~50 LOC). Without it, N simultaneous requests for the same uncached token fan out to N introspection calls. With it, only the first hits the IdP. Decision deferred to Step 2 review — implementer’s call based on review-budget vs IdP-load-protection tradeoff.

  • CRAIG_IDENTITY_ROLES_CLAIM_PATH deferred follow-up (per idp-integration.adoc line 75). Some IdPs emit role claims outside realm_access.roles (Okta groups, Azure AD roles, etc.). Step 2’s RFC 7662 → Claims normalization layer is the natural home for this — same claim-path mapping applies on both JWS and introspection paths. Plan F Step 2 rolls this in.

  • Cache-key keyed-hash variant (HMAC-SHA-256 keyed by a service-local secret). Considered + rejected (see Alternatives considered) — the per-process cache makes cross-node correlation a non-threat. If a deployer later wants HMAC-keyed hashes, the swap is a one-line change in TokenClaimsCache.

Alternatives considered

Dispatch shape:

  • Enum (TokenValidator::Jws | Introspect) — Plan-E style. Static match arms; no dyn indirection. Rejected because adding a third mode later (DPoP, mTLS-bound, public-key JWE, push-revocation) requires touching the enum + every match site, whereas the trait pattern absorbs new modes by adding an impl. The dispatch cost of Arc<dyn> is negligible (one indirection per request; introspection latency dominates by ~5 orders of magnitude).

  • Either-pattern (AuthLayer { jws: Option<…​>, introspect: Option<…​> }). Carries both; dispatches on which is Some. Rejected because it introduces an exactly-one-set invariant the type system can’t express, and provides no advantage over the trait.

Default mode:

  • jws (preserve current behavior; opt-in to introspect). Rejected because it leaves the Kanidm / ZITADEL-opaque / Keycloak-lightweight / authentik-encrypted operators in the cold by default — they’d need to know to flip a config to even get the IdP working. Auto mode means new operators can start with their IdP’s native token format without per-service tuning.

  • Operator must pick explicitly (no default). Rejected because it breaks every existing devstack + CI run that doesn’t set the env var. The migration cost is high for a system where most deployers will pick Auto anyway.

Outage posture:

  • Fail closed always (no cache fallback). Rejected because a transient IdP blip becomes a CRAIG-wide outage for every protected request. The opt-in serve_on_outage tunable lets operators choose this strict posture when needed.

  • Fail open on transient errors, log + alert. Rejected because it’s a meaningful security weakening with no clear bound on "transient." Authentication failures should not silently succeed.

Cache key hashing:

  • HMAC-SHA-256 keyed by a service-local secret (defense against side-channel cache-key timing correlation across nodes). Rejected because the cache is per-process in-memory; cross-node correlation isn’t a threat at this layer. Adds a key-rotation surface that doesn’t earn its keep. If a deployer in a high-side-channel-risk context wants this later, the swap is a one-line change.

Cache scope:

  • Cross-replica sharing via Redis or similar. Rejected because it reintroduces an ops dep (Redis was deliberately excluded from CRAIG’s infrastructure per ADR-013) and adds a revocation-propagation correctness layer (cache invalidation across replicas under concurrent revocations). Per-process cache is correct because the only thing it affects is performance — revocation latency is bounded by TTL regardless of cache topology.

Cache clock:

  • SystemTime (wall-clock). Rejected because NTP corrections during normal operation cause spurious eviction or retention. Instant is monotonic and immune.

Edit this page · latest