ADR-029: Introspection-Mode Token Validation
On this page
Status
Accepted (2026-05-13). Plan F Step 3 (!271) ships the runtime refactor
that gives this ADR effect: trait ClaimsExtractor with three impls
(JwsExtractor, IntrospectionExtractor, AutoExtractor); per-service
token_validation_mode env-var dispatch in bootstrap; six new
ServiceSettings fields; hard-fail boot conditions for missing
prerequisites; classifier covering JWS / JWE / opaque / malformed token
shapes.
Originally filed Proposed 2026-05-12 alongside Plan F Step 1 (!269).
Plan F body: Introspection-Mode Token Validation plan.
Context
ADR-026 establishes CRAIG’s IdP-neutrality stance: any OIDC-compliant issuer can authenticate CRAIG users without code changes. ADR-028 tightens that to "any OIDC backend with `client_credentials`" — service identity (Plan E) made that grant load-bearing, and backends without it (Dex 2.45.1 being the worked example) can’t host CRAIG.
Two months of bringing up multi-backend testing (idp-multibackend-tests.adoc — Authentik + ZITADEL via !266 / !267 / !268) surfaced a second narrowing: CRAIG only validates JWS-signed access tokens. Today’s path is craig-auth::middleware::auth_middleware → JwksProvider::validate_token → jsonwebtoken::decode::<Claims> at crates/craig-auth/src/jwks.rs:197. It expects a 3-segment JWS (header.payload.signature) decodable against the issuer’s JWKS.
That excludes a real and growing share of the OAuth ecosystem:
-
Kanidm v1.10.1 (Rust-aligned, would be the natural fit for CRAIG’s stack) issues
client_credentialsaccess tokens as JWE (alg=A128KW, enc=A128GCM, 5-segment). Source:kanidmd/lib/src/idm/oauth2.rs:1861callsjwe_a128gcm_encrypt(&access_token_data, ct)unconditionally. The intended validation path is the introspection endpoint (RFC 7662). No configuration knob to switch to JWS. Sandbox-confirmed 2026-05-12. -
ZITADEL v4.15.0 defaults to opaque (reference) tokens. Operators can pin
access_token_type = ACCESS_TOKEN_TYPE_JWTper machine user (which !268 does), but that’s working around the IdP’s default. Tokens issued in default mode are opaque strings validated only via/oauth/v2/introspect. -
Keycloak supports "lightweight access tokens" (a smaller, opaque format) — operators in regulated environments may pin this for log-volume / data-minimization reasons.
-
authentik offers an encrypted-token mode for high-confidentiality deployments.
Introspection isn’t a Kanidm-specific oddity. It’s the other mature half of the OAuth design space: tokens are deliberately opaque to relying parties; validation happens at the issuer; revocation is centrally controlled. RFC 7662 exists for this. Each named backend gives operators a real choice between the two modes.
CRAIG’s current architecture forces the JWS half — which is the right default (fast, locally-decoded, resilient to IdP outages) but excludes operators whose IdP doesn’t take that path. Plan F adds the introspection strategy as a peer of the JWS strategy, broadening CRAIG’s neutrality claim to "any OIDC backend with client_credentials, validated locally (JWS) or via introspection (RFC 7662)."
The decision space has three sub-questions: how to express the dispatch in code (trait vs enum vs either-pattern), what the default validation mode should be (JWS-only vs explicit-operator-pick vs auto-detect), and how the system should behave when the introspection endpoint is unreachable (fail-closed always vs cache-fallback vs fail-open).
Decision
Three coupled decisions:
-
Dispatch as a
trait ClaimsExtractorincraig-auth, with three impls today (JwsExtractor,IntrospectionExtractor,AutoExtractor) and room for future extractors (DPoP, mTLS-bound, public-key JWE, push-revocation). -
Default mode is
Auto— at runtime, dispatch on token shape: 3 segments → JWS (decode locally); 5 segments → JWE (introspect); no dots → opaque reference (introspect). Operators can pin toJwsorIntrospectexplicitly via per-service env varCRAIG_<SVC>__TOKEN_VALIDATION_MODEfor predictable cost. -
Fail-closed by default on introspection-endpoint failures; opt-in tunable
CRAIG_<SVC>__INTROSPECTION_SERVE_ON_OUTAGE=truelets cache entries continue serving during an IdP outage for a bounded hard-grace window of one additionalcache_ttl_max_secondspast their TTL (worst case2 * cache_ttl_max_secondssince the last successful introspection, since the TTL itself is capped at the same knob — #909 corrected earlier wording that understated this exposure as "until their TTL").
Cache layer: per-process dashmap::DashMap<TokenHash, CachedClaims>. Key: [u8; 32] SHA-256 of the raw bearer (hashing happens before any introspection call, so the raw token never sits in memory as a key). Clock: std::time::Instant (monotonic). TTL: min(claims.exp - now, config.cache_ttl_max_seconds) with default cache_ttl_max_seconds=60. Default max_entries=10_000. Eviction is lazy on get plus opportunistic batch eviction on insert-at-cap (#1535 corrected earlier wording that claimed a background sweep task — none is spawned; bounded memory holds via the entry cap).
Outbound auth for the introspection call: reuse the per-service OidcServiceToken source from Plan E Step 4. CRAIG presents itself to the IdP as the service introspecting on a worker’s behalf.
Cross-cutting invariants (codified in Plan F’s design section + step-2/3 inline tests): claim-semantics parity between JWS and Introspect paths (audience, type, expiry, nbf, sub, preferred_username, email, realm_access.roles, aud, azp); actor JWT lifting (X-Craig-Actor) is unchanged — CRAIG’s own ES256-signed tokens are validated against the peer-JWKS map regardless of how the outer bearer was extracted.
Logging discipline: the raw bearer never appears in logs. Diagnostic correlator is the first 8 bytes of the SHA-256 hex digest. Wrong-credential errors quote the token hash, not the token itself.
Consequences
Unlocks:
-
Kanidm becomes a viable CRAIG backend (currently excluded by the JWE issue described above).
-
ZITADEL’s default opaque-token mode becomes supportable (today only the JWT-mode pin in !268 works).
-
Keycloak’s lightweight access tokens become supportable.
-
authentik’s encrypted-token mode becomes supportable.
-
Per-service per-deployment choice between cost-models: operators in low-latency contexts keep JWS; operators in high-confidentiality contexts can choose introspection (centralized revocation, opaque tokens in logs).
Locks in:
-
Arc<dyn ClaimsExtractor>indirection inAuthLayer. Trivial perf cost; gives forward extensibility for DPoP / mTLS / public-key JWE. -
Runtime dependency on the IdP for every protected request in
Introspectmode (amortized by cache; visible in metrics; tunable via cache knobs +serve_on_outage). Operators picking that mode see a different SLO than JWS-mode operators. -
Per-process cache state. Plan F explicitly does not coordinate across replicas — each builds its own. Acceptable because the cache only affects performance, not correctness; revocation latency is bounded by TTL regardless.
Follow-on work implied:
-
Plan F Step 6 reframes idp-integration.adoc from a "JWS only" matrix to "JWS or introspection" matrix.
-
ADR-026 + ADR-028 gain revision notes in Step 6 sharpening the IdP-neutral claim to span both validation modes.
-
cargo xtask identity verifygains an introspection-endpoint probe arm incargo xtask validate(Step 2 deliverable). -
Operators monitoring p99 latency need the new metrics counters/histograms (
introspection_requests_total{result},introspection_cache_hits_total,introspection_request_duration_seconds,introspection_cache_size) added in Step 3.
Does not affect:
-
ADR-021's strict-validation rules. Both paths enforce identical audience / token-type / expiry / nbf semantics. The introspection path applies the same gates after parsing the RFC 7662 response.
-
X-Craig-Actor verification. Actor JWTs are CRAIG’s own ES256-signed tokens validated against the peer-JWKS map post-extraction. The IdP-side validation mode doesn’t affect this layer.
-
Plan E
OidcServiceTokenoutbound. Outbound + inbound are decoupled; a service inIntrospectinbound mode still emits outbound tokens via the sameclient_credentialspath. -
Audit log
actor_service+actor_user_subcolumn population — both derive fromClaims, which is mode-agnostic.
Open questions
-
Push-based revocation (CAEP / SSE token-revocation streams). The cache’s TTL-bounded revocation latency is a known tradeoff (default 60s window between revoke + cache eviction). Backends that support a revocation push (CAEP is an emerging standard) could close that window. Plan F doesn’t ship push support; future work may. Trait abstraction leaves room.
-
Singleflight deduplication for concurrent introspection of the same bearer. Step 2 lists this as a nice-to-have (~50 LOC). Without it, N simultaneous requests for the same uncached token fan out to N introspection calls. With it, only the first hits the IdP. Decision deferred to Step 2 review — implementer’s call based on review-budget vs IdP-load-protection tradeoff.
-
CRAIG_IDENTITY_ROLES_CLAIM_PATHdeferred follow-up (peridp-integration.adocline 75). Some IdPs emit role claims outsiderealm_access.roles(Oktagroups, Azure ADroles, etc.). Step 2’s RFC 7662 →Claimsnormalization layer is the natural home for this — same claim-path mapping applies on both JWS and introspection paths. Plan F Step 2 rolls this in. -
Cache-key keyed-hash variant (HMAC-SHA-256 keyed by a service-local secret). Considered + rejected (see Alternatives considered) — the per-process cache makes cross-node correlation a non-threat. If a deployer later wants HMAC-keyed hashes, the swap is a one-line change in
TokenClaimsCache.
Alternatives considered
Dispatch shape:
-
Enum (
TokenValidator::Jws | Introspect) — Plan-E style. Static match arms; nodynindirection. Rejected because adding a third mode later (DPoP, mTLS-bound, public-key JWE, push-revocation) requires touching the enum + every match site, whereas the trait pattern absorbs new modes by adding an impl. The dispatch cost ofArc<dyn>is negligible (one indirection per request; introspection latency dominates by ~5 orders of magnitude). -
Either-pattern (
AuthLayer { jws: Option<…>, introspect: Option<…> }). Carries both; dispatches on which isSome. Rejected because it introduces an exactly-one-set invariant the type system can’t express, and provides no advantage over the trait.
Default mode:
-
jws(preserve current behavior; opt-in tointrospect). Rejected because it leaves the Kanidm / ZITADEL-opaque / Keycloak-lightweight / authentik-encrypted operators in the cold by default — they’d need to know to flip a config to even get the IdP working.Automode means new operators can start with their IdP’s native token format without per-service tuning. -
Operator must pick explicitly (no default). Rejected because it breaks every existing devstack + CI run that doesn’t set the env var. The migration cost is high for a system where most deployers will pick
Autoanyway.
Outage posture:
-
Fail closed always (no cache fallback). Rejected because a transient IdP blip becomes a CRAIG-wide outage for every protected request. The opt-in
serve_on_outagetunable lets operators choose this strict posture when needed. -
Fail open on transient errors, log + alert. Rejected because it’s a meaningful security weakening with no clear bound on "transient." Authentication failures should not silently succeed.
Cache key hashing:
-
HMAC-SHA-256 keyed by a service-local secret (defense against side-channel cache-key timing correlation across nodes). Rejected because the cache is per-process in-memory; cross-node correlation isn’t a threat at this layer. Adds a key-rotation surface that doesn’t earn its keep. If a deployer in a high-side-channel-risk context wants this later, the swap is a one-line change.
Cache scope:
-
Cross-replica sharing via Redis or similar. Rejected because it reintroduces an ops dep (Redis was deliberately excluded from CRAIG’s infrastructure per ADR-013) and adds a revocation-propagation correctness layer (cache invalidation across replicas under concurrent revocations). Per-process cache is correct because the only thing it affects is performance — revocation latency is bounded by TTL regardless of cache topology.
Cache clock:
-
SystemTime(wall-clock). Rejected because NTP corrections during normal operation cause spurious eviction or retention.Instantis monotonic and immune.