ADR-024: Policy Engine on zen-engine + RMQ Cache Invalidation

AuthzEngine trait in new crates/craig-authz crate. Public surface: check(claims, resource_ref, action) for single-row predicates; auto_scope_list(claims, resource_type, action, jurisdiction) returning ListScope for SQL-filter scoping on LIST endpoints.

Engine wraps zen-engine’s `DecisionEngine. Each (jurisdiction, resource_type) pair maps to a JDM ruleset with name {jurisdiction}-authz-{resource} stored in craig-rules' existing rule_sets table.

Per-replica in-memory cache loaded at boot via bulk-fetch (GET /v1/rules/sets?prefix={jurisdiction}-authz-). Cache miss at request time → fail-closed deny + audit event.

Boot-load failure → service bails (analogous to platform-stab-2’s encryption-mode-required boot guard).

Cache invalidation: craig-rules publishes ruleset.changed events on its outbox when admin endpoints mutate policies. Consuming services bind exclusive auto-delete queues to a new craig.cache_invalidations topic exchange (via subscribe_exclusive). Receiving the event invalidates the local cache entry; next request fetches fresh.

TTL fallback (1-hour default; env-overridable via CRAIG_<SVC>__AUTHZ_POLICY_TTL_SECONDS) catches missed invalidation events. Per-replica jitter avoids thundering herd at TTL boundary.

No new DSL/parser/evaluator — reuses CRAIG’s house pattern

Real-time invalidation via RMQ; TTL fallback for partition recovery + missed events

Cross-service authz works without per-request HTTP hops (cache + denormalized ownership data per Plan A §D5)

Operational dashboards: authz.cache_miss + authz.cache_refreshed + authz.access_denied audit events reach audit_log via the injected sink (ADR-050 / #908; access_denied since #1063, volume-gated by the full-dimension aggregate with suppressed-count reporting — ADR-050 §Amendment #1063)

AuthzError::PolicyMissing is now source-attested: lazy_load distinguishes "the RulesetSource itself returned None`" (→ `PolicyMissing, 403) from a refresh fault (fetch/compile failure → new AuthzError::PolicyLoad, 500) — a fault can no longer masquerade as missing policy on any path (check, auto_scope_list, resolve_field_permission).

The trait gains two provided typed seams — check_typed / auto_scope_list_typed — surfacing AuthzError so craig-rules' bootstrap fallback admits on PolicyMissing ONLY (admin → any action; service caller → Read/List; every admit stages a rules.authz_fallback_admit event). The defaults report an opaque non-admitting deny; test doubles inherit them unchanged.

Degraded boot keeps the REAL source: boot_authz_engine no longer substitutes an empty InMemoryRulesetSource (which falsely attested universal absence and could never self-heal) — it degrades to ZenAuthzEngine::boot_degraded (empty cache, real source; self-heal via per-request lazy loads + the invalidation subscriber + the bounded post-boot warmup — the TTL task only re-fetches names already cached). With authz_require_full_coverage=true a double fetch failure refuses to boot, naming source availability as the remedy.

craig-security’s admin fallbacks (authz_check_or_admin_fallback / authz_list_or_admin_fallback) are retired outright: NO error class admits anyone, and a healthy engine’s Denied list verdict is authoritative even for admins. The seeded reference policies no longer wildcard service callers (rule_set → read/list; rule_evaluation → create only, the cases→rules evaluate relay).

claims.is_service is set at evaluation time only when NO actor was lifted (engine.rs — Plan E §9b), so the provisioned BFF never matches service rows; they exist solely for bare machine principals.

Removing a service row is fail-closed (trailing default-deny → 403 / empty list page) — but removal alone still lets a machine token carrying stray human roles fall through to worker rows.

The #1083 subsidy leading-DENY closes that: with hitPolicy: first, a service DENY row ahead of the worker rows is terminal for every bare machine token.

The engine now REQUIRES an Arc<dyn AuthzAuditSink> at construction (boot / boot_degraded); craig-bootstrap’s OutboxAuditSink stages each envelope on the host service’s transactional outbox in a one-shot tx, fire-and-forget. authz.cache_miss emits on every source-attested miss (never on PolicyLoad faults) with acting-worker attribution; authz.cache_refreshed emits from every refresh path with trigger attribution (rmq_event/ttl_refresh/boot/warmup/lazy_load). Full design + alternatives: ADR-050.

Evaluation budget. Every decision funnels through ONE serial !Send zen eval thread per service; a slow ruleset head-of-line blocked every queued caller indefinitely. evaluate now bounds the whole round-trip (channel send + queue-wait + eval) with a configurable budget (CRAIG_<SERVICE>__AUTHZ_EVAL_TIMEOUT_MS, default 5000 — the craig-rules #784 pattern). Elapse = typed fail-closed EvalTimeout → 503 (never a 403 that would read as a deny, never an unbounded hang). Eval-thread pooling remains a possible follow-up.

Negative absence cache. A source-attested-absent ruleset was re-fetched over HTTP and re-staged an authz.cache_miss audit event on EVERY request. Absences now cache for a short TTL (30s default): repeat lookups answer locally and the audit event stages once per TTL window. Bounded by construction (ruleset names derive from the closed jurisdiction x ResourceType space); a fresh seed becomes visible within one TTL even if the ruleset.changed event is missed.

Compile outside the lock. refresh_entry parsed + compiled JDM under the cache write guard (blocking all readers per refresh), and bulk_refresh held it across ~50 compiles. Compilation now happens before the guard; the lock holds only for map mutation.

The JS bound is owned at boot, fleet-wide. zen’s function_timeout_millis is ONE process-global atomic (default 5 s, compiled in) sampled by every evaluation in the process; before this amendment only craig-rules' process set it. Every service now publishes CRAIG_<SERVICE>AUTHZ_FUNCTION_TIMEOUT_MS (default min(5000, authz_eval_timeout_ms); explicit values validated 1..=authz_eval_timeout_ms at settings load — a JS node outliving the eval budget turns a clean typed runtime error into the #1126 EvalTimeout 503) through boot_authz_engine into craig_authz::apply_function_timeout. Single-owner-per-process rule: craig-rules passes None through that seam — its CRAIG_RULESFUNCTION_TIMEOUT_MS (#1046, applied earlier in its boot) owns the one atomic in that process, and the shared step must never overwrite it (last-writer-wins). Process inventory: craig-rules runs TWO zen engines (RulesEngine + its embedded authz engine) under its one bound; the other seven stateful services run exactly one authz engine each; craig-web / craig-intake / craig-intake-keyring link no zen at all.

The wedge deadman, extracted and applied. The #1046 primitive (in-flight stamp for exactly the zen-future window — queue wait never stamps; a watchdog sampling every 5 s; threshold max(10 × eval budget, 60 s); fire-once; no in-process respawn) now lives in craig_common::eval_deadman — extraction over clone-twinning because the deadman is a supervision primitive (the #1229 precedent put the supervisor registry in craig-common) and a twin would either trip the B8 duplicate-body budget or diverge to evade it. The authz eval loop stamps the RULESET NAME as its diagnostic (rules stamps the rule_evaluations audit id); ZenAuthzEngine::eval_thread_ended widens from death-only (#1228, previously documented only in craig-bootstrap + ADR-061) to death-OR-proven-wedge via the shared death_or_wedge future, feeding the SAME authz-eval Critical registration — a wedge now rides ADR-061 fail-fast (supervisor shutdown → nonzero exit → orchestrator restart). Design rationale
rejected alternatives (in-process respawn, N-thread pool, consecutive-timeout proxy): ADR-006 §Amendment #1046, unchanged.

JS boundedness pinned, write-path refusal rejected. Authz rulesets today are table/expression-based (no functionNode in any fixture), and the ruleset write path is deliberately a compile-only shape check — a name-convention-scoped functionNode refusal would be fragile and add nothing once the bound is owned: a JS node in an authz ruleset is legal-but-bounded by construction. Pinned by v2_function_node_is_interrupted_at_the_published_authz_bound (a 250 ms published bound interrupts a live spin, thread survives) and the end-to-end wedge proof wedge_deadman_fires_during_a_js_spin_while_the_thread_lives.

ADR-022 — Outbox/inbox patterns this engine reuses

ADR-023 — Architectural rationale

ADR-050 — Audit-event emission port/adapter design (#908)