ADR-050: Authz Audit-Event Emission via an Injected Sink (AuthzAuditSink)

On this page

Status

Accepted (2026-07-18). Issue #908 (P2-medium, epic &64); the deferral that re-scoped #908 to this design is recorded in the archived comment-accuracy-remediation plan (Step 1, decided 2026-07-16).

Context

crates/craig-authz/src/audit.rs has carried unit-tested builders for authz.cache_miss / authz.cache_refreshed envelopes since Plan A §D14, with zero production call sites — the miss branches in check_typed / auto_scope_list_typed / resolve_field_permission and every refresh path only warn!-logged. Four code comments falsely claimed emission happened; the 2026-06-29 comment-accuracy audit filed that as its highest-priority finding (audit-trail gap on an access-control decision), the comments were corrected to the warn-only truth, and the wiring was deferred here.

Constraints the design must respect:

  • The engine is a pure decision library. craig-authz deliberately has no sqlx dependency (the ADR-047 boundary discipline); the documented intent — "stage on a caller’s outbox transaction" — was unreachable from inside it.

  • No transaction exists at emission time. Every handler checks authz BEFORE opening its DB transaction, and a denied request never opens one; the refresh paths (RMQ invalidation, TTL sweep, post-boot warmup) run on background tasks with no request context at all.

  • The AuthzError → ApiError bridge cannot emit. It is a synchronous From impl applied inside the engine, so the ~140 untyped-path handlers never see AuthzError, and a From impl can hold neither a tx nor context.

  • #786 coordination. PolicyMissing is now source-attested (absence the RulesetSource itself confirmed); refresh faults are PolicyLoad. The cache-miss event must keep that distinction: a fault must never masquerade as a miss. craig-rules additionally stages a service-level rules.authz_fallback_admit on every bootstrap fallback admit — the library event must coexist with it, not replace it.

  • All eight engine-hosting services (cases, security, rules, composition, exchange, placement, financial, reporting) already run the ADR-022 transactional outbox with a pool in scope before boot_authz_engine.

Decision

D1 — Port: AuthzAuditSink, a required constructor dependency

craig-authz defines the port and the engine cannot be constructed without it (mirrors the existing Arc<dyn RulesetSource> injection):

#[async_trait]
pub trait AuthzAuditSink: Send + Sync {
    /// Stage `envelope` for delivery. Infallible by contract: audit
    /// emission must never fail or block an authz decision — impls
    /// swallow their own failures (warn!-log, never propagate).
    async fn stage(&self, envelope: EventEnvelope);
}

ZenAuthzEngine::boot / boot_degraded take Arc<dyn AuthzAuditSink> as a required parameter — never an Option that silently no-ops. NoopAuditSink (explicit, for callers without an outbox) and RecordingAuditSink (test assertions) are exported alongside.

D2 — Adapter: OutboxAuditSink in craig-bootstrap, one-shot tx, inline

The single production impl holds the service’s PgPool and stages each envelope in a one-shot transaction (begin → stage_event → commit), every failure branch warn!-only — generalizing craig-rules' notify_fallback_admit pattern verbatim, including its accepted worst case (DB down ⇒ the miss path stalls up to the pool’s acquire_timeout before the warn; a degraded window on an already-failing 403 path). The call is awaited inline: each request pays for its own audit write, so there is no hidden queue, no shedding, and no unbounded task spawning. AuthzBootSpec gains an audit_sink field; all eight services pass Arc::new(OutboxAuditSink::new(pool)).

D3 — authz.cache_miss: source-attested absence only, attributed, unthrottled

Emitted at all three lookup miss branches (check_typed, auto_scope_list_typed, resolve_field_permission) — only AFTER lazy_load’s `? has propagated refresh faults, so PolicyLoad never emits. The payload gains actor attribution: sub (acting worker — the actor when an X-Craig-Actor assertion is present, per ADR-028), is_service (evaluation semantics: service caller with no actor), and service_id (outer caller for S2S). request_id stays null from engine-side emission — correlation rides the envelope’s trace_context. No producer-side throttle: volume is 1:1 with requests hitting the fail-closed 403 path, matching the rules.authz_fallback_admit precedent, and operators see true miss volume.

D4 — authz.cache_refreshed: every refresh path, trigger-attributed

refresh_entry gains a trigger parameter and emits on every actual state change: always for Updated (with old_version/new_version); for Removed (new_version: null, upstream deactivation) only when an entry was genuinely evicted (old_version present) — a no-op removal, i.e. a lazy-load miss on a never-cached name, stages nothing, because that absence is the cache_miss event’s job and emitting both would duplicate every miss with a phantom "refreshed to nothing" row. boot() emits one Boot-trigger event per loaded ruleset; bulk_refresh takes a trigger and the post-boot warmup passes Warmup. The trigger vocabulary grows from {rmq_event, ttl_refresh, boot} to add warmup and lazy_load, so the healthy-boot double-report (boot inline + warmup re-fetch) is honestly attributed to two distinct mechanisms rather than deduplicated away.

D5 — Consumption: existing pipeline, attribution probe extended

Events ride the ADR-022 outbox to craig.events; craig-security’s wildcard audit subscriber (ADR-003) lands them in audit_log. The catch-all parse_event_type arm already yields (cache_miss|cache_refreshed, authz) — no dedicated arm needed. The best-effort attribution probes gain sub, so both authz.cache_miss and the pre-existing rules.authz_fallback_admit rows attribute the acting worker instead of landing as system.

D6 — Overlap with rules.authz_fallback_admit is deliberate

During a craig-rules bootstrap window one request can emit BOTH events: the library observes policy absence (authz.cache_miss), the service records its admit decision (rules.authz_fallback_admit). Different layers, different event types, both wanted — the miss says "policy absent", the admit says "and here is what the bootstrap gate did about it".

Consequences

  • Operators finally get the ADR-023 "403 + audit event" behavior for missing policy, with fault/absence honesty and full refresh-lifecycle visibility.

  • Pre-1.0 breaking changes (CHANGELOG’d): ZenAuthzEngine::boot / boot_degraded / refresh_entry / bulk_refresh signatures, build_cache_miss_event takes a CacheMissInput struct, build_cache_refreshed_event’s `new_version becomes nullable, AuthzBootSpec gains a required field. All consumers updated in the same change.

  • ADR-024’s Consequences bullet ("audit events per audit_log table") stops being aspirational; amended to cite this ADR.

  • Steady-state volume is negligible (cache_refreshed ≈ rulesets/hour/replica on the default 3600s TTL); a sustained cache_miss flood is bounded by request rate on a path that is already returning 403s.

Open questions

  • request_id plumbing into engine-side envelopes (trait-signature change across 8 services vs a task-local) — deferred until trace-context correlation proves insufficient.

  • authz.access_denied (the third dormant builder) stays unwired: it fires on the hot deny path and deserves its own volume decision. Tracked separately (follow-up issue filed from #908 close-out). Resolved by the #1063 amendment below — emitted through the sink behind the full-dimension volume gate.

  • If a miss flood ever proves operationally noisy, a per-ruleset time-gate in OutboxAuditSink is the revisit point (the port makes it a sink-local change).

Alternatives considered

  • Bounded channel + host drain task — engine `try_send`s envelopes, a per-service task batches them into the outbox. Rejected: channel-resident events die with the process; a bounded channel silently sheds under exactly the flood operators most need to see (contradicts the no-throttle stance); adds a task + shutdown lifecycle per host with zero in-repo precedent.

  • Handler-side staging at the ApiError bridge — infeasible as specified: the bridge is a sync From impl inside the engine with no context, and the background refresh paths have no handler. Salvage means new middleware plus migrating ~140 handlers to the typed seam, plus a second mechanism anyway.

  • Direct Publisher::publish from the engine — at-most-once, bypasses the ADR-022 outbox durability the rest of the audit pipeline leans on.

  • Engine writes audit_log directly — rejected since Plan A §D14; couples every service to craig-security’s schema and breaks the single-consumer wildcard-subscriber pattern.

Amendments

  • #1063 (2026-08-22) — authz.access_denied emitted; the volume posture. The third builder is wired: check_typed’s `PolicyDenied arm stages authz.access_denied through the same injected sink (PolicyMissing stays the cache-miss event’s job; faults are never denies; the trait’s opaque default-impl wrappers and the field-permission/list surfaces are out of scope — an explicit row-check deny is the ADR-023 "403 + audit" promise this closes). The steer-ratified posture combines BOTH allowed options: the aggregation key is FULL-DIMENSION — (sub, resource_type, resource_id, action, ruleset_name, denied_reason), resource_id deliberately included so a scan across N resources emits N first-occurrence events (no audit dimension is ever collapsed; only EXACT repeats aggregate) — and suppression always reports:

    • First occurrence per key per 60 s window emits immediately (kind: "deny", full payload incl. ruleset_version and the acting worker under the sub key — #1063 also renamed the payload key from user_sub, which craig-security’s actor extraction never read).

    • Exact repeats inside the window suppress and count. The tally rides the next same-key emission (suppressed_prior
      suppressed_window_seconds), and every deny call opportunistically drains up to 4 expired sibling windows as standalone kind: "suppression-flush" payloads of the SAME event type — no background task (the engine has none; the #1535 class of phantom-sweeper docs is exactly what this avoids).

    • Bounded memory: 4096 keys per engine (<1 MiB). At capacity with every window live, a new key EMITS with aggregate_overflow: true — over capacity the gate degrades toward MORE emission, never dropped evidence.

    • Window semantics are PER-EMITTER (each of the 8 services aggregates independently; a fleet-wide dedupe would need cross-service state this design deliberately avoids). Reasons are ruleset-authored categorical strings, never user input. Process death loses at most one window of in-flight counts per key; the denies themselves were already emitted first-occurrence. Sink-failure posture unchanged: the port is infallible by contract and never alters the decision (§D2’s bounded inline-await trade applies — the path is already refusing). Implementation: crates/craig-authz/src/deny_gate.rs; paused-time unit pins + engine-level RecordingAuditSink pins + the authz_audit_emission.rs outbox-row boundary test.

  • #1140 (2026-08-09) — the fetch-reorder guard; Superseded stages nothing. refresh_entry fetches and compiles outside the cache’s write lock (#1126), so a slower refresh could commit AFTER a fresher concurrent one: a reordered None evicted a freshly seeded ruleset and recorded a maximally-fresh false absence (≤negative-TTL of PolicyMissing 403s with no further ruleset.changed event due), and the mirror-image reordered Some resurrected a deleted ruleset until the next TTL sweep. Every commit now carries its fetch-START instant; under the write guard (RulesetCache::supersedes) a commit is discarded when the cache already records a fact — live entry or attested absence — from a refresh whose fetch started later. The discarded commit returns the new RefreshOutcome::Superseded, which stages NO cache_refreshed event (the cache did not change; the winning commit staged the real transition — the D4 "actual state change" rule extended). Absences are now stamped with the observing fetch’s start instant rather than commit time, which honestly shrinks the negative window by the fetch→commit latency. bulk_refresh applies the same per-entry guard (a slow warmup cannot displace newer RMQ-driven facts) and its skipped entries likewise stage nothing.

Edit this page · latest