ADR-006: zen-engine Rules Engine

Safety assessments (structured decision-making tools)

Intake screening decisions

Title IV-E eligibility determination

Placement matching criteria

Timeliness compliance (CFSR/NCANDS deadlines)

Rust-native: zen-engine is a Rust library, avoiding FFI overhead or a separate rules server process.

JDM format: JSON-based decision models can be authored in GoRules' visual editor or any JSON tool. No proprietary DSL — just JSON decision tables and expression nodes.

Versioned and auditable: Rule sets are stored in the database with version tracking. Imports auto-increment versions. Every completed evaluation is recorded for audit and compliance (errored/timed-out dispatches are not — see the Architecture bullet above).

Cross-service evaluation: Other services call POST /v1/rules/evaluate with context — the rules engine evaluates and returns results. Event-driven evaluation is also supported via RabbitMQ.

Jurisdiction configurability: Different states can deploy different rule sets without code changes. The jurisdiction prefix ensures rule sets don’t collide.

Single-threaded evaluation: The !Send constraint of zen-engine means evaluations are serialized through one thread. This is acceptable for child welfare workloads (low evaluation volume) but would need architectural changes for high-throughput scenarios.

Cache invalidation overhead: Every rule set mutation publishes a rules.cache_invalidated event, causing all instances to reload all rule sets from the database. This is a simple but heavy-handed approach. (To be superseded in part by the #1188 amendment below — design accepted, implementation tracked in that plan: peers are to reconcile incrementally instead of wholesale-reloading, with a periodic sweep bounding staleness when the event is lost.)

GoRules dependency: While JDM is JSON-based, the visual editor for authoring complex rule sets is a commercial product (GoRules). Simple rule sets can be authored in any JSON editor.

The enforced change token is the pair (id, revision). rule_sets.revision (BIGINT, DEFAULT 1) is bumped by a BEFORE UPDATE trigger — monotonic per row for every writer including direct SQL. Revision ALONE is insufficient: the cache is name-keyed, name is UNIQUE-total and rename frees it, and a recreate under a freed name starts at revision 1 — only the id half detects rebinding.

Every cache mutation is a conditional apply (craig_rules::decision_refresh, a self-contained lib module the bin and the integration tests both drive): install if absent, or id differs (name rebound), or same-id-higher-revision. Removal only after a fresh per-name re-probe confirms absent/inactive. The API insert path, the cross-instance subscriber (becomes reconcile, replacing reload_all — the wholesale swap could overwrite a concurrent local mutation), and the periodic sweep all speak this protocol; reload_all becomes boot-only.

The periodic reconciler bounds lost-event staleness: a probe of (name, id, revision) for active rows, diffed against the cache; refreshes and confirmed removals applied per name; fail-fast decomposed report. Interval knob CRAIG_RULES__DECISION_REFRESH_SECONDS (default 300, devstack 15). The honest bound: under a healthy DB with compilable rows, staleness after a lost event is ≤ ~2×interval + pass duration (≈10min at the default; ≈30s in devstack) — never claimed as "interval seconds".

Fail-closed compile policy: a changed row whose stored content no longer compiles EVICTS the cached entry (error! + report) — a 404 beats serving superseded policy. Only direct-SQL writes can produce this; the API pre-validates.

Recorded residuals (bounded ≤1 pass, self-correcting, both from the same re-probe/apply window; the generation-guard/tombstone that would close them fully is declined at this severity): a wrong-removal window (fail-closed interim) and its mirror, a stale re-install after a concurrent delete (fail-open interim, narrowed to milliseconds by the active-filtered fetch). The ROOT residual — non-transactional event staging — was CLOSED by #1216 (2026-08-03): all four rule-set mutation paths (create / update / delete / import) stage rules.cache_invalidated + ruleset.changed.<name> in the SAME transaction as the row write (RulesEngine::stage_cache_invalidated / stage_ruleset_changed join the caller’s tx; a staging failure rolls the mutation back — the one-shot warn!-only notify helpers are retired). The sweep remains as the backstop for DELIVERY-side loss (outbox → broker → consumer) and for the originator’s own narrow post-commit window (the local cache apply runs after the tx; a failure there 500s with row + events already committed, and the originator skips its own event) — no longer for staging loss, which is unrepresentable.

Failure rows ride their OWN short transaction — never the caller’s. The inbox path (evaluate_in_tx, ADR-062 A5) rolls its attempt tx back on this very error; the forensic row must survive, and each retried attempt leaves one row by design. The success path is unchanged: #1130’s fail-closed posture (no response until the row commits) still holds for decisions.

Failure audit is best-effort, the deliberate inverse of #1130. A failure response must carry the ORIGINAL error; an audit-write failure is WARN’d, never allowed to mask it. (A success is not served without its row; a failure is never hidden behind its row’s failure.)

Late completion converges, it never inserts. The caller mints ONE evaluation id per attempt and writes the timeout row on the error path; the zen loop routes a dead-oneshot SUCCESS to a recorder task that flips that row to late_completed (UPDATE-only, bounded retry — the trade of a narrow no-row WARN window against cloning every request’s input through the eval channel for a rare event). A late FAILURE adds nothing over the timeout row and is dropped. The recorder is supervised Observed (eval-late-recorder); its loss degrades convergence, corrupts nothing.

error_detail is PII-safe by construction: engine error text only — ruleset-authored content and engine internals; caller input is never interpolated into EngineError messages (the input itself lands in the row’s input column exactly as completed rows do, under the same access control).

The MQ half is explicitly scoped OUT. rules.evaluated stays success-only: failure rows are queryable at GET /v1/rules/evaluations?disposition=… and ride the #1129 archive tier, and a new rules.evaluation_failed event would create consumer obligations (security audit_log copies, retention, replay) with no present consumer need. Revisit only with a concrete consumer.

Wire surface: RuleEvaluation.output became nullable and the rows carry disposition / error_detail / budget_ms / dispatched (pre-1.0 breaking, CHANGELOG’d); the list endpoint filters by disposition with an allow-list 400 on unknown values.

Own the bound instead of inheriting it. The 5s upstream function_timeout_millis default was a load-bearing safety property nobody set. Boot now publishes CRAIG_RULES__FUNCTION_TIMEOUT_MS into zen_engine::ZEN_CONFIG (default = min(5000, eval budget); an explicit value must sit in 1..=eval budget — a JS node outliving the dispatch budget would turn a clean typed runtime_error row into a caller timeout). The bound is per-NODE: a multi-JS-node graph can legitimately sum past it, and the #784 budget stays the per-dispatch caller ceiling. The setting is process-global, so craig-rules' own embedded authz engine is covered; every OTHER service’s embedded authz eval thread still inherits the default silently — the fleet-wide mirror is #1328.

Detect permanence with a precise signal, not a proxy. The eval loop stamps its single in-flight evaluation (id + start instant); a watchdog samples it and fires when ONE evaluation has run past max(10 × eval budget, 60s). Queue wait never stamps the slot, so backlog cannot trip it — unlike the rejected consecutive-EvalTimeout counter, which false-positives on load bursts. The threshold is deliberately generous: it detects a bound-escaping BUG (every designed bound sits far below it), not slowness — latency remains the #784 budget’s job.

Recovery is platform restart through the existing health surface. The deadman widens the SAME zen-eval Critical liveness signal #1228 registered (thread death OR proven wedge), riding ADR-061 fail-fast: supervisor shutdown, nonzero exit, orchestrator restart. The eval thread is stateless (compiled decisions live engine-side), so a restart loses nothing but an unconsumable late flip — already a WARN’d #1048 loss window.

Rejected: in-process respawn. A wedged std thread cannot be cancelled; respawning would abandon it alive — a leaked core, a leaked runtime, unreclaimable C-level rquickjs state, unbounded across repeated wedges — while the process claims health. That is running on a corrupted runtime and calling it self-healing. Also rejected: an N-thread eval pool (decorrelates wedges without fixing one; no observed throughput need) and the consecutive-timeout detector (above).