Decision-Cache Sweep (#1188, epic &74)

On this page

Status

Unit Description Status

U0

This plan page + the ADR-006 == Amendment — #1188 (identity-aware CAS protocol, honest bound, fail-closed eviction, recorded residuals) + nav

Done (2026-07-30) — MR !1112, merge ad34fa19

U1 (MR-A)

rule_sets.revision migration + BEFORE UPDATE trigger; RuleSet.revision (API-visible by decision; cargo xtask api-docs verified DRIFT-FREE — the generated pages tabulate request fields only, response schemas render as status lines, so the additive response field changes no page); the self-contained decision_refresh lib module (identity-aware CAS: apply_if_newer / remove_if_confirmed / stale_decision_names / reconcile + DecisionRefreshReport); engine delegation; subscriber reload_all → reconcile (probe_error → handler Err); the metadata-staleness fix in swap_compiled_decision_if_changed; comment sweep; the unit + CAS-race + method-level test matrix

Done (2026-07-30) — MR !1113, merge 84fed342. Both api pins observed RED against the pre-fix image (revision Null vs 1 on the wire; audit row stamped the superseded 1.0.0 after a version-only update), GREEN post-rebuild; 194/194 integration. J-review remediations recorded on the MR: the metadata-only cached-Arc reuse was DELETED (a concurrent content update could bind stale content to a fresh token, undetectable thereafter) — every update recompiles the RETURNING row’s content (apply_updated_decision); sweep-tense comments qualified "(MR-B)"; the CAS claim narrowed to INSTALLS (local delete/rename removals act on RETURNING knowledge — recorded residual class)

U2 (MR-B)

CRAIG_RULES__DECISION_REFRESH_SECONDS (default 300, 10..=86400, shared bounded-integer parser + proptest); spawn_decision_refresh_task (instance_id-keyed deterministic start offset, fixed cadence); compose + .env.example; the fault-category E2E with the observed-RED protocol; observability logs; the docs sweep

Done (2026-07-31) — MR !1114, merge ee158a0e. Fault e2e observed RED against the MR-A-only image (60.16s timeout still serving marker A), GREEN post-rebuild (converged 4.3s; boot log pins interval 15s/offset 3s). J-review: narrow J5 FAIL remediated (the healthy-DB qualifier added at the two doc sites missing it) + fault-test robustness P3s applied

Ship

Battery pushes, MRs merged, J-records, closing comment (3-AC walk), #1188 closed, epic &74 ticked, Plan Completion Audit + archive

Done (2026-07-31) — J-record notes on !1113/!1114; closing comment with the 3-AC walk + bare SHAs; #1188 auto-closed by !1114; epic &74 ticked (11/15); follow-ups #1216–#1219 stand

Issue: #1188 (epic &74, weight 3, Plan::DECISION-SWEEP)
ADR: ADR-006 == Amendment — #1188
Follow-ups filed at preflight: #1216 (atomic write+outbox — the ROOT cause; this sweep is the backstop), #1217 (cache-invalidation subscriber zero coverage), #1218 (vacuous oracle in evaluate_after_update_uses_new_content), #1219 (rules metrics plumbing + last-success gauge)
Review-round log (plan-lifecycle): round 1 internal (P2s applied) → round 2 EXTERNAL REJECTION of v1 — 7 blocking findings: the probe→snapshot→unconditional-apply reconciler was race-unsafe (stale overwrite / delete resurrection / wrong removal, reintroducing the class rules-cache-race-fix.adoc retired), updated_at unenforced as a token, fail-open compile failure, overclaimed bound, missing method/race tests, missing docs surface, sizing → full v2 redesign → round 3 (P1: bare-revision CAS broken by rename-freed-name recreates — the token became the (id, revision) pair; P2s: lib self-containment, api-docs blocking gate, absent-arm residual recording) → round 4 delta review: SHIP, all truth tables verified adversarially, the one condition (dual-residual recording) applied.

Context

Rule-set writes auto-commit; invalidation events are staged in separate warn!-only transactions (engine.rs:376-396). Scaled peers reload only on rules.cache_invalidated (main.rs:275-311 → reload_all()); evaluate() never consults the DB on a hit, and a miss is a hard RuleSetNotFound — a lost event leaves an updated ruleset stale until restart, and the engine.rs:365-371 "repopulate on next miss" claim is doubly false (AC-2 corrects it).

Adjacent defects to be fixed in-scope: the subscriber’s wholesale reload_all swap can overwrite a concurrent local API mutation (the same race class the completed rules-cache-race-fix.adoc (archive-indexed) fixed for the API handlers); and swap_compiled_decision_if_changed no-ops on metadata-only updates (api.rs:509-527), leaving the cached audit version stale locally (self-events are skipped).

Design (normative text: the ADR-006 #1188 amendment; this section carries the implementation detail)

D1 — the token

Migration rule_sets.revision BIGINT NOT NULL DEFAULT 1 + BEFORE UPDATE trigger NEW.revision = OLD.revision + 1. Bump-always, no WHEN guard (the store UPDATE always writes updated_at/updated_by; metadata-only bumps are REQUIRED by D3; the placement WHEN-guard precedent was lock-motivated, inapplicable). RuleSet gains the field by hand; it is API-visible by decision (pre-1.0 additive; lets the E2E read revision) → cargo xtask api-docs regen rides MR-A (blocking drift gate). CachedDecision gains revision: i64 (already carries id). The compare token is the PAIR — see the amendment for why revision alone is broken by rename-recreates.

D2 — the CAS module (craig_rules::decision_refresh, self-contained lib)

Owns CachedDecision (relocated; bin imports by crate path — the retention_archive wiring), FreshnessRow { name, id, revision }, its own queries (freshness probe; per-name re-probe returning (id, revision, active) REGARDLESS of active — no existing store fn distinguishes inactive from absent; per-name active = true-filtered full fetch), and:

  • apply_if_newer(cache, row, decision) — install if absent, or id differs, or same-id-higher-revision.

  • remove_if_confirmed(cache, name, reprobe) — absent → remove; inactive (any id) → remove; ACTIVE → refresh candidate, never removed.

  • stale_decision_names(cached: name→(id, rev), probed) — pure diff: refresh = id differs OR revision differs OR probe-only; removal candidates = cache-only.

  • reconcile(pool, cache, compile_fn) → DecisionRefreshReport — probe → diff → confirmed removals FIRST (cheap) → per-name fetch+compile+apply; per-name failures recorded, never starve a pass; compile_fn errors are lib-owned String`s (the engine’s closure maps `EngineError in). Fail-closed compile eviction. DecisionRefreshReport { refreshed/removed/evicted: BTreeSet<String>, failed: BTreeMap<String, String>, probe_error: Option<String> } — sorted, the test oracle.

Recorded residuals + the benign probe-snapshot-lag wasted-fetch: the amendment.

D3 — call-site unification

insert_decision delegates to apply_if_newer (all four API mutation paths inherit CAS). Subscriber: reload_all() → reconcile(), probe_error → handler Err (preserves MQ retry). reload_all becomes boot-only. The swap_compiled_decision_if_changed no-op arm is changed to re-apply the CACHED decision Arc with the fresh row (metadata refresh). Comment sweep: engine.rs:365-371, engine.rs:350, main.rs:280 corrected; engine.rs:445-448 scoped to its craig-authz downstream consumers, not rewritten.

D4 — the worker + knob

CRAIG_RULESDECISION_REFRESH_SECONDS default 300, validated 10..=86400; the bounded-integer parse is FACTORED SHARED with EVAL_TIMEOUT_MS in config.rs (clone-twin would regress B8; the existing pinned error-prose tests stay green untouched) + proptest. spawn_decision_refresh_task: deterministic hash start offset keyed by instance_id (scaled replicas must not sweep in lockstep — retention tolerates it only via its advisory lease), then fixed-cadence biased select!. Honest bound: ≤ ~2×interval + pass work, healthy-DB conditional (≈10min default; ≈30s devstack). Observability: startup info! (effective interval), per-pass summary info! when non-empty / debug! when empty, warn! + report on probe failure; the gauge is #1219. Compose: CRAIG_RULESDECISION_REFRESH_SECONDS: ${CRAIG_RULES__DECISION_REFRESH_SECONDS:-15}; .env.example row.

Tests

MR-A (lib seam drives PRODUCTION code; shared devstack craig_rules DB, unique names + cleanup; #[ignore = "requires devstack"] + connect-else-return; every test @axis:-tagged):

  1. Unit (pure, identity-aware): stale_decision_names exhaustive incl. changed-id-same-revision (the rename-recreate pin); apply_if_newer truth table incl. different-id-ANY-revision → install; remove_if_confirmed decision table.

  2. CAS order-independence: apply(v1);apply(v2) ≡ apply(v2);apply(v1) → v2; rename-recreate both orders → the new row wins; reconcile-vs-concurrent-insert converges; create-during-reconcile never removed.

  3. Method-level vs devstack DB: first pass refreshes; second pass idempotent FOR THE TEST’S ROW — every assertion is scoped to the test’s own rows, because the shared DB has concurrent writers by construction (nextest runs one process per test, so no in-process gate can serialize the group, and the rules api suite mutates rule_sets in parallel during the battery; a whole-report is_empty() claim flaked exactly this way and was scoped); direct-SQL update BY ID (RETURNING revision, exactly-one-row) → refresh; deactivate → confirmed removal; rename+recreate → fresh row installs; invalid-JDM → eviction (fail-closed pin); probe_error via closed pool.

  4. Worker tick (paused-time, #784 pattern): generic-refresher loop, start-offset
    N intervals → N calls; prompt cancellation.

MR-B — tests/fault/decision_cache_staleness.rs (new tests/fault.rs crate root, standard allow block; @axis: fault): baseline marker-A via API; content B INDEPENDENTLY validated (temp-name create → marker B → delete); direct-SQL content swap BY ID (RETURNING revision); whole poll in tokio::time::timeout(60s) (4× devstack interval; the compose-knob coupling named in the test doc), 1s cadence (~60 audit-row side effects, unique-name-keyed, panic-safe cleanup before asserts); no mid-poll stale assert (flaky — a healthy sweep can fire immediately). Observed-RED protocol: pre-fix image verified by container creation time → focused run times out still serving A → record; cargo xtask dev start → GREEN.

Verification

Per MR, gates run separately (check-docs exit 3 = the 4 standing overrides): scoped clippy -D warnings; --bins (engine units) + --lib (decision_refresh pure units) + --lib --run-ignored=all -E 'test(decision_refresh)' (method-level); integration binary(api) or binary(fault); cargo xtask api-docs (MR-A); scoped cargo llvm-cov nextest -p craig-rules; quality-budgets fail-on-regression; axis-coverage; plan-lint; local Antora render. Full battery via the pre-push hook per push. Gate pre-empts: fns ≤40 lines (reconcile decomposed); B8 via the shared parser; B4 no new allows; F-054 clear (13 counted methods today); no rand dep.

Delivery

  1. Preflight — DONE 2026-07-30: assigned, weight 2→3 + note, Plan::DECISION-SWEEP; follow-ups #1216/#1217/#1218/#1219 filed + related.

  2. U0: this page + the amendment + nav → plan-lint → battery push → MR → MERGE before code.

  3. U1 (MR-A): migration → lib module → delegation/subscriber/metadata → tests → api-docs regen → gates → battery → Draft MR at first push → merge per the standing green-local-pre-push directive. The 500-LOC increment guideline is knowingly exceeded by the test matrix — rationale in the MR description.

  4. U2 (MR-B): knob/parser → worker → compose/.env.example → fault E2E (RED protocol) → docs sweep → battery → MR (Closes #1188) → merge.

  5. Close-out: J-records; closing comment (3-AC walk + honest bound + follow-ups, bare SHAs); close; epic tick; Plan Completion Audit + archive; cleanup.

Edit this page · latest