ADR-060: Locked-Row Staleness Recheck for Assignment-Gated Mutations

On this page

Status

Accepted (2026-07-30). Decision record for #1187 (epic &74). The Proposed→Accepted discussion ran as two independent contextless plan-review rounds (the first rejected the v1 engine-under-lock design — see D2 — and reshaped this decision; the second found no blocking findings) plus the user’s plan approval; the implementation plan is Authz Staleness Recheck (archived — all three units shipped 2026-07-30).

Cites (not amends): ADR-050 (the authorize-before-tx premise and the audit sink’s second same-pool transaction — both load-bearing rejections below), ADR-054 (craig-financial’s in-tx assignment re-check is the precedent this decision generalizes from; its acting-worker identity-compare shape is unchanged and remains correct for financial’s model), ADR-023 / ADR-024 (the multi-jurisdiction authz architecture and the zen-engine + RMQ cache-invalidation design: an authorization verdict is always one cached ruleset version at evaluation time — this ADR does not change that, it serializes the resource inputs the verdict consumed).

Context

update_placement authorized on assigned_worker_sub read from an unlocked pre-read, then took SELECT … FOR UPDATE and re-validated only the state transition — never the authorization-relevant field. The concurrent writer is real and case-scoped: the case.assignment_changed inbox handler rewrites assigned_worker_sub by case_id — its UPDATE takes row locks like any other, but nothing serializes it against the handler’s unlocked pre-read, which is where the authorization decision was made. Georgia’s caseworker-update rule keys on exactly that field (claims.sub == resource.assigned_worker_sub), so a reassignment committing between pre-read and lock let the formerly-assigned caseworker’s mutation proceed (#1187, external review W4a, CONFIRMED). update_foster_home had no transaction at all; kinship update/delete shared placement’s hole with no lock either (the #1187 audit).

What policy actually consumes (ruleset-verified, both shipped bundles): for the placement service’s resource types, no Georgia ruleset consumes resource.attrs; the only resource-field condition is claims.sub == resource.assigned_worker_sub, and only the placement + kinship_option builders pass a live value (foster_home / education / health / home_document builders pass None — role-only decisions). Texas rulesets additionally test resource.supervisor_sub — inert today only because every placement builder passes supervisor_sub: None, a constant that cannot go stale.

Decision

D1. The conjunctive two-snapshot policy

For mutations whose authorization consumed a concurrently-mutable resource field:

allow = allow(pre-read snapshot) AND unchanged(policy-consumed input, under the mutation row lock)

  1. The pre-read + decisive authz.check keeps its position before the transaction (ADR-050’s premise): unauthorized callers are refused with 403 without ever taking a row lock or opening a transaction.

  2. Inside the mutation transaction, the handler re-reads the row under SELECT … FOR NO KEY UPDATE and performs a pure in-memory compare of the policy-consumed mutable input — for placement and kinship_option, assigned_worker_sub. A difference refuses with 409 CONCURRENT_MODIFICATION ("assignment changed since authorization — re-read and retry"), the same problem-type vocabulary ADR-054’s financial recheck uses.

The declared consequences are conservative for ALL principals, and deliberately so:

  • a supervisor racing a reassignment receives a retriable 409 even though a fresh evaluation would allow them — the retry succeeds;

  • a worker whose assignment arrives mid-flight is denied at the pre-read gate and succeeds on retry;

  • denials are never less strict than the locked state; spurious denials are transient and retriable. No new status codes enter the API vocabulary.

D2. Why not re-run the engine under the lock (rejected)

Re-evaluating authz.check against the locked row is the conceptually direct form, and was the v1 design. It is rejected on measured mechanism, not taste — a held row lock would wait on:

  1. the strictly serial evaluator thread (every check in the process queues through it — craig-authz/src/eval_thread.rs) under a 5s default eval budget (engine.rs, DEFAULT_EVAL_BUDGET);

  2. on a cache miss, untimed service-token/OIDC paths (craig-auth/src/service_token.rs) and ruleset HTTP fetches on the shared 30s client timeout (craig-common/src/http.rs), plus JDM compilation — with no outer lock-held deadline;

  3. worst, ADR-050’s audit sink, which stages cache-eviction audit rows in a second transaction on the same DB pool (craig-bootstrap/src/audit_sink.rs) — holding a pool connection + row lock while requesting another pool connection is a pool-starvation amplifier under concurrency.

It also contradicts ADR-050’s authorize-before-tx premise. The staleness compare is µs-bounded by construction: one indexed SELECT the transaction needs anyway, one in-memory field compare. (No latency numbers are claimed here; the bound is structural.)

D3. Why not the identity-compare or a role-skip (rejected for placement)

ADR-054’s financial recheck compares the locked assignment against the acting worker and skips realm-scoped principals — correct there because financial has an ADR-028 acting-worker model that defines "realm-scoped" precisely. Placement has no acting-worker concept: a placement-side role-skip list would hand-roll ruleset knowledge in Rust and silently drift against replaceable rulesets. The input-staleness compare needs no role knowledge at all.

D4. The honest guarantee: serialized against the locked local projection

assigned_worker_sub on placement rows is an asynchronous local projection of case assignment (store/models.rs). At this ADR’s landing the inbox writer applied events with no causal/revision guard — out-of-order case.assignment_changed deliveries could rewind the projection. Strengthened by #1214 (2026-08-03): the producer bumps a per-case assignment_revision in the same transaction as the assignment write (so revision order == commit order) and the event carries it; the placement inbox applies an event iff its revision is strictly greater than the stored one — out-of-order deliveries match zero rows, and the strict guard subsumes the former IS DISTINCT FROM idempotency; the backfill xtask writes the case’s (worker, revision) pair with the same never-rewind guard (value-converging only at equal revision). The projection is causally monotonic FOR EVENT AND BACKFILL WRITES (still asynchronous — this decision’s serialization against the locked local projection is unchanged and remains the mutation-side guarantee). The formerly-recorded birth-window residual (rows born at revision 0 with a create-time-minted worker) is CLOSED by #1213: placement creates now hydrate the (worker, revision) pair from the case_assignments projection at insert (see the amendment below). #1312 ported the strict guard to the financial and exchange copies of the handler (payments / payment_adjustments / subsidy_agreements; icpc_requests / icpc_home_studies / icpc_attachments — each with the revision column, the hard-required payload key, and the backfill pair-write), so every case.assignment_changed projection fleet-wide is causally monotonic. Their rows still BORN at revision 0 carry the birth-window note: financial/exchange rows are created by their own domain flows (not caller-minted assignment), and the next assignment event at revision ≥ 1 converges them.

D5. Evolution boundary (bidirectional)

The compare covers the policy-consumed mutable input as of this decision. Two triggers require extending it, and both must be checked when either side changes:

  • Builder side: extending a *_resource_ref builder to pass a live row-derived value for any policy-testable field (supervisor_sub, any attrs entry) — texas’s standing supervisor_sub conditions are the concrete example of a ruleset condition lying in wait for exactly this;

  • Ruleset side: a ruleset gaining conditions on a field a builder already passes live.

Post-state/destination authorization (a policy scoping on a field the mutation itself changes, e.g. admin_unit) is a distinct semantics question — resolved by #1212 (see the amendment below): the update handlers now evaluate the projected candidate ref too, deny unless BOTH allow. This ADR’s staleness recheck still compares before-images only.

D6. Kinship delete rides Action::Update

The kinship_option rulesets define no delete action; delete_kinship_option authorizes (and staleness-rechecks) under Action::Update — edit permission intentionally implies deletion for this resource. Introducing a distinct delete action is a ruleset-vocabulary change, out of scope here.

D7. Lock strength: FOR NO KEY UPDATE

The staleness/lock re-reads use FOR NO KEY UPDATE: the placements and kinship_options UPDATE SET lists touch no unique/key columns, so the weaker (still self-conflicting) tuple lock preserves all handler-vs-handler and handler-vs-inbox serialization while not blocking FK KEY SHARE acquisition from child-row inserts. One narrow exception is recorded: foster_homes.license_number is UNIQUE and updatable — a request changing it auto-upgrades that UPDATE’s tuple lock to key strength, which is legal and acceptable. The occupancy trigger’s own FOR UPDATE (#1117) is a separate statement and unaffected.

Consequences

  • update_placement, update_kinship_option, delete_kinship_option gain the staleness gate; update_foster_home gains the transaction + locked-row transition validation it never had (no staleness gate — no policy-consumed mutable input; role-only rules).

  • Kinship mutations acquire live-row semantics: the locked read filters active = true, so a concurrently soft-deleted option reads as gone (404) and can never be resurrected by a racing update.

  • Former 200/204 outcomes on raced mutations become 409/404/400 — an intentional, observable security behavior change (pre-1.0; no new response shapes).

  • Implementation plan: Authz Staleness Recheck (the 18-handler audit table lives there). Follow-ups routed: #1212 (post-state semantics), #1213 (create handlers mint assignment authority — closed; see the amendment below), #1214 (projection causal ordering), #1215 (kinship GET returns soft-deleted rows).

Amendment — #1213: creates derive assignment from the case_assignments projection (2026-08-03)

The two case-scoped creates (create_placement, create_kinship_option) previously authorized Action::Create with a nil-id ResourceRef carrying the CALLER’s own claims.sub as assigned_worker_sub and then persisted that value — i_assigned was trivially true at create, so any caseworker could self-seed record-level authority on an arbitrary case_id (the #1187 external review’s blocking finding 4). As-built fix:

  • case_assignments — a placement-local projection table (case_id → (assigned_worker_sub, assignment_revision)), fed by the inbox: case.created seeds revision 0 (ON CONFLICT DO NOTHING — a replay can never displace a newer projection) and heals rows born while the case was unknown from the projection pair; case.assignment_changed insert-or-updates under the #1214 strictly-greater guard. The seeder emits the projection rows directly (direct-SQL seed publishes no events); cargo xtask backfill-cross-service-assignment gained the projection leg for pre-projection cohorts.

  • Derive, never trust: the create handlers read the projection (pool, pre-transaction) to build the ResourceRef — a case unknown to the projection yields None, fail-closed for assignment-gated rules — then re-read it FOR SHARE inside the claim transaction as the FIRST domain statement; the PERSISTED pair is always the in-transaction read, and worker drift from the authorization-time value refuses 409 (this ADR’s D1 posture applied to creates). The rows stamp the projection’s assignment_revision, so a stale late event cannot rewind a fresh row (#1214’s guard now protects creates too).

  • Lock-order contract: the inbox handlers write case_assignments FIRST — a create transaction holding FOR SHARE blocks the inbox’s upsert until commit, after which the inbox’s row UPDATEs see (and correct) the committed created rows. Writing the entity tables first would let a raced create commit a stale pair no delivery ever re-scans.

  • Ruleset tightening (georgia + texas, placement + kinship_option, versions → 1.1.0): the "Caseworker create" row now requires i_assigned == true. Admin/supervisor/service rows stay role-flat — their create on an unknown case is born UNASSIGNED (NULL, revision 0) rather than caller-owned, and heals when case.created arrives.

  • Accepted residual: a role-flat create racing the very first case.created delivery for its case can commit after the heal scanned (nothing to FOR SHARE — no projection row yet); the row stays unassigned until the next reassignment event. Fail-closed (no caseworker gains access), and strictly better than the pre-#1213 shape where EVERY create minted the caller.

Amendment — #1212: source-vs-candidate-state authorization (2026-08-09)

The D5 boundary named the open question; this amendment closes it. The three placement mutations that change fields their own policy attrs carry (update_foster_home → admin_unit/license_status, update_placement → status, update_kinship_option → approved) previously authorized against the before-image ONLY — a policy scoping on such a field (e.g. a county-director rule keyed on resource.attrs.admin_unit) had no destination enforcement point: authority over the source scope sufficed to move the resource into ANY scope. Not exploitable at decision time (no shipped ruleset consumes resource.attrs for these types — re-verified 2026-08-09), but armed by either D5 trigger.

Decided semantics — evaluate both, deny unless both allow:

  • craig_authz::check_update_dual_state(engine, claims, before, after, action) — the shared enforcement point. before is today’s ref; after is the SAME builder’s ref with the request-supplied policy-visible values projected in (exactly what the policy would see post-write). The before-eval short-circuits (a caller without authority over the row they hold gets today’s deny; the after-eval would only double-stage cache-miss audits). The after-eval is SKIPPED when the projection changes no policy-visible input (assigned_worker_sub, supervisor_sub, attrs) — the no-op-projection path stays at one evaluation.

  • Both denials surface as the SAME payload-free 403 (Insufficient permissions) — which snapshot denied is a server-side log distinction only, preserving the no-policy-probing stance.

  • Applied to the three named handlers. delete_kinship_option flips only active/deleted_at (no policy-scopeable change — single eval unchanged); create_foster_home already evaluated the REQUESTED admin_unit in its nil-id create ref (the pre-existing destination-state precedent this amendment generalizes); the #1213 creates derive assignment from the projection and stay as-built.

  • Evaluation cost: both evals run pre-transaction against the same cached compiled decision under the #1126 budget — no engine call ever runs under the row lock (D2’s premise intact).

  • Staleness interplay: D1’s locked-row compare still pins before-images only. The moment a ruleset starts consuming an attr a mutation can change, the destination eval (this amendment) authorizes the mutation’s OWN write, and concurrent writers of that field remain serialized by the existing row locks — a new staleness input arises only if a NON-request writer mutates a policy-consumed attr, which is the standing D5 trigger.

Pinned by crates/craig-authz/tests/dual_state_check.rs: the four contract legs (two-eval, destination-deny, source-short-circuit, identical-projection single eval) against a counting double, plus two REAL-zen legs booting an admin_unit-scoped JDM proving the projected attrs reach the policy and the move-into-unauthorized-unit denies end-to-end.

Amendment — #1367: the supervisor half goes live (2026-08-09)

The Context’s "inert today only because every placement builder passes supervisor_sub: None`" premise is retired — the D5 builder-side trigger fired. #1367 (the #1084 recipe applied to the remaining row surfaces) gave `placements / kinship_options (and exchange’s three ICPC tables) the supervisor_sub denorm, and the placement + kinship + icpc-request builders now pass the live row value. Per D5, the compares extended in the same change:

  • Staleness gates (validate_locked_placement, refuse_stale_assignment): the locked-row compare now covers the assignment PAIR — drift on either half between pre-read and row lock refuses with the same 409.

  • #1213 create drift: claim_create_placement / claim_create_kinship_option compare the FOR SHARE projection re-read against the pre-transaction (worker, supervisor) pair the authz decision consumed, and persist the in-transaction triple.

  • #1212 interplay: check_update_dual_state’s changed-projection set already listed `supervisor_sub; the projected refs inherit the live value through the shared builders with no further change.

Exchange note: the icpc builder previously passed None for BOTH pair halves; #1367 wired both, making texas’s standing i_assigned PTO-read row real alongside i_supervises. Exchange mutations keep their ADR-062 §G2 expected_status pins; an exchange assignment-staleness gate on the now-live pair remains bounded by the same D1 analysis (the assignment event writer serializes behind the row locks the mutation transactions already hold).

Edit this page · latest