Per-Diem Payment Idempotency (placement.activated double-pay)

On this page

Status

Unit Description Status

U0

Choreography: issue amendment, follow-ups #1192/#1193, this plan

Done (2026-07-28) — commit 9738efec

U1

Fail-closed dedupe migration + migration test + fixture reworks + constraint tests

Done (2026-07-28) — commit df55af0f; J-review PASS (gate sound, no money-corruption holes); 3 reviewer notes adopted (created_by in the identity tuple; stamp-only/adjusted/divergent-identity refusal pins; staged-set discipline); 9/9 migration tests green on clones; live probes deterministically red pre-migration

U2

Arbiter-first handler (early probe, derivation compare) + replay/conc tests

Done (2026-07-28) — crash-window replay test verified deterministically RED against the pre-fix binary (second payment found); J-reviewed

U3

Blocking invariant + classification test + docs (data-model, ADR-053, ADR-022, testing-reference, CHANGELOG)

Done (2026-07-28) — invariant red/green harness on throwaway DB; blocking-classification xtask test; stale data-model index prose fixed

Ship

Battery (post devstack wipe+reseed), MR, merge, close-out

Done (2026-07-28) — MR !1095 merged (fb339152); full battery green on attempt 2 (attempt 1 fast-failed at the axis cheap gate — 3 tags relocated beside the test attribute, 1794f7f8); #1179 closed with AC walk; epic &74 ticked

Epic: &74
Issue: #1179 (Plan::PERDIEM-IDEM)
Follow-ups filed: #1192 (fixture residue cleanup), #1193 (ended-before-activated ordering hole — NOT solved here)
Review state: v1/v2 internal (scout workflow wf_8c5bca48-944); v3 folded the 2026-07-28 external review (~25 findings: lifetime key, fail-closed migration, arbiter-first effects, early probe, derivation compare, migration test, selector fixes). Approved 2026-07-28.

Context

  • Path: handle_placement_activated (services/craig-financial/src/main.rs:595-672) → persist_placement_payment (:705-751) → create_payment (store/payments.rs:54-85): plain INSERT — no ON CONFLICT, no existence check; record_subsidy_overlap and the payment_created outbox stage surround it in one tx. A re-execution mints a second live payment AND stages a second event, silently.

  • Inbox dedup is best-effort at-least-once (documented by #1180; redesign is #1178): crash-before-stamp re-runs a succeeded handler; concurrent Path A/C; >30-day retention bypass; admin replay.

  • Zero DB uniqueness and zero invariant coverage on agreement_id IS NULL rows (both subsidy partial uniques and both duplicate-shaped invariants filter them out — verified exhaustively).

  • Lifecycle trap: placement.ended voids pending/approved (store/payments.rs:197 area) and an ended placement can never reactivate (services/craig-placement/src/transitions.rs:18-26) — a voided-exclusive key would let a stale replay create a fresh payment AFTER the ending that nothing ever voids again. The key must outlive row status.

Design

# Decision Substance

D1

Lifetime key: CREATE UNIQUE INDEX payments_one_perdiem_per_placement_period ON payments (placement_id, period_start) WHERE agreement_id IS NULL — NO status filter

One per-diem row per (placement, period_start) EVER. The producer emits placement.activated at-most-once per placement (no re-activation path) ⇒ a collision is always a replay artifact, never legitimate money. Bare (placement_id) fails the seeder (3 rows/placement); (placement_id, payment_month) fails the leg-3 drift shape date-dependently (two live rows 14 days apart on one placement — tests/store/review_sweep.rs:1712-1722 — a load-bearing domain shape that stays representable under period_start granularity). period_start = payload.started_at.date_naive() is identical across every replay class. Voided does NOT free the key.

D2

Fail-closed dedupe migration, one tx: preflight → narrow auto-repair → ledger → outbox cleanup → zero-collision assert → index

SET LOCAL statement_timeout = 0; SET LOCAL lock_timeout = '10s'; LOCK TABLE payments IN SHARE MODE;. Duplicate groups per key among agreement_id IS NULL rows: auto-repair ONLY groups entirely identical, untouched, pending (same money fields; no lifecycle stamps; no payment_adjustments rows; no subsidy_perdiem_handoffs refs) — keep earliest by (created_at, id), DELETE the rest, record each into the new payments_dedupe_repairs ledger (canonical_id, duplicate_id, prior_status, row snapshot JSONB, reason, created_at); delete UNPUBLISHED payment_created outbox rows whose payload payment_id is a deleted dupe. Any other group shape → RAISE EXCEPTION naming manual reconciliation (dev remedy: wipe+reseed). payments has NO voided_by/voided_at and none is added — the ledger IS the attribution.

D3

Arbiter-first handler with early probe + derivation compare

(a) Early probe before the DOB/rate lookups using immutable payload data only: existing row for (placement_id, period_start, agreement_id IS NULL) → compare + absorbed Ok (an outage replay must not DLQ). (b) create_payment gains ON CONFLICT (placement_id, period_start) WHERE agreement_id IS NULL DO NOTHING + fetch_optional → Option<Payment>, and runs FIRST in the tx; record_subsidy_overlap AND publish_payment_created run ONLY on Some (the overlap enqueue is re-enqueueable once processed — subsidy/store.rs:3115 — so it is arbitrated too). (c) On absorb: reload incumbent and compare — identity (child/case/home/type) always; money (period_end/rate/day_count/gross) only on the arbiter race path, where it is derivable (the early probe deliberately runs BEFORE the lookups, so it compares identity alone) — the persist_expected_row precedent (subsidy/generator.rs:1088-1112): match → structured info; mismatch → structured tracing::error, no new row, Ok.

D4

Blocking invariant over ALL per-diem rows

crates/craig-test-lib/sql/invariants/craig-financial/perdiem_placement_duplicate.sql: GROUP BY (placement_id, period_start) HAVING count>1 over agreement_id IS NULL rows, no status filter, LIMIT 20; grace 0s, severity high, remediation cites #1179. Added to BLOCKING_INVARIANTS (holds by construction via D1, same MR). Deliberately NO broader child/month invariant — mid-month placement moves are legal two-row shapes.

D5

Fixture reworks (collide under the lifetime key)

placement_ended_void.rs::seed_payments is called per STATUS BATCH with generate_series restarting at 1 — add a base_offset param (call sites pass 0/120/150/153); period_start = DATE '2026-01-01' + (base_offset + gs), period_end = period_start + 30; assert the total seeded row count. eligibility_bulk_update.rs::seed_case_payments: placement_id = gen_random_uuid() per row. Other fixtures/seeder already key-distinct.

Units

U0 — issue amendment + label; follow-ups #1192/#1193 filed + related; this plan + nav entry; cargo xtask plan-lint.

U1 — migration per D2 (header carries the deploy runbook); scratch-DB migration test (the standing_alert_dedupe_backfill.rs pattern) asserting: survivor determinism incl. (created_at, id) ties, ledger rows exact, unpublished outbox rows removed, and ERROR raised for mixed-status / divergent-money / adjustment-dependent / handoff-referenced / voided-member groups; fixture reworks per D5 (same commit); constraint tests tests/constraints/perdiem_placement_unique.rs (// @axis: evil): dup live → 23505 naming the index; voided does NOT free the key; distinct period_start coexists; subsidy rows unaffected.

U2 — D3 code; // @axis: replay scenarios in placement_activated_billing.rs: (1) crash-window (clear processed_at, assert rows_affected == 1, republish same envelope → 1 payment, error_count = 0, exactly one payment_created whose payment_id = the sole row — scoped by run-unique child, the payload has no placement_id); (2) ended-then-stale-replay → NO new payment; (3) outage replay (delete rate bands first) → early probe absorbs before the rate lookup; (4) derivation mismatch (SQL-bump incumbent rate) → detection-only, no new row, Ok. Plus // @axis: conc store-level two-tx race: one Some, one None, one row (event discipline pinned by scenario 1; the staged-iff-Some branch lives in the private binary fn — recorded layering rationale).

U3 — invariant + BLOCKING_INVARIANTS; tests/invariants.rs root + tests/invariants/perdiem_placement_duplicate.rs (// @axis: fault): red-proof on a THROWAWAY DB (live-DB in-tx DROP INDEX would hold ACCESS EXCLUSIVE mid-battery), teardown-before-assert, loud setup failures; live-DB green fn. Blocking-classification xtask test: every BLOCKING_INVARIANTS name resolves to exactly one catalog entry. Docs: data-model-financial.adoc (fix stale index prose + add the lifetime index with contrasting semantics), ADR-053 amendment (per-diem idempotency + derivation compare), ADR-022 amendment (first arbiter-backed idempotent consumer), testing-reference selector fix (--run-ignored=all), CHANGELOG with destructive-migration disclosure.

Deploy runbook

Stop ALL craig-financial replicas → verify no sessions on craig_financial → recoverable backup → start ONE new-image instance (boot migration; fail-closed on hostile data) → validate payments_dedupe_repairs + counts + invariant sweep → scale out. Rollback to the old binary is unsafe (old writer lacks ON CONFLICT → replay = 23505 → handler error → DLQ). Dev volumes: expect one-time fail-closed on test residue → wipe + reseed.

Verification

  • Constraint probes: cargo nextest run -p craig-financial --test constraints --run-ignored=all (red pre-migration, green post).

  • Replay/conc/migration/invariant targets: same pattern with --test <target>. MR CI runs --lib --bins only — integration assurance is the local pre-push battery.

  • cargo xtask invariants sweep green post-battery; full battery after a devstack wipe+reseed (existing volumes carry residue the fail-closed migration refuses).

Edit this page · latest