Send-Jobs Claim Lease + Generation Fencing (double-send on rolling restart)

On this page

Status

Unit Description Status

U0

Choreography: label, issue amendment, #1195 filed (early-terminal gap), this plan

Done (2026-07-28) — commit adf302b3

U1a

Test seam: lib.rs exposes the store (visibility only, no behavior)

Done (2026-07-28) — commit e79b5804; NARROW inline lib module (3 files only) so unrelated store surfaces don’t become lib dead code

U1b

Migration + lease claim + generation-fenced finalizers + transaction guard + test matrix (RED evidence)

Done (2026-07-28) — commit 15236cf9; RED 5/6 against the pre-fix fns (only the FUSL single-winner pin passed); 8/8 green post-fix; J-review PASS with 4 minor flags remediated pre-commit (stale sweep comment, NULL-healing test added, refreshed-claimed_at assert strengthened, const-name doc)

U2

Docs: ADR-022 amendment, data-model-exchange send-jobs section, CHANGELOG

Done (2026-07-28) — send-jobs table + lease/generation semantics + index rows added to the previously-silent data-model page

Ship

Battery, MR, merge, close-out

Done (2026-07-28) — MR !1097 merged (961317e8); battery attempt 2 green (attempt 1 fast-failed at the axis cheap gate — stale opt-out for the deleted recovery test, re-blessed da5de851); #1185 closed with AC walk; epic &74 ticked (6/15)

Epic: &74
Issue: #1185 (Plan::SENDJOB-LEASE)
Follow-ups: #1195 (early terminal paths never stamp the transaction / stage exchange.failed — pre-existing, filed by U0; they DO get the generation fence here)
Review state: internal pass (Explore scout + Plan-agent verification, all anchors re-verified file:line). Approved 2026-07-28.

Context

exchange_send_jobs (craig-exchange’s partner HTTP send queue) kept the pre-#1128 claim pattern:

  • claim_pending (store/send_jobs.rs:61-82) stamps no claim time/owner — only status='in_flight', attempts+1; next_attempt_at keeps its scheduled-eligibility value.

  • recover_stuck_in_flight (:153-168) flips ANY in_flight row with next_attempt_at < now()-60s back to pending — the WRONG clock: a job claimed out of a >60s-overdue backlog (post-outage drain) is INSTANTLY "stuck", so a peer (re)boot re-claims it mid-send → two workers POST the same transmission.

  • finalize_sent (:87-99) and finalize_failed_terminal (:130-147) UPDATE by bare id — no status/owner guard (the retry path alone carries AND status='in_flight', proving the pattern was known) — so whichever racing worker finishes LAST wins, and both stage exchange.sent/exchange.failed outbox events.

  • update_transaction_status (store/transactions.rs:107-130) has no transition guard: stale success ↔ failed flips.

  • Recovery runs once at bootstrap (main.rs:223); drain_once deliberately abandons un-dispatched claims "for the recovery sweep" — which won’t run until the next boot.

  • The worker module doc’s "harmless to apply twice" idempotency claim is exactly what this issue falsifies.

  • Design-verification correction: the two early terminal arms in dispatch (partner-missing, resolve-miss) DO open a tx but skip the transaction stamp + events — pre-existing gap, now #1195; they get the fence here, not the missing stamp.

Design

# Decision Substance

D1

Migration <ts>_send_jobs_claim_lease.sql

ADD COLUMN claimed_at TIMESTAMPTZ, ADD COLUMN claim_generation BIGINT NOT NULL DEFAULT 0; DROP INDEX idx_exchange_send_jobs_stuck_in_flight (only consumer = the deleted sweep); CREATE INDEX idx_exchange_send_jobs_lease ON exchange_send_jobs (claimed_at) WHERE status = 'in_flight'. No backfill: legacy in_flight rows carry claimed_at NULL = expired lease = immediately re-claimable (deliberate healing of stranded rows).

D2

Claim absorbs recovery (single-statement UPDATE)

Inner SELECT WHERE (status='pending' AND next_attempt_at ⇐ now()) OR (status='in_flight' AND (claimed_at IS NULL OR claimed_at < now() - make_interval(secs ⇒ $2))) ORDER BY next_attempt_at LIMIT $1 FOR UPDATE SKIP LOCKED; outer SET status='in_flight', attempts=attempts+1, claimed_at=now(), claim_generation=claim_generation+1 RETURNING *. claim_pending(pool, batch_size, lease_seconds) — lease parameterized (tests pass 0). ORDER BY stays next_attempt_at (NOT NULL, shared monotone key — no starvation, no NULL trap). Index: bitmap-OR over the two partials; top-N sort bounded by live-queue cardinality (terminal rows retention-swept). DELETE recover_stuck_in_flight + run_recovery_sweep + the bootstrap call + STUCK_RECOVERY_THRESHOLD_SECS; recovery is CONTINUOUS (every 1s poll) — strictly supersedes the issue’s "periodic sweep" criterion. attempts+1 on reclaim matches old sweep→claim accounting.

D3

CLAIM_LEASE = 90s

Every adapter send is capped at 30s (per-adapter send_timeout, incl. SHINES; transports apply per request; shared-client 30s backstop) — 90s = 3× worst case (2× timeout + margin for the sub-second finalize tx + pool wait; lease comparison is DB-side now() only). The outbox’s 60s is too tight for a full partner send. Const with rationale beside the worker-policy consts.

D4

Generation-fenced finalizers, typed outcome

pub enum FinalizeOutcome { Applied, Stale }. finalize_sent(tx, id, claim_generation) / finalize_failed_terminal(tx, id, claim_generation, last_error) / mark_pending_for_retry(pool, id, claim_generation, next_attempt_at, last_error) — each AND status='in_flight' AND claim_generation = $n; rows_affected → Applied/Stale. NEVER name a parameter gen (edition-2024 keyword). Worker: the job-row CAS already runs first in both finalize txs; on Stale → rollback + structured warn! (exchange.stale_finalize_fenced) — no transaction stamp, no events (conflicting-event emission closed structurally: events stage only after Applied, same fn, same tx). Early terminal arms pass job.claim_generation via a shared terminal_fail_fenced helper (relieves dispatch’s 40-line budget). `ExchangeSendJob gains claimed_at/claim_generation (all loads RETURNING *; no other struct consumers).

D5

update_transaction_status terminal guard (defense-in-depth)

AND ($2 NOT IN ('success','failed') OR status NOT IN ('success','failed')). Retry-safe: retry_failed_transaction exits terminal via its OWN guarded failed→pending UPDATE (same tx as restage + correlation repoint) — terminal→terminal never legitimate. Call sites stop discarding the Option: None = the defense fired → warn! (don’t fail the tx; the job CAS is the source of truth). Spec drift ANNOTATED not refactored: can_transaction_transition models Failed→Retry→Pending (production goes failed→pending directly); can_send_job_transition gets a doc line that lease reclaim is expiry WITHIN in_flight.

D6

Test seam: lib exposes the store

lib.rs pub mod store; (the craig-financial pub mod subsidy precedent); store/mod.rs flips models/send_jobs/transactions pub; rustdoc on every flipped item. mod store; stays in main.rs (same-source dual-target is not the #1131 drift class — that was SQL copies, which this seam KILLS: the old recovery test’s inline schema + inline sweep SQL + mirrored const all die). Scratch fixture: hand-rolled 13-column exchange_transactions parent (every ExchangeTransaction field pre-correlation_id — RETURNING * + FromRow needs all; drift self-heals at decode) → raw_sql(include_str!(REAL 20260505185937 migration)) (adds correlation_id; uuidv7() native on devstack PG18) → the new lease migration. Real DDL under test, zero copy drift. No Cargo.toml changes.

D7

Recorded honesty

Duplicate (NOT conflicting) exchange.sent remains possible: crash after partner-commit, before finalize → lease expiry → re-send. At-least-once per ADR-022, bounded by the lease; craig_correlation_id = partner-side dedup, inbox dedup = consumer-side. Poison-job crash-loop reclaims (worker dies mid-send repeatedly; MAX_ATTEMPTS counts adapter errors only) match old sweep behavior — known limitation, not bolted on (a claim-side cap would strand rows). Mixed fleet: drain old replicas first (old binaries double-POST mid-send AND carry the unfenced bare-id finalize).

Units

U0 — label Plan::SENDJOB-LEASE; issue amendment note; #1195 filed + /relate; this plan + nav; cargo xtask plan-lint.

U1a — the seam commit: lib.rs pub mod store;, visibility flips + rustdoc. No behavior change; compile-green both targets; mechanical carve-out.

U1b — one commit: migration (D1); store rework (D2/D4/D5); worker + main.rs (D2/D4 incl. terminal_fail_fenced, module-doc rewrite killing "harmless to apply twice", D5 annotations); new test root tests/send_jobs_lease.rs (scratch-DB per retention_sweep.rs; axis tags immediately above the test attributes): (1) conc — fresh claim with ancient next_attempt_at NOT re-claimable; (2) happy — expired lease reclaimed with generation+1/attempts+1/claimed_at refreshed; (3) fault — stale finalize_sent fenced, row stays in_flight under the new generation, new-generation finalize applies; (4) fault — stale finalize_failed_terminal cannot overwrite sent (the issue’s named defect); (5) fault — stale retry cannot yank an active claim (documents why the old status-only guard was insufficient); (6) sad — transaction terminal guard refuses conflicting flips; retry’s guarded failed→pending is the only legal terminal exit; (7) conc — two concurrent claimers, one winner (new coverage). In-module string pin extended (FUSL + both lease predicates + the generation increment). The old throwaway-DB recovery test DELETED (superseded; its inline copies die). Live e2e stage_then_async_finalize unchanged. RED procedure: the fix re-signs the fns under test, so stash won’t work — with U1a committed, write the tests against the OLD fns, RUN, record the failures (sweep-flips-fresh-claim; stale finalize stamps over the new claim; sent→failed overwrite; stale retry yank), then apply the fix + mechanically update call shapes (assertions unchanged); RED output quoted in the commit message. Worker-level stale-skip needs no harness — events stage after the Applied branch of the same tx (layering rationale in the module doc).

U2 — ADR-022 dated == Amendment — #1185 (after #1183): the lease (90s + sizing), the generation CAS and why the outbox’s unguarded stamp was NOT ported, the D7 honesty items, the spec-drift annotations, drain-first. data-model-exchange.adoc: the page omits exchange_send_jobs entirely — add the table section (columns incl. the new two, status vocabulary, lease/generation semantics, indexes). CHANGELOG === Fixed with the drain-first deploy note. Plan Status per unit.

Verification

  • cargo nextest run -p craig-exchange --profile integration --run-ignored=all -E 'binary(send_jobs_lease)' (devstack up); joins cargo xtask validate’s workspace `--run-ignored=all battery automatically; no nextest test-group needed (unique scratch DBs).

  • Unit pins: cargo nextest run -p craig-exchange --lib (string pin) + the bin-module const pins.

  • Targeted clippy -p craig-exchange --all-targets --locked — -D warnings; budget hazards recorded (B3b: seed via json!; axis tags; fn budgets — terminal_fail_fenced relieves dispatch; gen keyword).

  • Full battery pre-push. No devstack reseed (additive migration; NULL claimed_at heals).

Out of scope (routed)

  • Early-terminal transaction stamp + exchange.failed event → #1195 (they DO get the fence here).

  • Poison-job crash-loop reclaims → known limitation, ADR amendment.

  • can_transaction_transition Retry-path spec drift → annotated only.

Edit this page · latest