ADR-061: Worker Supervision & Readiness Gating

Pre-1.0 behavior changes: /readyz gates on the MQ publisher connection (for the first time, despite prior documentation) AND on critical-worker liveness; its 503 body is informative; ApiServer::serve takes the token (bounded drain); AppState changes shape.

The observable on critical death is "503 on open connections, then connection refusal, then process exit" — orchestration reacts to refusal exactly as to 503, so the pod leaves rotation either way; this ADR deliberately does NOT claim a fresh network probe reliably sees the 503 during the drain window.

Devstack has no compose restart policy: a fail-fasted service stays down LOUDLY (cargo xtask dev start is the remedy) — consistent with the fail-loud doctrine.

Scope limitation (stated): this is HANDLE-liveness, not functional health. A hung worker, a reconnect-forever supervisor, or persistent warn-and-continue failures remain Running. #1231 (2026-08-04) ships the functional-health complement: the worker_last_success_timestamp_seconds{worker=…​} gauge family (craig_common::metrics::worker_heartbeat) — each instrumented loop records its last SUCCESSFUL pass (failure paths stay silent, so staleness alerts honestly; metrics-only, never coupled to readiness per the issue’s AC). Wired at the shared-crate seams that cover the fleet uniformly: the outbox drain (injected as a plain fn hook so craig-mq stays craig-common-free), the three spawn_local_sweep consumers (request-claims-retention, financial reconcile-sweep, exchange send-jobs-sweep), the JWKS/OIDC refresh loops (now name-threaded so the gauge label matches the registry name), and authz-ttl-refresh. Alert guidance: deployment guide § Worker functional-health alerting.

#1321 (2026-08-05) completes the tail — the consumer-liveness tick + the per-service loops. Subscriber design (the amendment #1231 deferred): a per-message heartbeat would page on every quiet queue, so the honest signal is a consumer-liveness tick — a third tokio::select! arm on a per-session 30 s interval (LIVENESS_TICK_PERIOD, missed-ticks skipped) inside run_session and run_dlq_session: it beats exactly while a consume session is LIVE (consumer registered, channel open; the interval’s first tick fires on session entry) and is structurally silent during reconnect backoff. Injected as the same craig-common-free Option<fn()> hook as the outbox. Adopted by every subscriber worker: events-subscriber ×6, audit-subscriber, dlq-subscriber (tick in the consumer arm only — one registry name, one tick source; the #1232 child-token teardown stops the tick when the consumer arm dies, so the replayer/sampler arms cannot mask it), cache-invalidation, composition-invalidation, and authz-invalidation. The per-service loops adopt record_success at their genuine-success points: decision-refresh (probe-clean pass — the gauge #1219 deferred), jws-cleanup, upload-attempt-reconciler ×4 (ONE shared label like outbox; beats only on a clean three-step pass), send-worker (clean claim/dispatch-spawn — per-job failures are async by design, the exchange isolation exception), detection-scheduler, subsidy-generator + review-sweep (EXECUTED runs only — a peer-held lease is not this replica’s success), and retention-archive. The two sibling gauges stay named exceptions, not aliases: archive_last_success_timestamp (ADR-058; 0-at-boot, unlabeled, no _seconds suffix) and subsidy_sweep_last_success_timestamp_seconds (engine-scoped — fires for manual runs too, lease-winner only — a different signal from the scheduler-tick heartbeat deliberately kept alongside it).

OS-thread coverage (#1228, 2026-08-04): the two dedicated !Send-eval OS threads (rules zen-eval, the fleet-wide authz-eval) — outside the JoinHandle model this ADR was written against — register through WorkerHealth::watch_liveness: a sentinel task select!`s the registry token against a death-signal future (the eval channel’s receiver dropping; the future holds its own sender clone, so the loop cannot exit by all-senders-dropped while watched — a pre-shutdown resolution is unambiguously thread DEATH, never a clean exit). Critical tier: a dead eval thread fails every authz-gated request / rules evaluation typed — strictly worse than the staleness risk that already made the authz cache legs Critical — and no durable poison input exists to re-trigger a crash post-restart (the exchange isolation exception does not transfer). The sentinel cannot carry the panic payload (`Dead { panicked: false } with the thread’s own stderr panic output as the diagnostic), and the handle-liveness scope limitation applies unchanged: a wedged-but-alive thread reads Running; the #784/#1126 per-call budgets own the caller experience.

Own-router services (#1229, 2026-08-04) — the three services outside ApiServer::router got their supervision posture decided and recorded:

Recorded residuals: a watcher’s own panic is the one unsupervisable task class (its classification core is pure and unit-tested; the record is a parking_lot write); the wrong-token class was reduced, not eliminated at this ADR’s landing — eliminated for router-internal workers by #1233 (2026-08-03): ServerOptions.shutdown (the second channel, whose Default was a detached never-cancelled token) is DELETED; router-internal workers now derive shutdown from WorkerHealth::token() — the registry they register with — so token and watcher share one authority by construction (pinned by pruner_exits_when_the_registry_token_cancels; main-spawned workers still hand-thread the supervisor token, where a foreign token remains representable — the exact-set pins and the supervision-chain fault tests own that surface); the tie-window (a clean death racing an unrelated signal within scheduler latency) loses only a diagnostic, never availability.

Exchange isolation exception: the send worker’s per-job dispatch panics are contained (craig_mq::contain) and logged with the job id but deliberately do NOT propagate to the Critical parent — a poison job that panics deterministically must not crash-loop the whole service; the #1185 claim lease recovers the job, and the in-flight JoinSet is drained bounded on shutdown. Counter metric is #1230.

Interior mutability: the registry is a project-authored Arc<parking_lot::RwLock<BTreeMap<..>>> — sanctioned here per the conventions rule requiring an ADR for project-authored interior mutability (short, non-async critical sections; no lock held across .await).

Implementation plan: Worker Supervision & Readiness (mechanics, tier table, test matrix, review log). Program: child epic &76; units #1220/#1221/#1222/#1223/#1186/#1224/#1225.

ADR-003's "readyz reflects the publisher path only" statement is superseded in part by this ADR (its Amendment #1186 points here): the publisher path genuinely gates readiness, and a subscriber SUPERVISOR’s death gates it too — while the #1127 self-heal semantics inside a live supervisor are unchanged.

Whether Missing should distinguish "expected, boot still in progress" if a future service ever registers workers after serve binds (today all 8 register strictly before bind; revisit only if that ordering changes).

Publisher-CHANNEL health vs the parent connection — RESOLVED by #1235 (2026-08-03): MqHealth carries the #1128 confirm-selected publisher channel and readiness gates on BOTH (disconnected vs publisher channel closed as structured healthz categories). lapin channels close permanently, so the readiness flip’s restart is the supervised recovery path; callers composing their own channels via connect_mq keep connection-only gating until they attach one (recorded carve-out).

Shared JoinSet supervisor — rejected: handles come from heterogeneous spawn seams (mains, boot helpers, the router); a JoinSet erases per-worker identity unless every seam is rebuilt around task IDs, and it cannot express Disabled/Missing.

Typed worker-exit protocol (every loop returns an exit reason) — rejected: causality becomes explicit, but it rewrites every worker loop fleet-wide for information the order-serializing watcher already recovers at the supervision layer.

In-process respawn with backoff — rejected per the user-decided policy: post-panic shared state is untrusted (money-writing generators, claim leases), the lint wall makes panics rare enough that each one is a real defect, and process restart restores clean state for free via the orchestrator.

Mark-not-ready-only (no exit) — rejected: readiness alone never restarts a pod; the result is a zombie requiring paging. Fail-fast converts the failure into the orchestrator’s native recovery loop.

Two-phase readiness window before cancel (so probes reliably observe the 503) — rejected: adds a delay knob for a purely cosmetic distinction; refusal fails probes identically, and the drain window already surfaces the 503 on open connections.