Plan: Per-Jurisdiction Search-Scheme Override

On this page

Status

Unit Description Status

U0

Plan + epic &80 + child issues + follow-ons (#1399, #1400)

Done (2026-08-11) — epic &80 + this plan page

U1

craig-state-bundle contract — SchemeName + SearchSchemeSpec + defaulted search_schemes() (#1392)

Done (2026-08-11) — this MR

U2

craig-cases-fields pure allowlist resolver + canonical spec codec (#1393)

Done (2026-08-11) — this MR (as-built deltas: validate_registry + its three slice helpers + check_blind_index_base relaxed off &'static in craig-search — semantically free, required so the pure owned resolver can validate without leaking; DEFAULT_REGISTRY added here as the single-sourced 5-pair covered set; resolver also validates override-free defaults so RegistryInvalid is reachable; marker readers get effective_from_markers with duplicate/version/unknown-row refusals)

U3a

craig-cases as the 8th bundle consumer — features, matrix, compose, Dockerfile (#1394)

Done (2026-08-11) — this MR (as-built deltas: the boot call lands in run() as VALIDATION-ONLY — activation + jurisdiction↔bundle coherence + override resolution all refuse startup, the resolved registry itself unused until U4b; the jurisdiction mismatch refuses BOOT outright, deliberately harder than financial’s per-lookup degradation)

U3b

crypto_field_lineage marker + migrate-gate reconcile + serving verify (#1395)

Done (2026-08-11) — this MR (as-built deltas: the gate hook rides a new generic post_apply seam threaded through craig-api’s run_process_modes_with_post_apply / run_migrate_gate_with_post_apply — the seven hook-less services keep the plain wrappers via a shared no_post_apply; the migration’s REVOKE is role-CONDITIONAL (the fleet’s first role-referencing migration — test-plane/bare-postgres DBs have no app principal); the cases gate env gains the bundle selector + jurisdiction (+ mode) — the recorded cases-specific extension of the ADR-063 one-var contract, mirrored in compose; the gate’s mode parse defaults required exactly as ServiceSettings does)

U4a

Planner runtime effective-scheme path — presence-gated, fail-closed (#1396)

Done (2026-08-11) — this MR (as-built deltas: the effective-spec LOOKUP is presence-INdependent — an incomplete registry fails every plan over the entity, only the capability re-check is presence-gated; EffectiveRegistry::new re-runs validate_registry so a malformed slice is never installable; the absent-value Opaque equality arm emits $N IS NULL OR FALSE (fail-closed on the value side, sibling no longer referenced); substring emission stays compiled-target (column names are the registry key — only the per-constituent capability consults the effective spec); relations documented registry-neutral — the bound value compares against VIA-table columns this registry does not describe)

U4b

Resolved registry threaded through every boundary (#1397)

Done (2026-08-11) — this MR (as-built deltas: the leak-once interning is single-sourced in the new craig_cases_fields::intern leaf module (shared by the service materializer + the seed — a cross-crate B8 duplicate otherwise); the service materializer + the seed BOTH short-circuit the no-override path to the compiled ALL consts — zero allocation, VALUE identity pinned by test (a const materializes per use-site, so a pointer pin is unobtainable by language rule); craig-seed resolves BUNDLE-SIDE via the same pure resolve() instead of marker rows (it is a DB-less SQL generator — Architecture diagram corrected); EffectiveRegistries::from_parts added as the harness/e2e constructor powering the HTTP 400 test pre-activation; the five formerly-inlined handler mode-matches consolidated into api/encryption::search_runtime, which takes the registry by VALUE so a handler cannot forget it)

U5

Activation — tx-stub override, lifecycle fingerprints, e2e proof, ADR-049 amendment (#1398)

Done (2026-08-11) — this MR (as-built deltas: the integration proof runs through the PRODUCTION resolver + materializer against the keyed harness (tx-stub is not compiled into devstack images — CRAIG_CASES_FEATURES build-arg, recorded; the tx-stub declaration is pinned in its own crate); the A6 divergence test lands at the verify-seed sibling cross-check (Opaque-written rows fail a BlindIndex reconstruction); the identity recipe bump rides MARKER_PROTOCOL_VERSION 1→2 per the golden test’s own protocol; the gate now stamps the GA shape (ONE row) on every deployment — the U3b empty-shape pin flipped to its designed post-activation form)

Epic: &80
Issues: #1025 (anchor; closed by U5), #1392 #1393 #1394 #1395 #1396 #1397 #1398 (children), #1399 (U6 follow-on: mid-life migration tooling), #1400 (capabilities ∩ descriptor ops)
Branches: feature/1025-u<N>-*

Plan v2 was externally reviewed 2026-08-11 (~40 findings, blocking) and this v3 folds ALL of them; the reviewer’s "sound direction" is adopted verbatim. Every file:line below was verified against main @ 0e213d2a by three dedicated source-verification passes (allowlist basis, read/planner surface, deployment/roles/workers).

Context

ADR-049 (capability-based encrypted search) made the craig-cases field registry config-ready for per-jurisdiction encryption schemes, but the override wiring was deferred out of C3 v1 (user scope decision 2026-07-13). Today the craig-cases-fields registries are compile-time constants: every deployment encrypts the same fields the same way. Multi-jurisdiction deployments (ADR-032) need per-deployment field encryption — the reference case: a Texas deployment stores persons.ssn_last_four as Opaque (unsearchable ciphertext) where Georgia’s default is BlindIndex (equality-searchable via the HMAC sibling).

The invariant that shapes everything (ADR-049 §Blind-index domain separation): a field’s at-rest scheme is a property of the stored data. The blind-index HKDF domain is baked into the derived subkey, so a scheme change is a migration + reseed — never a runtime toggle. An Opaque→Plaintext flip would reinterpret ciphertext as searchable plaintext (a PII leak). This plan therefore wires the override fail-closed: fresh deployments work end-to-end; any scheme change over existing data is refused pre-serving until migration tooling (#1399) exists.

What source verification proved (and the rejected v2 got wrong):

  • Eligibility is an allowlist of ONE. Only persons.ssn_last_four is crypto-mediated end-to-end with a safe alternative scheme: BlindIndex ⇄ Opaque{Text} — the at-rest bytes are identical under both, only the ssn_hmac sibling toggles, and the sole affected index is partial (idx_persons_ssn_hmac, unused under Opaque). Everything else fails the gate: cases
    investigations have zero crypto accessors (api/encryption/rows.rs); the Opaque reporter/narrative/JSONB fields have no safe target (→BlindIndex needs a sibling _hmac column no migration created; →Plaintext exposes PII); full ssn is load-bearing (dedup/attestation/federal export); the rest carry CHECK / UNIQUE / trigram / scope / lifecycle SQL that assumes plaintext. →Plaintext is ineligible for every allowlisted field.

  • The stamping actor is the ADR-063 migrate gate, not the seeder — production has no fixture seeder, and the devstack seed gates on healthy craig-cases (docker-compose.yml:1246), so seed-time stamping would deadlock fresh boot.

  • SearchRuntime::new has 37 call sites — 5 production (one per entity list handler) + 32 test.

  • Keyless + Optional stores plaintext under encrypted labels (craig-search/src/row.rs:251-252) — overrides must require EncryptionMode::Required.

  • Neither devstack staleness (xtask/src/devstack_guard.rs:351-377) nor the _seed_marker identity (tools/craig-seed/src/seed_marker.rs:74-99) binds the active bundle or a registry fingerprint today — a host-only override flip would silently skip recreate + reseed.

Scope

In scope:

  • The override contract (StateBundle::search_schemes(), defaulted), the shared-leaf resolver
    allowlist, the crypto_field_lineage marker + gate reconcile + serving verify, the planner runtime path, full boundary threading, deployment wiring (features/matrix/compose/Dockerfile), lifecycle fingerprint binding, and atomic activation with the tx-stub reference override.

Out of scope (filed as issues, not deferred silently):

  • Mid-life scheme migration over existing data — #1399. Until it lands, the gate refuses the change with the remedy named.

  • Capabilities ∩ declared-descriptor-operations intersection — #1400 (pre-existing over-advertisement, orthogonal to overrides which only narrow).

  • Widening any field’s search capability (e.g. Opaque→Plaintext): permanently a code change
    migration, never a bundle override.

Design

Decisions

Fork Decision

Eligibility

Hard-coded allowlist: v1 = persons.ssn_last_four → {BlindIndex, Opaque}. Every other (entity, column) refused by resolve(). Extending the allowlist = code + migration, never config.

Surface

DEFAULTED StateBundle::search_schemes() trait method (mirrors subsidy_policy() / uas_codes(), crates/craig-state-bundle/src/bundle.rs:44-54). A bundle names a SchemeName only; every scheme parameter comes from the compiled registry. (The issue’s BundleContribution.search_schemes name predates the trait-method pattern; per-jurisdiction identity maps are trait methods, not mergeable axes.)

Resolver home

Shared leaf craig-cases-fields, with its own neutral SchemeOverride type (the service maps the bundle type onto it). Pure + owned output; no leak inside resolve() — a separate boot materializer interns once to &'static (craig-search’s deliberate immortal-registry contract: encrypt_row/Projection/emit_* all traffic in &'static [&'static FieldSpec]).

Marker

crypto_field_lineage rows keyed (table_name, column_name) storing the full canonical spec (scheme kind, at_rest, field_domain, version, canon, codec, sibling, logical_type — ONE shared codec in the leaf) + spec_version. Exact-set over the allowlist: extra / missing / malformed rows ⇒ refuse.

Stamping

The ADR-063 migrate gate (owner role, pre-serving, parallelism-1) reconciles the marker insert/equal-only. Serving boot verifies read-only, presence required (the gate is mandatory per ADR-063 — no absent-arm at serving). Runtime app role: SELECT-only via explicit REVOKE (the devstack/postgres/init.sql:42-46 hardening pattern).

Lookup

Entity-keyed (table, column) — column names collide across entities (admin_unit, status, reporter_type, …). A planner miss when a registry is present is an invariant error — never a fallback to the const scheme.

Runtime check

Fires only on a PRESENT filter value (the planner visits every declared predicate; presence lives in the bind’s inner Option — plan.rs:186/222 NULL-guard). Concat/AnyOf all-constituent atomic.

Sorts / summary / scope

Static invariant tests pin: no allowlisted field appears in any SORT_OPTIONS, any default_sort, SUMMARY_COLUMNS, or a scope column — the gaps are structurally unreachable rather than runtime-checked.

Encryption mode

Overrides declared + EncryptionMode != Required ⇒ gate and boot refuse (keyless-Optional stores plaintext; a confidentiality override would be a lie).

Activation

Atomic: the production allowlist constant stays EMPTY until U5; tests exercise the machinery via parameterized allowlists. tx-stub declares its override only in U5 with the last boundary. GA acceptance = behavioral/wire compatibility.

Bundle name / env

tx-stub (crates/craig-state-tx-stub/src/lib.rs:63); CRAIG__ACTIVE_STATE_BUNDLES (double underscore; compose anchor docker-compose.yml:43).

Single-writer, many-readers

The migrate gate is the only writer of scheme truth; everyone else reads the marker or the compiled defaults:

                    +-- compiled defaults (craig-cases-fields consts)
resolve() ----------+
  (gate + serving)  +-- bundle.search_schemes() -> leaf SchemeOverride (allowlist-checked)
        |
  gate: reconcile crypto_field_lineage (insert/equal-only; refuse on conflict)
        |
  serving boot: verify marker == resolved (read-only; presence required) -> materialize &'static
        |
  &'static ResolvedRegistry --> Extension (5 HTTP list handlers, wrappers, capabilities endpoint)
                            --> inbox Deps (report_converted -> matching -> ssn_promotion)
                            --> boot_verify lineage-absent scan (compiled DEFAULT_REGISTRY --
                                pre-resolution arm; both schemes share the at-rest envelope)
  xtask verify-seed         --> reconstruct effective registry FROM marker rows + defaults
                                (shared leaf codec -- no second resolution to drift)
  craig-seed                --> bundle-side resolve() (same pure leaf; the seed is a DB-less
                                SQL generator, so it CANNOT read markers -- divergence from the
                                gate-stamped truth is caught by the serving verify + verify-seed)

Gate reconcile (5 cases, insert/equal-only)

  1. marker present, == resolved → no-op.

  2. marker present, != resolved → refuse (scheme change over existing data; remedy: reseed now, #1399 tooling later). Never overwrite.

  3. absent + covered columns provably clean (bounded scan, mirrors services/craig-cases/src/boot_verify.rs:229) → INSERT resolved. (Fresh deploy — devstack and production.)

  4. absent + data present + resolved == compiled default → INSERT default. Sound, not grace: pre-marker binaries had no override mechanism, so pre-marker data is provably default-written.

  5. absent + data present + resolved != default → refuse (unattested override over live data).

Planner path (craig-search; the leaf stays free of state-bundle/cases-fields)

SearchRuntime gains Option<EffectiveRegistry> (&'static [&'static FieldSpec] + table binding asserted against SearchEntity::table()). In emit_equality / emit_substring / emit_blind_equality (crates/craig-search/src/plan.rs:212-293): when the bind’s inner Option is Some, resolve the effective spec (registry present + miss ⇒ invariant error), re-check capability(effective.scheme, logical_type) ⇒ SearchError::UnsupportedFilter → 400 no-oracle (error.rs:23-29; craig-common/src/error.rs:691). Absent values keep today’s NULL-guard no-op and stable bind counts. blind_index derives off the effective spec. The capabilities endpoint (api/search_capabilities.rs:23) computes from the resolved registry — an Opaque’d ssn_last_four honestly reports equality: false; since v1 transitions only narrow, nothing new is over-advertised (the pre-existing capability-vs-descriptor gap is #1400).

Steps

Every unit: own MR on feature/1025-u<N>-*, full battery, Status row update here, CHANGELOG entry when operator-visible. Every unit ≤ ~500 LOC (splits pre-designed). GA behavior/ wire-compatible at every stage (empty production allowlist until U5).

U1 — Contract (#1392, w1)

Files: crates/craig-state-bundle/src/search_scheme.rs (new), src/lib.rs, src/bundle.rs.

SchemeName {Plaintext, Opaque, BlindIndex} + SearchSchemeSpec { entity, column, scheme } ('static plain data, the UasCodeSpec pattern); defaulted fn search_schemes(&self) → Vec<SearchSchemeSpec> { Vec::new() } after uas_codes() (bundle.rs:52). GA/tx-stub inherit the empty default unchanged. Pure additive.

U2 — Leaf resolver (#1393, w3)

Files: crates/craig-cases-fields/src/resolve.rs (new) + lib.rs.

Neutral SchemeOverride; ALLOWLIST const (empty in production until U5); pure owned resolve(defaults, overrides, allowlist) → Result<ResolvedRegistryData, ResolveError> with errors NotAllowlisted / UnknownEntity / UnknownColumn / UnsupportedTransition / DuplicateOverride / RegistryInvalid (post-resolution craig_search::validate_registry — a resolution can never yield an invalid registry silently). The canonical-spec codec (complete scheme serialization, shared by the marker writer and every reader) + the marker-row → effective-registry reconstruction helper. Proptests: resolution total/exact; codec round-trip; overrides-empty ⇒ resolved == compiled defaults.

U3a — Consumer wiring (#1394, w2)

Files: services/craig-cases/Cargo.toml, src/bundle.rs (new), xtask/src/cmd/feature_matrix.rs:12-42, docker-compose.yml:370-394, Dockerfile:41-73.

Cases becomes the 8th bundle consumer: state-ga/state-tx-stub features + optional deps (financial template Cargo.toml:15-17); candidate_bundles() wrapper + compile_error! guard (financial/src/subsidy/bundle.rs:38-47); fail-closed jurisdiction↔bundle check (mirrors :19-28). FEATURE_MATRIX_CRATES 7→8 + the "SEVEN"/"7-crate" prose (the fingerprint test feature_matrix.rs:182-215 forces this). Compose: cases gets *active_state_bundles. Dockerfile: CRAIG_CASES_FEATURES build arg wired into BOTH cook and build (the CRAIG_WEB_FEATURES cache-key precedent).

U3b — Marker + gate + boot (#1395, w3)

Files: services/craig-cases/migrations/<ts>_crypto_field_lineage.sql (new), src/boot_verify.rs, the cases migrate-gate path, src/lib.rs:133.

Migration: the marker table + explicit REVOKE INSERT/UPDATE/DELETE FROM craig_cases_app (runtime SELECT-only; the gate runs as owner; the devstack seed principal is the craig superuser and unaffected). Gate: the 5-case reconcile (§Design) appended to cases' ADR-063 migrate path, plus the EncryptionMode::Required requirement when overrides are declared. Serving boot: verify_field_lineage — read-only, presence required, exact-set — after verify_key_lineage. Tests: all 5 gate arms; exact-set violations; mode refusal; the case-4 upgrade backfill.

U4a — Planner (#1396, w3)

Files: crates/craig-search/src/plan.rs, src/runtime.rs, tests/planner.rs (14 call sites).

EffectiveRegistry + the SearchRuntime field + new() signature change (pre-1.0, no shim); presence-gated capability re-check + no-fallback lookup in emit_*; Concat/AnyOf atomic. Tests: present-value → UnsupportedFilter; absent value → NULL-guard no-op with stable binds; registry-miss / wrong-entity slice → invariant error; None-registry path byte-identical.

U4b — Service threading (#1397, w3)

Files: services/craig-cases/src/lib.rs (materializer + Extension + Deps), the 5 handlers (api/persons.rs:109, api/referrals.rs:211, api/cases/crud.rs:344, api/reports/crud.rs:258, api/investigations/crud.rs:213 — consolidating the 5× inlined mode-match), 18 service test call sites, api/encryption/{person,referral,report}.rs, boot_verify.rs scan, api/search_capabilities.rs, tools/craig-seed, xtask verify-seed.

Boot materializer interns resolved → &'static; Extension AND inbox Deps (the non-HTTP leg: report_converted → matching → ssn_promotion); every encrypt/decrypt/scan/capabilities boundary repointed at resolved. verify-seed reconstructs from marker rows via the U2 leaf helper; craig-seed resolves bundle-side through the same pure resolve() — it is a DB-less SQL generator (verified as-built) and cannot read markers; divergence from the gate-stamped truth is caught by the serving verify + verify-seed. Static invariant tests: allowlist ∩ SORT_OPTIONS / default sorts / SUMMARY_COLUMNS / scope columns = ∅, plus every equality/substring predicate field ⊆ its registry slice. HTTP 400 integration test.

U5 — Activation (#1398, w3; Closes #1025)

Added by the U4b review (A6): with the allowlist non-empty, add the executable divergence test the recorded seed deviation promises — a seed generated under a bundle whose schemes disagree with the gate-stamped markers must FAIL verify-seed (and the serving verify), so "divergence is caught downstream" stops being prose.

Files: crates/craig-cases-fields/src/resolve.rs (allowlist), crates/craig-state-tx-stub/src/lib.rs, tools/craig-seed/src/seed_marker.rs:74-99, xtask/src/devstack_guard.rs, ADR-049, CHANGELOG.adoc.

The allowlist gains persons.ssn_last_four → {BlindIndex, Opaque}; tx-stub declares the Opaque override; the registry fingerprint enters _seed_marker compute_identity AND a devstack_guard staleness input binds CRAIG__ACTIVE_STATE_BUNDLES (the field-key KCV-marker pattern) so a host-only override flip is never AlreadyCurrent. Integration proof on the test-plane DB: write a person under the tx-stub-resolved registry → raw SELECT asserts envelope ciphertext
ssn_hmac IS NULL; the ssn filter → UnsupportedFilter/400; gate-refusal arms (2/5). ADR-049 amended as-built (Consequences bullet + == Amendment — #1025). CHANGELOG; plan Status → Done; archive per lifecycle.

Verification

# Scenario Level Expected

1

Gate: all 5 reconcile arms (fresh-clean insert; provably-default backfill; override-over-data refuse)

gate unit/integration

per-arm; refusals name the remedy

2

Marker mismatch; extra/missing/malformed rows; mode≠Required with overrides

unit

refuse, typed errors

3

Serving boot with marker absent

unit

refuse ("run the migrate gate")

4

Present ssn filter under Opaque override

planner unit + HTTP

UnsupportedFilter → 400 no-oracle

5

Absent filters (equality/substring/Concat/AnyOf) under override

planner unit

NULL-guard no-ops, stable bind count

6

Registry present + descriptor field missing / wrong-entity slice (colliding columns)

planner unit

invariant error, never silent fallback

7

Sorts / summary / scope untouched

static tests

allowlist ∩ each = ∅

8

resolve() totality; post-resolution validate_registry

leaf proptests

typed errors; never an invalid registry

9

Fresh tx-stub deploy end-to-end

integration (test-plane DB)

gate stamps → boot verifies → raw DB shows envelope + ssn_hmac IS NULL → search 400

10

Existing GA DB upgrade (data, no marker)

gate integration

case-4 backfill, serves — behavior-preserving

11

GA default path, every unit

regression

resolved == consts; wire behavior unchanged; full battery green

12

Capabilities under override

integration

ssn_last_four.equality == false from resolved

13

Devstack host-only override flip

xtask test

staleness + seed identity flip → reseed, never AlreadyCurrent

14

Runtime role cannot write the marker

integration

INSERT/UPDATE/DELETE as craig_cases_app → permission denied

References

  • ADR-049 (capability-based encrypted search; §Blind-index domain separation; amended at U5)

  • ADR-063 (migration gate — the marker’s pre-serving writer), ADR-048 §D3 (boot_verify model)

  • ADR-032 (state bundles), the subsidy_policy()/uas_codes() DEFAULTED-method precedent (#1072)

  • plans/archive/c3-encrypted-search.adoc §Scope (Out) — the original deferral

  • #1025 (anchor) · epic &80 · #1399 (mid-life migration tooling) · #1400 (capabilities ∩ descriptors)

Edit this page · latest