ADR-049: Capability-Based Encrypted Search (search capability = f(scheme, logical_type))

On this page

Status

Accepted (2026-07-13). Resolves the open product/architecture decision that was epic &66’s C3 child (Make craig-cases Field Encryption Work End-to-End, row C3). C3 is decomposed into its own sub-epic (&67) governed by Capability-Based Encrypted Search; this ADR is that sub-epic’s MR0 (#983).

As-built (2026-07-13, #1021). Sub-epic &67 is implemented in full — MR1–MR6 merged 2026-07-13 (!971 framework, !972 reports, !973 referrals, !974 persons + the SSN domain migration, !975 cases/investigations + the scope channel, !976 the registry-driven write path), each after a 3-reviewer adversarial J1–J8 pass. The as-built deltas are folded into the plan’s Design (zero plan↔code diff); the load-bearing ones:

  • Descriptor home. The per-entity SearchEntity descriptors live in the craig-cases service (api::persons_search / api::referrals_search, and the nested api::{reports,cases,investigations}::search), next to the wire DTOs they bind (which pull sqlx); craig-cases-fields stays the pure sqlx-free capability registry both sides consult.

  • The write path generalized into the framework. craig_search::row (core, DB-free) carries FieldRef/EncryptableRow/impl_encryptable_row!/encrypted_fields(), the encrypt_row/decrypt_row walks, and the JSONB {"v": ct} envelope. The service’s encrypt_field/decrypt_field primitives are retired; the ADR-020 fail-closed matrix is preserved uniformly typed (SearchError::{MissingKeyRequired, CiphertextWithoutKey}, plus the fail-loud stale-shape alarms RowMissingColumn/RowMissingHmacSlot — a registry-encrypted column a write shape does not expose is a hard error, never a silent plaintext write).

  • One blind-index derivation. FieldSpec::blind_index (registry BlindIndexSpec canon
    craig_crypto::hmac_domain) is THE single derivation, consumed by the planner’s query hash, the service write, the seeder, and verify-seed — the ("ssn", 1, AsIs) tuple lives only in the persons registry. validate_registry additionally rejects an out-of-envelope blind-index base (UnsupportedBlindIndexBase) at validation time instead of a runtime canon-mismatch 500.

  • Substring shapes. SubstringTarget gained AnyOf (per-column OR — the pre-C3 referral semantics, preserved exactly) alongside Field and Concat (the persons trigram concat).

Per-endpoint search-semantics changes shipped with the retrofits (LIKE-escape everywhere, the reports repoint, the persons full-name seam) are catalogued in CHANGELOG.adoc == Unreleased (the five C3 entries). The committed API-reference pages are refreshed under #1022 (the generator scrapes running containers; see that issue).

Builds on ADR-020 (fail-closed field-helper semantics + the self-describing CGEF ciphertext envelope) and ADR-048 (key lifecycle). It changes neither: this ADR governs which searches are possible over which schemes and how the query layer is built, not the at-rest primitive. C3 is the last blocker before C7 activation.

Context

craig-cases field-encrypts PII (ADR-020: craig_crypto::FieldEncryptor, AES-256-GCM-SIV, CGEF envelope). Three list/search endpoints coupled encryption to searchability independently, and they diverged:

  • Reports — broken. The reports list/count endpoints substring-ILIKE narrative, reporter_first_name, and reporter_last_name (store/reports.rs:120-142 for the list, :151-168 for the count — the WHERE clause is hand-duplicated, so it can drift). But encrypt_report_pii (api/encryption/report.rs:99-118) encrypts exactly those columns to randomized CGEF ciphertext. Under EncryptionMode=Required the ILIKE matches nothing, so report substring search silently returns an empty page while the contract (crates/craig-cases-contracts/src/reports.rs:46) advertises substring search — a capability the storage layer cannot deliver.

  • Persons — blind index. SSN last-four exact match swaps the plaintext term for a deterministic HMAC blind index before the query (api/persons.rs:92-96), searching a sibling ssn_last_four_hmac column. This works, but the HMAC uses the single global blind- index domain (craig-crypto/src/lib.rs:172, BLIND_INDEX_INFO_V1) shared with every other blind-indexed column. Name search uses per-column ILIKE`s (`store/persons.rs:137,:163) that miss the GIN trigram index built on the first_name || ' ' || last_name concat expression (migrations/20260426193817_report_persons.sql:31).

  • Referrals — correct. Referral search is a correct plaintext instance (store/referrals.rs:171-230), but hand-made and independent of the other two.

Each instance re-derives, by hand, the same latent rule — what can I search given how this column is stored? — and one of the three got it wrong silently. CRAIG is also going multi-jurisdiction (ADR-032): different deployments will want different fields encrypted, so the encryption↔searchability mapping cannot be a per-field constant baked into hand-written SQL. The rule needs to be data, resolved in one place, so that changing a field’s scheme cannot leave a stale, capability-lying query behind.

Decision

A field’s search capability is a pure function of its (encryption_scheme, logical_type), resolved through one registry of canonical FieldId`s. Every boundary that touches a field — search predicate, sort, the write/encrypt path, decrypt, the seeder, `verify-seed, and the trigram query — references a FieldId resolved through that registry, so there is exactly one source of truth for how each column is stored and what may be done with it.

Capability as a total function

enum LogicalType { Uuid, Text, Date, Timestamp, Int, Bool, Jsonb }
enum OpaqueKind  { Text, Jsonb }
enum FieldScheme {
    Plaintext,
    Opaque     { at_rest: OpaqueKind },
    BlindIndex { at_rest: OpaqueKind, spec: BlindIndexSpec, sibling_hmac: &'static str },
}
struct Capability { substring: bool, equality: bool, ordering: bool }
const fn capability(scheme: &FieldScheme, ty: LogicalType) -> Capability;
scheme \ type Text Uuid/Int/Timestamp Date Bool Jsonb

Plaintext

substr+eq+ord

eq+ord (no substr)

eq+ord

eq

none

BlindIndex

eq

eq

eq

eq

eq iff canonicalizable

Opaque

none

none

none

none

none

Rules: substring ⇒ Plaintext AND Text; ordering ⇒ Plaintext AND an orderable type (not Jsonb; a UUID column is orderable — CRAIG uses time-ordered UUIDv7 for its keys); equality per type. The predicate, sort, and relation constructors are const fn that assert! the required capability against capability(spec.scheme, spec.logical_type), so an illegal scheme/operation combination is a const-eval build failure (Rust edition 2024) — a search over an Opaque column is structurally unrepresentable, not merely unlikely. A mandatory registry_is_well_formed unit test additionally checks every predicate/sort/relation is capability-legal and every sibling_hmac names a real column of the right type.

Injection-safe planner; one plan feeds list and count

The planner emits SQL from only three sources: registry &'static str literals (table/column names), planner-generated $N placeholders, and fixed keyword/operator literals. User-supplied values reach only Bind parameters — never string-interpolated. There is no free-form SQL variant. The base case is WHERE TRUE with ` AND (<clause>)` appended (never a bare WHERE), and a single SearchPlan ({ table, where_sql, binds, order_by, order_dir }) feeds both the list query and the count query, so the two can no longer drift (the reports list/count hand-duplication is eliminated by construction). Absent optional filters contribute a bound ($N IS NULL OR <clause>) guard so the bind count is stable regardless of which filters are present.

Two disjoint channels — user filters vs. policy, no silent drop

User filters and authorization/policy constraints are different categories and never mix:

  • User filters never silently drop. For the static v1, the capability function makes an unsupported search structurally impossible at build time. For the deferred runtime-override case, an active filter whose runtime scheme cannot support the operator is a hard Err(SearchError::UnsupportedFilter) (fail-closed) — never skipped, never widened.

  • Policy constraints are separate and non-degradable. ScopeConstraints (worker/supervisor row scoping, mapped from craig_authz::ListScope) compile to unconditional equality on their own columns with a concrete bind — no NULL-guard, so they cannot no-op and cannot be dropped. A policy constraint that cannot be enforced is a hard Err, never a widened result set. The mapping is entity-aware: a supervisor scope narrows on an entity that has a supervisor column (cases) but yields an empty page on one that does not (investigations — preserving api/investigations/crud.rs:139-143).

User filters are the existing typed query DTOs (ReportsQuery/ReferralQuery/CaseQuery/ PersonSearchQuery), not a string-keyed map, so missing/duplicate/unknown/type-mismatched keys are unrepresentable. A hostile ?worker=other cannot widen scope because the user worker filter and the policy scope column live in different types and different channels.

Blind-index domain separation

craig-crypto gains pub const BLIND_INDEX_SCHEME_V1: u32 = 1; and pub fn hmac_domain(&self, field_domain: &str, version: u32, canonical: &[u8]) → Result<String, CryptoError>, deriving a per-field HKDF domain (info = b"craig-crypto/blind-index/" || field_domain || b"/v" || version). It takes bytes (so a Date blind index can canonicalize to "%Y-%m-%d"). This replaces the single global BLIND_INDEX_INFO_V1 domain for new blind indexes and removes cross-column HMAC correlation. SSN last-four migrates to field_domain="ssn", version=1; because the domain is part of the derived key, a scheme change is a migration + reseed, not a runtime toggle — flipping a field’s scheme without re-deriving its index would silently produce lookups that never match.

Crate decision — build on CRAIG’s own crypto

No external searchable-encryption crate is adopted (see Alternatives rejected). v1 uses AES-256-GCM-SIV at rest (ADR-020) plus a domain-separated deterministic HMAC blind index for opt-in equality. The FieldScheme enum is extension-shaped so a future order-revealing or structured-encryption scheme is an additive variant, not a rewrite.

Threat model

  • A deterministic blind index leaks equality and frequency. Two rows with the same plaintext produce the same HMAC, and the ciphertext-frequency histogram is visible to anyone who can read the column. This is acceptable only for high-entropy opt-in fields. It is not acceptable for low-entropy identifiers: ssn_last_four has only 10⁴ possible values, so its blind index is vulnerable to a trivial offline dictionary over the (public) HMAC construction the moment the per-field key leaks — the index buys equality search, not confidentiality, and the confidentiality still rests on the key (ADR-048) and the AES-256-GCM-SIV ciphertext, not the index. Names stay plaintext (they must, for trigram substring search — ADR-019 / ADR-041 treat the persons index as operationally non-confidential); blind index is therefore an explicit per-field opt-in, never a default.

  • Domain separation limits correlation. Per-field HKDF domains prevent an attacker from correlating the same plaintext across different columns (e.g. an SSN-four appearing in two places) by comparing HMACs — each column’s index is under a distinct derived key.

  • Passive-at-rest, fail-closed. The at-rest threat model is unchanged from ADR-048 (a read-only attacker with database access, not an active attacker with the running key); the query layer stays fail-closed per ADR-020 (a missing key under Required errors rather than emitting plaintext, and UnsupportedFilter/UnenforceablePolicy error rather than widening).

  • No search oracle. UnsupportedFilter maps to a generic 400 with no field-specific detail, so the error cannot be used to probe which fields are encrypted.

Consequences

  • Reports search semantics change (pre-1.0, breaking). Report search moves from substring-over-narrative/reporter-names (which never worked under encryption) to the plaintext admin_unit + reporter_type columns. The encrypted columns become Opaque and are structurally unsearchable; the contract/OpenAPI are corrected to advertise only real capabilities. Recorded in CHANGELOG.adoc (Changed).

  • One registry, many consumers. The FieldId/FieldSpec registry lives in a new crates/craig-cases-fields leaf (deps: craig-search core + craig-crypto; no axum/sqlx) consumed by craig-cases, tools/craig-seed, and xtask — so the write path, decrypt, seeder, and verify-seed all derive their covered-column set from the registry rather than hand-maintained lists (for the entities each seeds — persons + referrals; SeedData has no reports today).

  • New leaf crate, no layering violation. craig-search (generic framework) depends only on craig-crypto, uuid, chrono, serde, thiserror, and sqlx (optional, for the execute_search glue only). It does not depend on craig-common (which would drag axum + sqlx into the search layer — craig-common/Cargo.toml:11,31), so it carries its own SearchError and its own SearchMode. From<SearchError> for ApiError lives in craig-common behind an optional search feature — the orphan-rule home, mirroring the craig-mq-error / From<StageError> precedent (ADR-047; craig-common/src/error.rs:493). No cycle: craig-search never reaches back to craig-common. CI builds both craig-search configs (core DB-free + --features sqlx).

  • SSN blind-index reseed. Migrating SSN to its own domain invalidates existing SSN indexes; the migration is co-located (service write + service query + seeder HMAC) in one MR so any post-migration reseed is self-consistent (devstack reseeds routinely; no production data pre-1.0; C7’s full-volume reseed picks it up).

  • DOB deferred, additively. DOB is modeled Plaintext2026-07-13 in C3 (behavior-preserving); BlindIndexSpec is Date-ready, so ADR-041’s DOB hardening (encrypt DOB + dob_hmac
    birth_year) is a later additive registry flip, not a redesign.

  • Config-ready, override deferred. Per-jurisdiction scheme override (BundleContribution.search_schemes + fail-closed resolve() that refuses an Opaque→Plaintext downgrade without a migration marker) is supported additively by the registry + fail-safe defaults but is not wired in v1 (filed as a follow-on). v1 honesty is corrected doc-comments, not a /search-capabilities endpoint (also deferred).

  • (#1025/#1398, 2026-08-11 — override deferral lifted.) The per-jurisdiction override is wired end-to-end and ACTIVE — allowlist-narrow (exactly persons.ssn_last_four → {BlindIndex, Opaque}), gate-stamped (crypto_field_lineage), fail-closed at every boundary, atomically activated. As-built: == Amendment — #1025 below.

  • (#1026, 2026-08-08; intersection #1400, 2026-08-12.) The machine-readable surface: GET /v1/cases/_meta/search-capabilities returns, per entity, each registered field’s ADVERTISED {substring, equality, ordering} — since #1400 the INTERSECTION of effective-scheme capability with the entity descriptor’s declared operations (advertised_capabilities: predicates + the sort whitelist; Concat/AnyOf all-constituent atomic like enforcement; relation locals contribute nothing; undeclared fields stay as all-false rows). Raw capability() alone over-advertised — a cryptographically-capable field with no wired predicate (full ssn, the confirmation-oracle exclusion) still claimed equality. Both inputs are the artifacts the planner enforces with, so the advertisement cannot drift from enforcement in either direction. Any authenticated role (the response is a pure function of the public registry + descriptor source; key-loaded state is not reflected).

Alternatives rejected

  • External searchable-encryption crates. cosmian_findex (Searchable Symmetric Encryption) is BUSL-1.1 — incompatible with CRAIG’s AGPL-3.0. ore-rs (order-revealing encryption) ships under a non-standard license. sifredb is an immature v0.1.x. None clears the license + fit
    maturity bar; a domain-separated HMAC blind index on CRAIG’s own vetted primitives delivers the one operation actually needed (opt-in equality) with no new dependency.

  • Fully homomorphic encryption (tfhe). FHE computes over ciphertext but does not provide the indexed equality lookup report/SSN search needs; it is the wrong tool (orders of magnitude slower, no index acceleration) for filtering a relational table.

  • Per-field ad hoc code (the status quo). Three hand-made instances already diverged and one broke silently; adding a fourth by hand repeats the defect and does not scale to multi-jurisdiction, where the encrypted set varies per deployment.

  • String-keyed search request. A HashMap<String, String> filter bag reintroduces missing/duplicate/unknown/type-mismatch keys and — critically — lets a user filter key collide with a policy scope key, weakening authorization. The typed-DTO + separate ScopeConstraints design makes that class of bug unrepresentable.

  • Fail-open degradation (skip an unsupported filter). Silently dropping a filter the storage can’t satisfy widens the result set — at the authorization boundary that is a confidentiality defect. C3 fails closed: an unsatisfiable user filter errors and an unenforceable policy constraint errors, neither widens.

Amendment — #1159 pagination depth & plan cache (2026-07-27)

LIMIT/OFFSET are bound, not inlined. The executor (craig-search/src/exec.rs) renders LIMIT $N+1 OFFSET $N+2 and appends the two binds after the plan’s binds on the LIST builder only — the count query carries the plan’s binds verbatim, so the one-plan-feeds-list-and-count shape and the WHERE clause’s stable bind count are untouched. Rationale: inlined literals made every distinct (limit, offset) pair a distinct SQL text, and sqlx’s per-connection prepared-statement cache is a 100-entry LRU (never overridden by craig-db) — a page walk evicted the working set (audit F22).

Plan-cache note (the optional-predicate pattern). The audit flagged a nuance worth recording: inlined pagination literals partially OFFSET the generic-plan risk of the ($N IS NULL OR <clause>) guards, because each distinct text gets a fresh custom plan that constant-folds the NULL arms. Binding was therefore assessed together with plan quality, not assumed: EXPLAIN (GENERIC_PLAN) on PG 18.4 (devstack, 2026-07-27) confirms the guard pattern with bound LIMIT $2 OFFSET $3 produces the expected plan shape — captured for the persons name-search arm:

Limit
  -> Sort (Sort Key: last_name)
    -> Seq Scan on persons
       Filter: (($1 IS NULL) OR (((first_name || ' ') || last_name)
                ~~* like_escape((('%' || $1) || '%'), '\')))

(the $N IS NULL OR … filter evaluates per-row; index/scan choice unchanged from the custom plan on the hot arms). If a future jurisdiction’s data shape makes a generic plan regress, the escape hatch is plan_cache_mode=force_custom_plan per-role — not re-inlining pagination.

Pagination depth is capped; per_page clamps. craig_common::PageRequest rejects page > MAX_PAGE (10,000) with a typed 400 naming the cap and field (validate_depth). Depth is rejected rather than clamped because a clamped page would silently serve a different window than the echo claims; per_page > 500 remains a silent clamp (the fleet’s tested posture — a clamped size still serves the same window prefix). Worst-case skip drops from ~2.1 × 10¹² rows to 5 × 10⁶. Since #1170 this is the ONE fleet contract: the per-service clamp clones (security/financial/exchange/ placement/rules/reporting) are retired and every paginated list endpoint routes through PageRequest — depth cap + the #822 count-only per_page=0 included (two recorded exceptions: the subsidy-import records list keeps its stricter garde 400s, and security’s workers.rs limit-only shape has no OFFSET to cap).

Keyset posture: DEFERRED with a named trigger. OFFSET pagination stays until a deployment shows either (a) a hot list table (audit_log is the fleet’s hottest unbounded surface) sustaining > ~1,000,000 rows inside its retention hot window, or (b) list p95 > 500 ms attributable to deep-page skips at the capped depth. When triggered: UUIDv7 primary keys make WHERE id < $cursor ORDER BY id DESC viable fleet-wide, and the internal keyset machinery in craig-financial (review_sweep.rs KeysetRow
generator.rs child_id cursors, import finalize chunking) is the in-tree precedent to generalize. Keyset changes the wire contract (opaque cursor instead of page) — a pre-1.0 breaking change if adopted, which is why it is not done speculatively.

Amendment — #1158 read projections (2026-07-27)

The executor gained a validated read projection. execute_search(pool, plan, projection: Option<&Projection>, limit, offset) — None renders the legacy SELECT ; Some renders the projection’s columns in declared order, so heavy/encrypted columns a list never renders are *never fetched at all. craig_search::Projection wraps &'static [&'static FieldSpec] and validates shape ONCE at construction (Projection::new refuses empty and duplicate column sets — two new SearchError variants, 500-class); rendering (SearchPlan::list_sql) is infallible afterward. Column provenance remains registry-literals BY CONVENTION (FieldSpec is publicly constructible) — each instance pins provenance with a subset-of-ALL registry test.

First instance — the reports summary list. craig_cases_fields::reports::SUMMARY_COLUMNS (five plaintext columns
narrative) backs craig-cases’ `api::reports::summary seam: ONE LazyLock static drives BOTH the SELECT column list and the decrypt walk, so an unfetched column structurally cannot be decrypted or serialized, and the list performs exactly one decrypt per row (the registry test pins the projection’s encrypted subset to exactly ["narrative"]). The row type (ReportSummaryRow) is deliberately not Serialize; the only exit is a consuming decrypt→truncate→DTO pass, making a summarize-before-decrypt ciphertext leak unrepresentable without new code at the seam. The wire shape change (6-key summary; prefix-160/wire-161 preview) and its stop-the-world deploy procedure are recorded in CHANGELOG.adoc § Unreleased.

Scope decision. The four other ADR-049 list endpoints (persons, cases, referrals, investigations) pass None — their row models are lean enough that projection is not yet worth a per-entity wire fork; each can adopt a projection independently by the same recipe when its shape warrants it.

Cross-references: ADR-020 (fail-closed
CGEF), ADR-048 (key lifecycle), ADR-041 (DOB hardening follow-up
plaintext-name rationale), ADR-019 (trigram needs plaintext names), ADR-025 (worker identity is a UUID — the scope channel binds worker/supervisor UUIDs), ADR-032 (multi-jurisdiction motivation).

Amendment — #1025 per-jurisdiction scheme override (2026-08-11)

The Consequences bullet’s deferred override wiring landed as the six-unit epic &80 program (#1392–#1398; plan archived at plans/archive/scheme-override.adoc). As built:

The override is a property of the stored data, allowlist-narrow. A field’s at-rest scheme is baked into the data (the blind-index HKDF domain lives in the derived subkey), so overrides are constrained to a hard-coded allowlist of provably-safe transitions — v1 is exactly ONE field: persons.ssn_last_four → {BlindIndex, Opaque} (at-rest bytes identical; only the ssn_hmac sibling and equality-searchability toggle). Extending the allowlist is code plus a migration path (#1399), never configuration. →Plaintext is ineligible everywhere.

Single-writer scheme truth. A bundle declares intent via the defaulted StateBundle::search_schemes() (the tx-stub bundle carries the reference Opaque declaration); the pure craig_cases_fields::resolve leaf resolves declarations against the allowlist fail-closed (unknown/duplicate/not-allowlisted/bad-transition all refuse). The ADR-063 migrate gate is the ONLY writer of scheme truth: it reconciles the resolved set into crypto_field_lineage (insert/equal-only over the allowlist’s exact set — a scheme change over recorded data refuses the deploy; the runtime role holds SELECT-only). Serving boot verifies the marker read-only, presence-required, then materializes the resolved registry ONCE (EffectiveRegistries; the no-override path short-circuits to the compiled consts).

Every boundary consults the resolved registry. The planner takes a table-bound EffectiveRegistry — its effective-spec lookup is presence-independent (a miss is an invariant 500, never a silent fallback) while the capability re-check is presence-gated (a present filter on an overridden field is the generic 400, no field oracle; absent filters keep NULL-guard no-ops and stable bind counts; Concat/AnyOf are all-constituent atomic). The encrypt/decrypt wrappers, the inbox consumer’s decrypt + SSN-promotion legs, the /search-capabilities endpoint (which therefore honestly narrows under an override), craig-seed (bundle-side resolution — a DB-less generator cannot read markers), and xtask verify-seed (marker-row reconstruction; a bundle↔marker divergence fails the sibling cross-check) all read the same resolved truth. Sort/summary/scope surfaces are pinned structurally disjoint from the allowlist by tests instead of runtime checks.

Lifecycle binding. The seed identity gained a registry_fingerprint component (MARKER_PROTOCOL_VERSION 1→2) and the devstack guard binds CRAIG__ACTIVE_STATE_BUNDLES as a staleness marker — a host-only bundle flip escalates to the volume wipe + reseed exactly like a key change. Mid-life scheme migration over live data remains refused until the #1399 tooling exists.

Edit this page · latest