ADR-035: 5-Layer Composition Engine for Dashboards + Case-Detail; NEW craig-composition Backend Service

On this page

Status

Accepted 2026-06-17. Anchors Plan S — Multi-Jurisdiction Foundation Step 19; the implementation lands in Plan X (Composition Layer Engine). This is the third of the five design-team engineering contracts to get its own ADR (after ADR-033 / ADR-034 / ADR-036). It realizes Contract 3 (Composition + persistence) and refines that contract’s service home: composition resolution, the override store, and the cross-replica invalidation publisher live in a NEW backend service services/craig-composition, NOT in the craig-web BFF. Contract 3’s five-layer model, the rulesets/<jurisdiction>/ baseline home, the merge semantics, the required=true + 12-column row grammar, role-filter-after-merge, and content-hash versioning are all PRESERVED. Builds on ADR-032 §4 + A11 (additive BundleContribution growth — compositions: CompositionContribution), ADR-038 §3 (registry materialization), ADR-028 (the X-Craig-Actor on-behalf-of auth seam the BFF uses to reach the new service), and ADR-024 (the existing zen-engine RabbitMQ cache-invalidation precedent this engine mirrors). The external precedent is canopy’s crates/canopy-composition/ (ported, not consumed) and its ADR-021 (invalidate-on-write composition cache).

Context

The design team’s "everything is config" goal for dashboards and case-detail layouts means a jurisdiction operator — and ultimately an end user — must reshape a surface without a code change. Contract 3 specifies the answer: every composable surface (each role’s dashboard, the case-detail workspace) resolves top-down through five layers, from most-specific to least-specific, and the resolved tree is hashable so downstream caches can validate by version.

The canopy project already runs this model in production (crates/canopy-composition/): a CompositionLoader walks the layers, merges JSON, role-filters after merge, and hashes the result. CRAIG ports that design rather than consuming the crate (canopy is a separate codebase; CRAIG re-implements against its own BundleContribution, Claims, craig-mq, and craig-db primitives).

The open architectural question this ADR settles is where the engine runs. The other two bundle-overlay contracts that have shipped — terminology (ADR-034) and theme (ADR-036) — resolve inside the craig-web BFF because they are read-only, compile-time bundle assets. Composition is different in kind: it has a mutable persistence dimension (live overrides + per-user deltas in a database), a multi-replica cache-coherence requirement, and a write API (Studio admin edits, user drag-to-reorder). Pushing a database, a migration set, a RabbitMQ publisher/subscriber, and an authz-bearing write surface into the BFF would dissolve the BFF’s defining property — craig-web has no sqlx, craig-db, or craig-mq dependency today and is deliberately a stateless aggregator (ADR-004). That is the reasoning behind the service-home decision below.

Decision

1. Composition is a NEW backend service, not the BFF

A new first-class service services/craig-composition owns composition resolution and persistence. It mirrors the existing 8001–8008 services exactly:

  • Port 8009 — the next unclaimed port after craig-intake (8008); craig-web stays on 8080.

  • Per-service database craig_composition — added to devstack/postgres/init.sql alongside the other seven service databases; migrations live at services/craig-composition/migrations/*.sql and run on boot via br.db.run_migrations(&sqlx::migrate!()).

  • Container image — a new craig-composition target stage in the repository’s single multi-stage root Dockerfile (CRAIG does not use per-service Dockerfiles; this ADR corrects the Step 19 sketch on that point) plus a service block in devstack/docker-compose.yml (build.target: craig-composition, depends_on postgres + rabbitmq + keycloak, a /readyz healthcheck, and CRAIG_COMPOSITION__* environment).

  • Dependency set — craig-api, craig-auth, craig-authz, craig-bootstrap, craig-common, craig-db, craig-mq, axum, sqlx, plus the composition crate (below). The main.rs follows the canonical orchestrator shape (bootstrap → ServiceDeps::build → spawn_workers → build_router → ApiServer::serve).

The portable engine itself lives in a NEW library crate crates/craig-composition (the port of canopy’s composition crate), so the merge/role-filter/hash logic is unit-testable without a running service and could later be reused by another host.

craig-web becomes an HTTP client of the new service: it asks craig-composition for "the resolved tree for this surface, this jurisdiction, this role, this user" and renders the answer. No new DB or MQ dependency enters the BFF.

2. The five-layer composition stack

A surface resolves by overlaying five layers, lowest precedence first. Higher layers win:

Layer (low → high) Source of truth Who edits

5. Product default

Compiled into crates/craig-composition (jurisdiction-neutral; the shape CRAIG ships)

Core engineering, via PR

4. Jurisdiction baseline

Git-managed TOML in rulesets/<jurisdiction>/ (dashboards.toml, case_detail.toml), loaded at runtime — the same rulesets/<jurisdiction>/ home CRAIG already uses for JDM authz rulesets

Ops, via normal git PRs. No in-UI baseline editing.

3. Jurisdiction live override

composition_overrides rows scoped to (jurisdiction, surface)

Jurisdiction admin via Studio (takes effect without a deploy)

2. Role override

composition_overrides rows scoped to (jurisdiction, role, surface)

Jurisdiction admin via Studio

1. User delta

composition_overrides rows scoped to (jurisdiction, role, user_sub, surface)

End user, via direct UI interaction (reorder / pin / hide)

The split is deliberate: baselines are ops territory edited by file + PR (deploy-time), while live overrides and user deltas are runtime data edited through Studio/UI and stored in the database (effective immediately). This matches CRAIG’s existing posture where rulesets/<jurisdiction>/ holds git-managed jurisdiction config loaded at runtime.

3. Merge semantics

Following the canopy precedent, two RFC-standard merge strategies apply at different layer boundaries:

  • Product default → jurisdiction baseline: RFC 7396 (JSON Merge Patch). A structural overlay — keys present in the baseline replace the default, omitted keys fall through, null removes a key. Total (cannot fail). Arrays are replaced wholesale.

  • Baseline → DB override layers (live, role, user): RFC 6902 (JSON Patch). Ordered op-lists (add/remove/replace/move/copy/test) keyed by RFC 6901 JSON Pointer. This carries the small, precise mutations a Studio edit or a CLI tweak produces.

The merge runs on serde_json::Value and the result is deserialized into the typed composition model afterward (the merge is schema-free; validation happens against the typed shape).

User deltas for dashboard surfaces use a versioned semantic envelope, not raw RFC 6902. Index-based JSON Patch is fragile when the baseline grows a panel underneath a stored user delta. CRAIG adopts canopy’s user_delta_v1 shape — a tagged enum (#[serde(tag = "type", rename_all = "snake_case")]) carrying hidden_slugs, per-slug span_overrides, and a slug_order — anchored to plugin slugs, not array indices. Unknown slugs are silently ignored on apply (forward-compatible with baseline additions). Case-detail and other non-dashboard surfaces continue to use RFC 6902 ops. The envelope is versioned so a user_delta_v2 can land additively.

User deltas are validated before persistence: every referenced slug must exist in the post-role-filter baseline; the slug’s plugin must be permitted for the writing role (defense-in-depth re-check); each span must be in the panel’s allowed_spans; and a dry-run apply must keep every row within its 12-column budget. required=true panels cannot be hidden or removed by any override or delta.

4. Role filtering applies AFTER merge

Once the five layers are merged into a resolved tree, each item is checked against the requesting worker’s role using the item’s plugin manifest required_roles (ADR-033). Items the role may not see are silently dropped — never rendered as "permission denied" tiles. Filtering after merge (rather than per-layer) means a jurisdiction can add a panel in its baseline and have it disappear cleanly for roles that lack the plugin’s permission, with no special-casing per layer.

5. Persistence: the composition_overrides table

A single table on the craig_composition database holds layers 1–3 (live, role, user), distinguished by which scope columns are non-null:

CREATE TABLE composition_overrides (
    id               UUID PRIMARY KEY DEFAULT uuidv7(),
    jurisdiction_code TEXT        NOT NULL,
    role             TEXT,            -- NULL ⇒ jurisdiction-live layer
    user_sub         UUID,            -- non-NULL ⇒ user-delta layer
    surface_key      TEXT        NOT NULL,
    delta            JSONB       NOT NULL,  -- RFC 6902 op-list OR user_delta_v1 envelope
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT now()
    -- UNIQUE (jurisdiction_code, role, user_sub, surface_key)
);

This is the shape Contract 3 specifies. It collapses canopy’s (layer enum, scope_key) pair into explicit role / user_sub columns, which CRAIG prefers for query clarity. Optimistic concurrency on writes uses an updated_at-derived ETag (RFC 7232) with a monotonic update guard, per the canopy precedent; the exact write-API surface (PUT/PATCH/DELETE semantics, archive-on-delete) is a Plan X detail.

6. Cache: invalidate-on-write, RFC 8785 content hash, RabbitMQ fanout

Each replica of craig-composition keeps an in-process cache keyed by (jurisdiction, role, user_sub, surface). The resolved tree is canonically serialized per RFC 8785 (JSON Canonicalization Scheme) and SHA-256 hashed into a stable version number. CRAIG implements true RFC 8785 canonicalization from the start: where canopy’s loader currently serializes in insertion order and flags RFC 8785 as a future hardening, CRAIG adopts it now, so the version is reproducible across replicas and process restarts. The BFF carries that version downstream for its own cache validation.

Writes invalidate on write, not on a timer:

  • The writing replica drops the affected cache entries immediately.

  • It publishes a composition.invalidated event on the existing craig.events topic exchange (ADR-003). Every replica subscribes via subscribe_exclusive on a per-instance UUID-named auto-delete queue (the established cross-replica cache-invalidation pattern from ADR-024's zen-engine reload), so all replicas drop the stale entry on the next render. Self-emitted events are skipped by instance-id.

  • A bounded TTL (10 minutes) is the safety net for the rare case of event-delivery failure, so a dropped invalidation self-heals rather than serving a stale layout indefinitely.

Publication is transactional: the composition.invalidated envelope is staged in the same DB transaction as the override write (the transactional-outbox pattern, ADR-022) and drained by the outbox worker.

7. Auth pass-through (ADR-028) and the BFF-as-client contract

The BFF holds the worker’s OIDC session; the composition service needs the worker’s role (to role-filter) and jurisdiction (to pick the baseline + scope overrides). Both ride the existing ADR-028 on-behalf-of seam — no new IdP or Keycloak dependency on craig-composition:

  • craig-web authenticates to craig-composition as itself with its client_credentials service token (the azp claim identifies the BFF).

  • It attaches an X-Craig-Actor JWT minted by ActorTokenIssuer (signed by craig-signing, aud: craig-internal-actor) identifying the acting worker.

  • craig-composition validates both JWKS per request (craig-identity for the bearer, craig-signing for the actor) and reads claims.actor for the worker sub + roles; audit records both actor_service and actor_user_sub.

This is exactly the flip craig-web already performs for its other backend calls (Plan E Phase B); reaching craig-composition adds one more callee, not a new auth mechanism. The user-delta write endpoint and the Studio live-override endpoints authorize against the actor’s role using the existing ADR-023 zen-engine stack hosted in the composition service.

8. The compositions: CompositionContribution bundle field

Per ADR-032 §4 + A11, the BundleContribution aggregate grows one field — compositions: CompositionContribution — added additively in the Plan X MR that introduces its supporting type. The field is pre-materialized (a parsed, I/O-free declaration of the composable surfaces the bundle provides), which resolves the materialization shape left open in ADR-038 A11 ("factory shape if compositions take per-state operator context at boot; pre-materialized otherwise") toward pre-materialized, consistent with the terminology and theme fields.

The field does not carry the jurisdiction baseline trees. Of the five layers, only two are statically known ahead of a request: the product default is compiled into crates/craig-composition (jurisdiction-neutral; canopy’s system_defaults() precedent), and the bundle’s compositions declaration names the surfaces in play for the active jurisdiction. The jurisdiction baseline (layer 4) is read at runtime by craig-composition from rulesets/<jurisdiction>/ — not embedded in the bundle — mirroring how CRAIG already runtime-loads JDM authz rulesets from that same rulesets/<jurisdiction>/ home, and how canopy’s loader reads its baselines. The DB layers (live, role, user) are likewise runtime data. The boot-time resource that needs operator context (the CompositionLoader, with its DB pool, MQ connection, and rulesets/ reader) is a service-level construct inside craig-composition, not the bundle field. A bundle that ships no composition (e.g. the craig-state-tx-stub reference) sets compositions: CompositionContribution::default(). The exact payload of CompositionContribution (a surface registry, optionally the compiled product-default trees) is a Plan X detail; the invariant fixed here is that jurisdiction baselines are runtime-loaded from rulesets/, never bundle-embedded.

Consequences

Positive

  • The BFF keeps its defining property — no database, no message broker, no migrations. Composition’s stateful concerns are quarantined in a service built for them.

  • The portable crates/craig-composition engine is unit-testable in isolation (merge, role-filter, hash, delta validation) with no service or DB.

  • Cross-replica cache coherence reuses a proven CRAIG pattern (ADR-024 RMQ invalidation) rather than inventing one; the RFC 8785 version hash makes downstream caching safe.

  • The five-layer model gives ops, jurisdiction admins, and end users each a non-overlapping editing surface, with required=true and the 12-column row budget as hard floors no layer can violate.

  • Auth is free of new dependencies — the ADR-028 actor seam already exists and is exercised by the BFF.

Negative

  • A ninth backend service is operational surface area: another image, database, migration set, healthcheck, and devstack entry to run and monitor.

  • Two merge algorithms (RFC 7396 + RFC 6902) plus a bespoke user-delta envelope is more machinery than a single strategy; contributors must learn which applies where.

  • The BFF gains a synchronous dependency on craig-composition for every composed page render (mitigated by the BFF’s own version-keyed cache + the service’s in-process cache).

Mitigations

  • The new service is a near-clone of the existing 8001–8008 services; the orchestrator, bootstrap, auth, and outbox plumbing are all shared crates, so the incremental code is the composition logic, not new infrastructure.

  • The 10-minute TTL bounds any cache-coherence failure; the invalidation event is transactional-outbox-staged so it cannot be lost relative to the write.

  • Plan X ships the reference baselines (rulesets/georgia/dashboards.toml, rulesets/georgia/case_detail.toml) and cross-replica invalidation tests as acceptance gates.

Open questions

  • Write-API shape. PUT-with-If-Match vs PATCH-append vs a higher-level Studio verb set, and whether deletes archive (canopy keeps a *_archive table). Deferred to Plan X.

  • Baseline reload granularity. Whether rulesets/<jurisdiction>/ baselines are read once at boot or hot-reloadable (JDM rulesets are reloadable via RMQ today); Plan X decides whether composition baselines join that reload path.

  • CompositionContribution payload. §8 fixes the invariant (baselines are runtime-loaded from rulesets/, never bundle-embedded; product default is compiled); what remains for Plan X is the exact field payload (a surface registry, and whether the compiled product-default trees ride on it or sit beside it in crates/craig-composition). Plan X Step 1 settles this when it ports the canopy crate and verifies the canopy source path still exists at canopy HEAD.

Alternatives considered

  • Resolve in the BFF (rejected). Putting the loader, the composition_overrides table, the RMQ publisher, and the write API in craig-web would force sqlx + craig-db + craig-mq into a service that is deliberately stateless (ADR-004). It also couples composition’s release cadence to the BFF’s. Rejected for the scope-creep reason in Context.

  • Resolve in craig-cases / an existing service (rejected). Composition spans dashboards (no single owning domain) and case-detail; no existing service is the natural home, and overloading one would blur its bounded context.

  • A pure config-file model with no DB (rejected). Drops the live-override and per-user-delta layers Contract 3 requires (immediate, no-deploy edits and personalization). The database layers are load-bearing.

  • Consume the canopy crate directly (rejected). canopy is a separate codebase with its own Claims, DB, and MQ assumptions; CRAIG ports the design against its own primitives (the same posture ADR-032 takes toward canopy).

  • Per-layer role filtering (rejected). Filtering at each layer rather than once after merge multiplies the checks and makes baseline-added panels harder to drop cleanly; after-merge filtering is canopy’s proven approach.

Out of scope

  • The Studio admin UI and the user drag-to-reorder UI (Plan X UI steps).

  • Field-level edit ownership and propose/approve queues — that is ADR-037 (anticipated) / Plan Y, a distinct concern from layout composition (cited in plain text until that ADR lands, per ADR-032’s convention).

  • The plugin manifest + render contract that supplies the panels being composed — ADR-033 / Plan W (already shipped).

Related decisions

  • ADR-032 — the BundleContribution aggregate this ADR’s compositions field extends (§4 additive growth; A11 per-field shape).

  • ADR-038 — §3 registry materialization (this ADR resolves the A11 open question toward pre-materialized).

  • ADR-033 — supplies the panels/sections being composed and the required_roles the role filter reads.

  • ADR-028 — the X-Craig-Actor on-behalf-of seam the BFF uses to reach craig-composition.

  • ADR-024 — the zen-engine RabbitMQ cache-invalidation pattern this engine reuses for composition.invalidated.

  • ADR-022 — transactional outbox for staging the invalidation event.

  • ADR-023 — the authz stack the write endpoints enforce against.

  • ADR-004 — the stateless-BFF property this decision protects.

  • Contract 3: Composition + persistence — the design-team contract this ADR realizes and whose service home it refines.

  • canopy crates/canopy-composition/ + canopy ADR-021 (external; ported, not consumed) — the working precedent for the loader, merge strategies, user-delta envelope, role filter, and invalidate-on-write cache.

Edit this page · latest