Known Issues & Lessons Learned

On this page

Project-specific gotchas live here; high-level CRAIG patterns (problem-type URL catalogue, RFC 9457 discipline, authz cousin-handler pattern, etc.) live in the Antora docs under docs/modules/ROOT/pages/. User-specific behavioral preferences live in user-local agent auto-memory. Read all three when investigating an unfamiliar failure mode.

Known Issues

  • Lost exec bit on .githooks/* (lesson 2026-06-07): Git stores the executable bit in the tree (100755 vs 100644). Cloning on Windows without core.fileMode set, an over-eager chmod -x .githooks/, or a clean re-add after chmod 644 can land the hooks at 100644. With core.hooksPath .githooks still set, Git silently no-ops the hook (treats it as non-executable text) and unvalidated code reaches the remote. CI job hooks-exec-check in .gitlab-ci.yml is the regression gate — runs on every MR touching .githooks/* AND every main-branch push, fails with the offending path + actual mode. Fix: chmod +x .githooks/* && git add --chmod=+x .githooks/* && git commit.

  • RabbitMQ user: Must be in definitions.json with SHA-256 hash (not env vars).

  • Keycloak token lifetime: 30 min. Long test sessions may hit 401s.

  • E2E seed-manifest alignment: tests/e2e/lib/seed.ts is committed and gets regenerated on cargo xtask e2e from the current CRAIG_SEED + CRAIG_FAMILIES env vars (defaults: 42 / 12). The committed file must match those defaults or fresh cargo xtask dev start seeds the DB with one shape while specs import IDs from another, and symptoms look like locator.selectOption: Timeout 10000ms exceeded. The generator itself is byte-deterministic for identical inputs — guarded by manifest_is_byte_deterministic_across_runs in tools/craig-seed/tests/integration.rs. If the committed manifest goes out of sync, regenerate with defaults: CRAIGACTIVE_STATE_BUNDLES=georgia cargo run -p craig-seed — --seed 42 --families 12 --output-dir tests/e2e/.seed-sql --manifest tests/e2e/lib/seed.ts (the seed jurisdiction comes from the active bundle since Plan U Step 10; CRAIGACTIVE_STATE_BUNDLES is required). E2E specs should avoid index-based lookups (casesList[0]); use the exported findCase/findHome/etc helpers with a predicate instead.

  • CI 2xlarge runner disk OOM is recurring: ld: final link failed: No space left on device during cargo nextest --workspace is upstream-runner infra, not code. The pre-push battery (cargo fmt --check, cargo clippy --workspace --all-targets — -D warnings, cargo nextest run --workspace, cargo xtask validate) is the authoritative quality gate and covers everything CI duplicates. The user has standing authorization to force-merge after pre-push passes — see Force-merge runbook below.

  • Keyed devstack: craig-cases refuses boot on key↔data mismatch (since epic &66 C7, #987): at startup cases verifies the mounted CRAIG_FIELD_ENCRYPTION_KEY against the seeded data’s crypto_key_lineage anchor (ADR-048 §D3) and refuses to start on a mismatch — compose then reports dependency craig-cases failed to start. The xtask paths self-heal (the .devstack/field-key.kcv staleness marker forces the wipe + keyed reseed automatically), but one edge does not: if .devstack/ markers are absent while an old data volume persists (fresh clone against an existing volume, or a manually cleared .devstack/), dev start takes the cold-start path — no staleness check runs — and cases refuses boot with KeylessLineageWithKey/KcvMismatch. This is the designed defense-in-depth, not a bug; the refusal message names the fix: cargo xtask dev reseed.

  • Inbox retries past strike 2 are operator-paced (epic &77 / ADR-062 §A, 2026-08-02): a failing MQ handler gets exactly TWO automatic attempts (broker 2-strike: nack-requeue, then reject → DLQ). The #1197 parking replayer serves the DLQ consumer’s own failures ONLY — nothing automatic returns a dead-lettered envelope to the source queue; attempts 3–5 come from the ADR-022 §D4 admin replay, and the counted-attempt budget (error_count, terminal at 5) spans those cycles. Symptom in tests/ops: an event_inbox row stuck at error_count=2, processed_at NULL is WAITING FOR AN OPERATOR, not retrying. Beyond-grace watchdogs + DLQ depth gauges are the signals; the operator escape hatch (delete the terminal row + replay) is in the ADR-062 runbook section. A fault-injection test whose fault does not self-clear before strike 2 can therefore never converge — see the cases degraded_pass_is_refused_counted_and_converges_on_the_retry leg for the self-disarming-trigger pattern that keeps such tests deterministic.

  • event_inbox.payload_hash legacy rows are NULL until #1273 (epic &77 A1): pre-program inbox rows carry no identity hash, so the collision check skips the hash leg for them (backfilled at next stamp). The post-horizon NOT NULL migration is #1273 — do not write code assuming the column is always populated before it lands.

  • request_claims replays are bounded by a 30-day horizon (epic &77 §B, ADR-062 §H): a client retry of a convert-class POST after CRAIGREQUEST_CLAIMSWINDOW_DAYS (default 30; 0 disables retention, never the guard) is a NEW operation by design — the claim row was pruned. Same boundary for §U upload/export attempts at the 31-day tombstone retention. This is the documented fresh-attempt boundary, not a bug.

  • RabbitMQ ≥4.2 metadata writes can transiently time out under I/O contention (Khepri) (#1401, forensics 2026-08-11): the broker’s Raft-backed Khepri metadata store makes a transient consensus-write timeout a DESIGNED failure mode — queue.declare/queue.bind answer with a connection-class INTERNAL-ERROR: … metadata store operation timed out (which closes the whole connection), typically minutes after a broker recreation (the volume-less broker cold-starts the store, including the Mnesia→Khepri table migration, on every recreate) under battery-tail disk contention. Posture: the image is pinned to an exact patch (devstack/rabbitmq/Dockerfile — the store engine can never change under an infra rebuild again); the test harness’s EventCollector retries the transient class boundedly with a fresh connection per attempt (craig-test-lib events.rs, fail-loud after 5); production subscriptions already self-heal via craig-mq’s supervised reconnect loop (its module doc: reconnect with capped backoff
    redeclare), so no per-declare retry was added there. Anything OTHER than the INTERNAL-ERROR class (ACCESS-REFUSED, PRECONDITION-FAILED, …) still fails immediately.

  • CI raw-compose devstack jobs need the CRAIG_CI_AGE_KEY project variable (ADR-064 U7, #1385; was CRAIG_FIELD_ENCRYPTION_KEY pre-switchover): the compose file env_file-requires devstack/field-key.env; the pentest/perf jobs materialize it via devstack/ci/write-field-key.sh, which decrypts the repo-committed store (secrets/dev.yaml) with the masked + protected CI age identity CRAIG_CI_AGE_KEY (minted via cargo xtask secrets init --for-ci; see #1031). Unset variable ⇒ the script hard-errors naming this fix (ADR-048 §D5: never a keyless CI devstack). Protected means a manual perf-stress run on an unprotected branch hard-errors by design — trigger it from main. These jobs remain allow_failure for the unrelated recurring runner-infra reds (disk OOM, above).

Workflow tool misuse: scout agents that edit files (2026-06-04 lesson)

When launching a Workflow with parallel subagents for scouting (categorize emissions, draft a per-crate playbook, etc.), the agents may interpret their task as "make the workspace fit the plan" and start editing files in parallel. During Plan M Tier 1d, a scout-only workflow commented out the Plan H Step 2 transitional allow blocks across multiple crates simultaneously, leaving the working tree broken across 13+ lib roots. The agents thought they were "probing what would fire" — but the edits were uncoordinated and conflicting.

Mitigation:

  1. Explicitly frame scout prompts as "read-only; no file edits". Repeat the constraint at the end of the prompt where it’s likely to be re-read.

  2. If parallel file mutation is genuinely needed, set isolation: 'worktree' so each agent gets its own checkout.

  3. For force-merge-serialized lift work (Plan M MRs ship one at a time through the force-merge runbook), Workflow adds overhead without parallelism — execute sequentially in the main loop. The single-agent inline loop is the right shape.

  4. Recovery: if a scout workflow has already edited files, TaskStop the workflow, then git checkout — . to revert all unwanted edits. Save only the intentional edits done outside the workflow.

Force-merge PUT /merge returns 405 while detailed_merge_status: preparing (2026-06-05 lesson)

This is distinct from the 405 Method Not Allowed documented in the Force-merge runbook below (which fires while detailed_merge_status: ci_must_pass). When opening a new MR via the GitLab API, the MR sits in detailed_merge_status: preparing for ~10-30s while CI registers; PUT /merge_requests/N/merge during this window also returns 405. Mitigation: poll detailed_merge_status until it leaves preparing before issuing PUT, OR retry the PUT after a 30s sleep — the same call succeeds once the state advances.

Additionally observed during Plan M force-merges: the PUT can return 405 even with cancel=200 immediately preceding, because the cancel hasn’t propagated. A re-check of MR state right after a 405 sometimes shows state: merged — the actual merge landed during the call’s race window. Always verify merge outcome via a follow-up MR fetch or git ls-remote, not the PUT exit code alone.

curl -sf swallows JSON-error bodies on 4xx/5xx. When debugging merge-call failures, switch to curl -X PUT without -sf and pipe through python3 -m json.tool to see the actual error body — the message field tells you whether the failure is 405 not_mergeable, 405 preparing, or a downstream conflict.

Force-merge runbook (CI-bypass for known-infra failures)

When CI fails on the 2xlarge disk-OOM pattern AND the local pre-push battery is green, force-merge via the three-step API sequence. Plain glab mr merge rejects client-side and raw PUT …​/merge returns 405 Method Not Allowed while detailed_merge_status is ci_must_pass. Clearing the running-pipeline gate AND the armed-auto-merge gate unblocks the endpoint.

PROJECT=gadhs%2Fapplication%2Fccwis%2Fcraig
IID=<merge-request-iid>

# 0. Arm auto-merge first so a pipeline ID is assigned to the MR
glab mr merge $IID --auto-merge --yes
sleep 25                                     # let the pipeline ID populate

# 1. Read pipeline ID from the MR
PIPELINE_ID=$(curl -sS -H "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.com/api/v4/projects/$PROJECT/merge_requests/$IID" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); p=d.get('head_pipeline') or {}; print(p.get('id',''))")

# 2. Cancel pipeline + cancel auto-merge + PUT merge (all in order)
curl -sS -X POST -H "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.com/api/v4/projects/$PROJECT/pipelines/$PIPELINE_ID/cancel"
curl -sS -X POST -H "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "https://gitlab.com/api/v4/projects/$PROJECT/merge_requests/$IID/cancel_merge_when_pipeline_succeeds"
sleep 2
curl -sS -X PUT -H "PRIVATE-TOKEN: $GITLAB_TOKEN" -H "Content-Type: application/json" \
  -d '{"should_remove_source_branch":true}' \
  "https://gitlab.com/api/v4/projects/$PROJECT/merge_requests/$IID/merge"

Verify with glab mr view $IID — state should flip to merged and merge_commit_sha populates.

Apply IMMEDIATELY after pre-push passes — don’t wait for CI to confirm what the local battery already validated. Waiting for the recurring 2xlarge disk-OOM costs 15–30 min per MR.

When NOT to apply: pre-push fails (real test failures, clippy lints, fmt drift). Fix the underlying issue rather than force-merge — the bypass exists for upstream-runner-infra problems on already-validated work.

GITLAB_TOKEN lives in /home/bitskrieg/code/craig/.env.local (not in global config); source it explicitly when running the runbook.

Don’t rely on merge_when_pipeline_succeeds. Because CI never passes on this project (the recurring 2xlarge disk-OOM above), arming auto-merge just delays the merge until a pipeline that will fail — a ~25-minute wait for nothing. Once pre-push is green, issue the merge directly: POST /merge_requests to open the MR, then PUT /merge_requests/N/merge immediately after. If you never arm auto-merge, the wait-and-cancel ceremony above is unnecessary — the endpoint is unblocked because there’s no armed-auto-merge gate to clear. The pre-push battery is the canonical correctness gate; CI only duplicates what it already validated.

Resuming a stale single-commit branch (2026-06-06 lesson)

When resuming feature work whose branch was authored weeks ago and never pushed (e.g., a paused per-partner crate sitting on a single commit off a stale parent), the rebase path works cleanly when the branch carries exactly one commit:

  1. git branch --contains <sha> confirms which branch holds the commit.

  2. git log <sha>^ -1 shows the parent — verify it’s reachable from current main (git merge-base main <sha> should return the parent or an earlier commit).

  3. git rebase --onto main <parent-sha> replays the single commit on top of current main.

  4. Conflicts will land in Cargo.toml (workspace members + [workspace.dependencies] — usually trivial; keep both sides) and CHANGELOG.adoc (Unreleased section conflict — place both entries chronologically, oldest below newest).

  5. After conflict resolution: cargo fmt --all + cargo clippy --workspace --all-targets — -D warnings + cargo nextest run --workspace.

  6. If the branch carries pre-Plan-M lint discipline (transitional #![allow(clippy::pedantic | clippy::cargo)] at lib root), the cargo xtask validate no-transitional-allows gate will reject the push. Apply the lift per Quality Gates & Enforcement (CRAIG) (§ Resuming pre-Plan-M crate authoring) as a follow-up commit on the same branch (never amend the original commit per the git-workflow standard).

For untracked working-tree work (no commit, just files sitting in crates/<name>/), branch off current main and git add the files directly — no rebase needed. Apply the same lint-discipline check before pushing. The no-transitional-allows gate will fire if the lib root retains the pre-Plan-M block.

Branch hygiene (CRAIG-specific cadence)

  • Wait for main before branching: when an MR is in flight, let auto-merge land, then git checkout main && git pull --ff-only origin main, then branch. Branching from local main while an MR is mid-merge produces CHANGELOG.adoc / Cargo.toml rebase conflicts that have to be untangled manually.

  • Branch cleanup after each merge (not at session end). After every merge, run:

    git branch --merged main | grep -v '^\*\| main$' | xargs -r git branch -d
git remote prune origin

    Keeps the local branch list and origin/refs aligned with what’s actually on the remote.

Don’t pre-source .ports.env before git push

The pre-push hook’s cargo xtask e2e stage refreshes the devstack mid-run, reassigning random host ports and rewriting .ports.env. If you have pre-sourced .ports.env into your shell, the stale ambient CRAIG_TEST__* variables now point at dead ports; later hook stages read those stale vars over the fresh file and fail with dead-port connection errors that masquerade as a security/perf-scan failure but are really a port race.

Push in a clean environment — verify with env | grep -c CRAIG_TEST__ returning 0 and let each xtask stage load .ports.env itself. (Sourcing .ports.env is still correct for a bare cargo nextest run --run-ignored all, which does not auto-load it.) Never --no-verify; if you must skip a stage, use its documented opt-out (SKIP_PERF=1 / SKIP_SECURITY=1 / SKIP_SDK=1) only after confirming that stage passes standalone.

cargo deny blocks all pushes repo-wide on a new RUSTSEC advisory

cargo xtask validate runs cargo deny check, so the moment a new RUSTSEC advisory lands against any dependency in the tree, every push starts failing — independent of what your branch changed. This is not a defect in your work.

Fix it in a dedicated fix(deps) MR, not inline in an unrelated branch: cargo update -p <crate> to the patched within-semver release on its own feature/ branch, land it, then git rebase origin/main your in-flight branch onto it (expect a CHANGELOG.adoc top-entry conflict — keep both entries). cargo deny check standalone confirms the fix before pushing. This recurs regularly (e.g. anyhow, ammonia).

check-docs treadmill: synced content live-tracks the template

Synced surfaces (.claude/rules/, docs/modules/standards/, .githooks/*, and the rest of the sync manifest) live-track the claude-quickstart template’s main, which advances faster than this repo re-syncs. A standards page or hook can therefore bump mid-session and re-block a push that was previously clean, via cargo xtask check-docs.

Fix: run cargo xtask check-docs --fix --yes (add --allow-exec when hooks changed) to adopt the latest synced content, and land it as a separate chore: commit (the hook/CI carve-out) rather than folding it into the substantive change.

Externally-blocked stakeholder-filed issues

CRAIG receives compliance and integration issues filed by agency stakeholders that look immediately actionable but actually depend on external teams or external-system coordination (e.g. SHINES → Gateway integration owned by the OGG GGUPM team, or SSA exchange agreements for IV-E data). These are not CRAIG dev-team code tasks: the implementation spans multiple teams/systems, so building the CRAIG half of an integration that can’t be exercised end-to-end burns cycles.

How to disposition: mark such issues externally-blocked and skip them from the immediate-action backlog until the external dependency clears — treat them like other externally-owned work rather than "ready to pick up next."

Signals that an issue is externally-blocked:

  • Body references to external coordination — "OGG CR", "Gateway", "external integration", an external CR URL, or a named legacy system (SHINES, SACWIS, IES, etc.) without a clear statement of which side CRAIG owns.

  • Author + shape — filed by a named agency stakeholder (not the bot) with empty or stub acceptance criteria.

Partial actionability via the mock server: where the CRAIG-side scope is well-defined and exercisable against tools/craig-mock-server/ (which emulates the SHINES external systems in devstack), the adapter can be built and tested locally without external-team coordination — only production cutover needs the external work. Default disposition stays externally-blocked; surface the mock-actionable interpretation only when explicitly directed.

Lessons Learned

  • Nextest parallelism: Default thread count equals logical CPUs (32 on dev machine). This overwhelms Docker devstack with concurrent HTTP connections, causing random 30s reqwest timeouts. Fixed by setting test-threads = 16 (physical cores) in .config/nextest.toml.

  • Docker image tags: Query registry API, don’t guess: curl -s "https://registry.hub.docker.com/v2/repositories/{owner}/{image}/tags/…​"

  • Keycloak 26+ hostname v2: KC_HOSTNAME_PORT is silently ignored; KC_HOSTNAME must be a full URL (e.g., http://host.docker.internal:8180).

  • curl -sf + -w '%{http_code}': -f causes status code concatenation. Use -s without -f, with || true for fallback.

Edit this page · latest