Troubleshooting

The S3-compatible Garage service needs bucket and key initialization after startup. Since C4-gate (#1012) that runs as the in-graph garage-init compose one-shot on every docker compose up path — xtask and manual alike — before the seed (which gates on it completing). If you see errors about missing buckets or access denied:

Keycloak 26+ uses "hostname v2" configuration. The KC_HOSTNAME_PORT variable is silently ignored. KC_HOSTNAME must be a full URL including the port:

Since #1310 the devstack splits each service’s DB credential: gates connect as craig_<svc>_owner, serving containers as craig_<svc>_app (DML-only). Symptoms and remedies:

The RabbitMQ user must be defined in devstack/rabbitmq/definitions.json with a SHA-256 password hash. Environment variables alone do not configure users. If you change the password, regenerate the hash and update definitions.json.

Default service ports:

If a port is in use, check for lingering Docker containers: docker ps -a

Nextest defaults to logical CPU count for parallelism (e.g., 32 threads on a dev machine). This overwhelms the devstack with concurrent connections.

Fix: Set test-threads = 16 (physical core count) in .config/nextest.toml. This is already configured in the repository.

Keycloak access tokens have a 30-minute lifetime. If your test session exceeds this, you’ll see 401 Unauthorized errors.

Fix: Re-run the tests. For very long sessions, consider increasing the token lifetime in the Keycloak admin console (Realm Settings → Tokens).

CI runs in Docker-in-Docker (DinD). Services are reachable via Docker Compose service names, not host.docker.internal or localhost.

Correct CI URLs:

All test output is saved to test-results/ at the repo root:

The pre-push battery’s final sdk-test stage runs the Python SDK suite in a python:3.13-alpine container that pip install`s the build backend (`hatchling) from pypi.org. It intermittently fails to resolve pypi.org with [Errno -3] Try again (EAI_AGAIN) — a transient DNS failure inside the container, not a code or test defect:

Fix: If the push fails only at the sdk-test/Python stage with this pypi DNS error and every earlier battery stage passed, just re-push (the whole battery re-runs). Confirm the branch actually landed afterwards with git ls-remote.

A reqwest error like ConnectError("dns error", …​) whose url is a literal IPv4 address (e.g. http://127.0.0.1:38901/) is almost always a redirect chain landing on an unresolvable hostname — not an actual DNS problem with the literal IP. reqwest’s Error::url() preserves the original requested URL even when the failure occurs mid-redirect, so the message names 127.0.0.1 while the failed lookup was really something like host.docker.internal (a container-only name that does not resolve from outside containers).

Fix: Before suspecting nsswitch / NSS / IPv4-vs-IPv6, disable redirect following — Client::builder().redirect(reqwest::redirect::Policy::none()). If the error disappears, you were following a redirect chain to an unreachable host. (A DNS error against a real hostname such as https://example.com may be a genuine DNS problem; this rule applies when the failing URL is a literal IP and the error is unexpected.)

Rust 2024 reserves gen and ref as keywords. If you see a compilation error about reserved keywords, rename the identifier.

For Askama templates specifically:

When using curl with -f (fail on HTTP errors) and -w '%{http_code}', the -f flag causes the status code to be concatenated with the error output.

Fix: Use -s without -f, and handle errors with || true:

The CI pipeline has 4 stages: scan → promote → deploy → triage (see .gitlab-ci.yml).

The full test battery (validate, integration, e2e, perf, security regression) runs in the pre-push hook on every developer machine — CI deliberately does not duplicate it (see the deployment guide's CI/CD Pipeline section). CI does run a devstack-free subset: ci-tests executes cargo xtask validate --skip-devstack --skip-docker (fmt, clippy, build, the ci nextest profile’s unit-test pass) on MR pipelines touching Rust/Cargo/xtask paths, and the scheduled pentest/perf-load/perf-stress/cluster-tests jobs exercise the promoted images out-of-band. Anything devstack-bound ran in the pre-push battery, not CI.

Adding a comment to an epic via POST /groups/<id>/epics/<iid>/notes returns 404 for this group. This is stable behavior (tied to the gitlab.com epic-vs-work-item migration state), not a transient bug or a permissions problem — issue notes (POST /projects/<id>/issues/<iid>/notes) work normally, and only epic notes 404.

Workaround: Append the comment text to the epic’s description instead — GET /groups/<id>/epics/<iid>, then PUT /groups/<id>/epics/<iid> with {"description": "<old>\n\n---\n\n<new>"}. Closing the epic still works normally via PUT /groups/<id>/epics/<iid> with {"state_event": "close"}.

glab api --method PUT …​ -f "description=@/tmp/file" does not read the file — it sends the literal string @/tmp/file as the field value, silently overwriting the target field with that string.

Fix: For large or multi-line PUT bodies (such as an epic/issue description), use curl with an explicit JSON content type:

glab api --input file.json returns 415 on this endpoint (it does not set Content-Type: application/json). Always re-fetch after a description PUT and assert the expected markers are present — a silent wrong-body write is otherwise invisible.