CI/CD Pipeline Overhaul

On this page

Context

The CI/CD pipeline has grown organically and has several issues that undermine reliability:

  1. E2E tests are broken — Playwright --with-deps calls apt-get on Alpine (docker:27), which doesn’t exist

  2. Integration test caches target/ — 20,000+ files uploaded/downloaded each run

  3. Integration test compiles Rust from scratch — installs rustup + nextest in docker:27 (Alpine), takes 20+ min

  4. Shell code duplicated 4× — Garage init and health checks copy-pasted between CI jobs and devstack scripts

  5. 8 identical container scan jobs — one per service, differing only by image name

  6. Missing GitLab templates — no .gitlab/ MR/issue templates (CODEOWNERS and CONTRIBUTING.adoc already exist)

  7. Silent test failures — allow_failure: true on integration/E2E means failures on main go unnoticed

  8. stop_review blocks feature branch pipelines — manual job without allow_failure: true leaves pipelines in "blocked" state

  9. No Docker layer caching — each docker-build starts fresh in a new dind VM

Status

All 6 phases implemented. Documentation updated. Ready for review.

Phases

Phase 1: Extract Reusable CI Scripts (DRY)

Extract duplicated Garage init (~30 lines × 2) and health-check wait (~30 lines × 2) into shared shell scripts.

Files:

  • Create devstack/ci/garage-init.sh — extracted Garage init logic

  • Create devstack/ci/wait-healthy.sh — from inline CI script

  • Edit .gitlab-ci.yml — replace inline blocks in integration-test and e2e-test with sh devstack/ci/*.sh

  • Update cargo xtask dev — source the shared scripts

Phase 2: Fix Integration Test Performance & 20K Artifacts

Switch RUST_IMAGE from rust:1.93-slim (Debian/glibc) to rust:1.93-alpine (musl). Use nextest archive to pass pre-compiled test binaries from rust-test to integration-test — zero recompilation. Remove target/ from cache.

Files:

  • Edit .gitlab-ci.yml — RUST_IMAGE, .rust-build-deps (apk instead of apt-get, linux-musl nextest), rust-test (add archive), integration-test (use archive, remove cache/rustup)

  • Edit .config/nextest.toml — add test-threads = 8 to ci and ci-integration profiles

Phase 3: Fix E2E Playwright on Alpine

Replace broken npx playwright install chromium --with-deps (calls apt-get on Alpine) with docker compose --profile e2e run using the existing tests/e2e/Dockerfile (mcr.microsoft.com/playwright:v1.58.2-noble).

Files:

  • Edit .gitlab-ci.yml — rewrite e2e-test job to use compose E2E service

Phase 4: Consolidate Container Scanning with Matrix

Replace 8 identical container-scan-* jobs (~40 lines) with a single parallel: matrix job.

Files:

  • Edit .gitlab-ci.yml — replace 8 jobs with 1 matrix job

Phase 5: Add GitLab MR & Issue Templates

Add .gitlab/ templates for standardized MRs and issues. CODEOWNERS and CONTRIBUTING.adoc already exist.

Files:

  • Create .gitlab/merge_request_templates/default.md

  • Create .gitlab/issue_templates/feature.md

  • Create .gitlab/issue_templates/bug.md

Phase 6: Harden Pipeline

  • Remove allow_failure: true from integration-test and e2e-test on main — failures should gate

  • Add allow_failure: true to stop_review — environment cleanup should not block feature branch pipelines

  • Add BuildKit inline caching to docker-build — --cache-from + BUILDKIT_INLINE_CACHE=1

Files:

  • Edit .gitlab-ci.yml — integration/E2E rules, stop_review rules, docker-build job

Documentation

  • .claude/docs/testing.md — CI pipeline description, nextest archive, E2E container approach

  • .claude/CLAUDE.md line 260 — rust:1.93-slim (CI) → rust:1.93-alpine (CI + Dockerfile)

  • .claude/docs/devstack.md — document devstack/ci/ scripts

Verification

  • Push to feature branch and verify CI pipeline runs (lint + test stages)

  • Phase 2: confirm no target/ cache upload, confirm nextest archive works

  • Phase 3: confirm E2E tests pass (Playwright in noble container, no apt-get error)

  • Phase 6: confirm stop_review no longer blocks, manual integration/E2E failures are reported

GitLab

  • Branch: feature/ci-cd-overhaul

  • Issue: TBD

  • Epic: N/A (single MR)

Edit this page · latest