CI Pipeline Optimization

On this page

Status

  • Create .adoc plan file and link in nav.adoc

  • Create GitLab issue (#37)

  • Create feature branch feature/ci-pipeline-optimization

  • Implement .gitlab-ci.yml changes (stage consolidation, runner upgrade, auto-trigger)

  • Fix E2E DinD networking (compose service name resolution)

  • CI cleanup: workflow:rules, remove rust-build, downsize rust-fmt, update review/stop_review

  • Update documentation (CHANGELOG.adoc, .claude/docs/testing.md)

  • Commit with Closes #37, push, open MR

  • Verify pipeline timing improvement

Problem

MR pipelines take ~75 minutes wall-clock despite only running lint + unit tests + security scans. Two root causes:

  1. Serial stages: rust-clippy (lint stage, ~35 min) must finish before rust-test (test stage, ~36 min) can start, even though they have no dependency on each other.

  2. Undersized runners: Both jobs run on saas-linux-medium-amd64 (4 vCPU). Rust compilation is CPU-bound and parallelizes well — more cores directly reduce compile time.

  3. No integration/E2E on MR: Integration and E2E tests are manual + main-only. The sort bug (duplicate sort_by params) and findInTable regression both slipped through because no MR pipeline ran them.

Current pipeline timing (pipelines 180-184)

Job Runner Duration Stage

rust-fmt

medium (4 vCPU)

~5 min

lint

rust-clippy

medium (4 vCPU)

~35 min

lint

rust-test

medium (4 vCPU)

~36 min

test (blocked by lint)

security scans (3 jobs)

default

~1 min

test

review (Antora)

default

~30s

deploy

Total wall-clock

~75 min

Solution

R1: Consolidate stages (clippy + test in parallel)

Replace separate lint and test stages with a single check stage. fmt, clippy, rust-test, and security scans all run in parallel.

stages:
  - check       # fmt + clippy + test + security scans — all parallel
  - build       # release binaries + Docker images — main only
  - integration # devstack integration tests
  - e2e         # Playwright browser tests
  - deploy      # Antora docs

R2: Upgrade runners to large (8 vCPU)

Change .rust-job template from saas-linux-medium-amd64 to saas-linux-large-amd64.

  • clippy: ~35 min → ~18 min (estimated)

  • rust-test: ~36 min → ~20 min (estimated)

  • CI minute cost: 3x multiplier vs 2x (50% more per job, but runs in half the time — net cost similar)

R3: Auto-trigger integration/E2E on MR pipelines

Add MR pipeline rules so integration and E2E tests run automatically when an MR targets main. Keep manual trigger for branch pushes and post-merge main.

# integration-test rules (add before existing main rule):
- if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"
  changes: ["**/*.rs", "**/Cargo.toml", "Cargo.lock"]
  # auto on MR — catches regressions before merge

R4: Cargo sccache (deferred)

Using sccache with an S3 backend would cache compiled artifacts across jobs. Higher-effort — deferred to a future plan. R1+R2+R3 give the biggest improvement for minimal effort.

Expected Result

Metric Before After (R1+R2+R3)

MR pipeline wall-clock

~75 min

~20-25 min

clippy + test

Serial (lint → test)

Parallel (check)

Runner size

medium (4 vCPU)

large (8 vCPU)

Integration on MR

Never

Auto

E2E on MR

Never

Auto

Implementation Detail

File: .gitlab-ci.yml

  1. Replace lint + test in stages: list with single check

  2. Change rust-fmt stage from lint to check

  3. Change rust-clippy stage from lint to check

  4. Change rust-test stage from test to check

  5. Change sast, secret_detection, dependency_scanning stage overrides from test to check

  6. Change .rust-job tags from saas-linux-medium-amd64 to saas-linux-large-amd64

  7. Add MR auto-trigger rule to integration-test (before existing main rule)

  8. Add MR auto-trigger rule to e2e-test (before existing main rule, with additional change paths)

File: .claude/docs/testing.md

Update the CI pipeline section to reflect: * Stage consolidation (check replaces lint + test) * Runner upgrade (medium → large) * Integration/E2E auto-trigger on MR pipelines

File: CHANGELOG.adoc

Add entry under == Unreleased → === Changed.

Files Modified

File Action

.gitlab-ci.yml

Stage consolidation, runner upgrade, MR auto-trigger rules

.claude/docs/testing.md

Update CI pipeline documentation

CHANGELOG.adoc

Add entry

docs/modules/ROOT/pages/plans/ci-pipeline-optimization.adoc

This plan file

docs/modules/ROOT/nav.adoc

Link plan file

Verification

  1. Push to a feature branch — verify fmt/clippy/test all run in parallel in check stage

  2. Create MR targeting main — verify integration-test and e2e-test auto-trigger

  3. Compare pipeline duration to previous runs (~75 min → ~25 min target)

  4. Verify main pipeline still works (build/deploy stages unchanged)

GitLab

  • Issue: chore: CI pipeline optimization — parallel stages, large runners, auto integration/E2E

  • Branch: feature/ci-pipeline-optimization

  • Commit message must include Closes #N

Edit this page · latest