Performance Testing Infrastructure

On this page

Status

Step Description Status

1

k6 project structure and shared helpers

Not started

2

Per-service k6 test scripts (craig-rules, craig-cases, craig-placement)

Not started

3

Per-service k6 test scripts (craig-exchange, craig-financial, craig-reporting)

Not started

4

Per-service k6 test scripts (craig-security, craig-intake, craig-web)

Not started

5

VU ramp profiles (smoke, load, stress, soak)

Not started

6

Threshold definitions and assertions

Not started

7

cargo xtask perf command

Not started

8

CI integration (manual job on main)

Not started

Epic: &TBD
Issues: #TBD
Branch: feature/performance-testing

Context

CRAIG has 8 backend services and a BFF (craig-web) — all need performance characterization before production deployment. No load testing infrastructure currently exists. Without baseline metrics, regressions go undetected and capacity planning is guesswork.

k6 is chosen over alternatives (Locust, Gatling, JMeter) because it uses JavaScript for test scripts, runs as a single Go binary with no runtime dependencies, produces structured JSON output suitable for CI assertions, and has first-class support for threshold-based pass/fail.

Scope

In scope:

  • k6 test runner setup with shared utilities (auth token acquisition, base URL resolution)

  • Per-service k6 load test scripts targeting at least 3 critical endpoints each

  • Four VU ramp profiles: smoke (1 VU), load (50 VU), stress (100 VU), soak (10 VU for 30 minutes)

  • Threshold definitions with p95 latency targets per endpoint type

  • cargo xtask perf command to run tests locally

  • CI integration as a manual pipeline job on main branch

  • Baseline documentation format for recording results

Out of scope:

  • Production monitoring and alerting (covered by OpenTelemetry plan)

  • Capacity planning and autoscaling configuration

  • Micro-benchmarks with criterion (separate effort)

  • tokio-console profiling (separate tooling concern)

Design

Directory Structure

tests/k6/
  helpers/
    auth.js          # Keycloak token acquisition
    config.js        # Base URL resolution, environment detection
    thresholds.js    # Shared threshold definitions
  services/
    rules.js         # craig-rules load tests
    cases.js         # craig-cases load tests
    placement.js     # craig-placement load tests
    exchange.js      # craig-exchange load tests
    financial.js     # craig-financial load tests
    reporting.js     # craig-reporting load tests
    security.js      # craig-security load tests
    intake.js        # craig-intake load tests
    web.js           # craig-web (BFF) load tests
  scenarios/
    smoke.js         # 1 VU, 1 iteration per endpoint
    load.js          # 50 VUs, 5 minute ramp-up, 10 minute sustained
    stress.js        # 100 VUs, 2 minute ramp-up, 5 minute sustained
    soak.js          # 10 VUs, 30 minute sustained
  run-all.js         # Orchestrator that imports all service scripts
  baselines/
    YYYY-MM-DD.json  # Recorded baseline results
    README.md        # Baseline format documentation

Threshold Definitions

Endpoint Type p95 Target Examples

Read (GET single)

<50ms

GET /v1/cases/cases/{id}, GET /v1/rules/rulesets/{id}

List (GET collection)

<100ms

GET /v1/cases/cases, GET /v1/placement/homes

Write (POST/PUT)

<200ms

POST /v1/cases/referrals, PUT /v1/placement/homes/{id}

Cross-service

<500ms

POST /v1/cases/investigations/{id}/safety-assessment (calls craig-rules)

BFF page load

<300ms

GET /dashboard, GET /cases

VU Ramp Profiles

Profile VUs Duration Purpose

Smoke

1

1 iteration per endpoint

Sanity check — endpoints respond correctly under zero load

Load

50

5min ramp + 10min sustained + 2min ramp-down

Expected production traffic simulation

Stress

100

2min ramp + 5min sustained + 1min ramp-down

Peak traffic / breaking point discovery

Soak

10

30min sustained

Memory leak and connection pool exhaustion detection

Authentication Strategy

k6 tests acquire a Keycloak token once per VU init phase using ROPC grant (same as integration tests):

// tests/k6/helpers/auth.js
import http from 'k6/http';

export function getToken(username, password) {
    const url = `${__ENV.OIDC_INTERNAL_URL}/realms/craig/protocol/openid-connect/token`;
    const payload = {
        grant_type: 'password',
        client_id: 'craig-api',
        username: username,
        password: password,
    };
    const res = http.post(url, payload);
    return JSON.parse(res.body).access_token;
}

Steps

Step 1: k6 Project Structure and Shared Helpers

Files: tests/k6/helpers/auth.js, tests/k6/helpers/config.js, tests/k6/helpers/thresholds.js

Create the directory structure. Implement shared helpers:

tests/k6/helpers/config.js — base URL resolution from environment variables with devstack defaults:

export const BASE_URLS = {
    rules:     __ENV.CRAIG_RULES_URL     || 'http://localhost:8001',
    cases:     __ENV.CRAIG_CASES_URL     || 'http://localhost:8002',
    placement: __ENV.CRAIG_PLACEMENT_URL || 'http://localhost:8003',
    exchange:  __ENV.CRAIG_EXCHANGE_URL  || 'http://localhost:8004',
    financial: __ENV.CRAIG_FINANCIAL_URL || 'http://localhost:8005',
    reporting: __ENV.CRAIG_REPORTING_URL || 'http://localhost:8006',
    security:  __ENV.CRAIG_SECURITY_URL  || 'http://localhost:8007',
    intake:    __ENV.CRAIG_INTAKE_URL    || 'http://localhost:8008',
    web:       __ENV.CRAIG_WEB_URL       || 'http://localhost:8080',
};

export const OIDC_INTERNAL_URL = __ENV.OIDC_INTERNAL_URL || 'http://localhost:9090';

tests/k6/helpers/auth.js — Keycloak ROPC token acquisition (see Design section). Cache token per VU to avoid repeated auth calls during iterations.

tests/k6/helpers/thresholds.js — shared threshold definitions:

export const READ_THRESHOLDS = { http_req_duration: ['p(95)<50'] };
export const LIST_THRESHOLDS = { http_req_duration: ['p(95)<100'] };
export const WRITE_THRESHOLDS = { http_req_duration: ['p(95)<200'] };
export const CROSS_SERVICE_THRESHOLDS = { http_req_duration: ['p(95)<500'] };
export const BFF_THRESHOLDS = { http_req_duration: ['p(95)<300'] };

Step 2: Per-Service Scripts (craig-rules, craig-cases, craig-placement)

Files: tests/k6/services/rules.js, tests/k6/services/cases.js, tests/k6/services/placement.js

Each script exports a default function and named scenario functions. Minimum 3 critical endpoints per service:

tests/k6/services/rules.js:

  • GET /v1/rules/rulesets — list rulesets

  • GET /v1/rules/rulesets/{id} — get specific ruleset

  • POST /v1/rules/evaluate — evaluate a rule (the hot path)

tests/k6/services/cases.js:

  • GET /v1/cases/cases — list cases

  • POST /v1/cases/referrals — create referral

  • GET /v1/cases/cases/{id} — get case detail

  • POST /v1/cases/investigations/{id}/safety-assessment — cross-service call to rules

tests/k6/services/placement.js:

  • GET /v1/placement/homes — list foster homes

  • POST /v1/placement/homes — create foster home

  • GET /v1/placement/placements — list placements

Each test function follows this pattern:

import http from 'k6/http';
import { check } from 'k6';
import { BASE_URLS } from '../helpers/config.js';
import { getToken } from '../helpers/auth.js';

let token;

export function setup() {
    token = getToken('jane.doe', 'password');
    return { token };
}

export default function(data) {
    const headers = { Authorization: `Bearer ${data.token}`, 'Content-Type': 'application/json' };

    // List
    const listRes = http.get(`${BASE_URLS.cases}/v1/cases/cases`, { headers, tags: { endpoint: 'list_cases' } });
    check(listRes, { 'list cases 200': (r) => r.status === 200 });

    // Create
    const createRes = http.post(`${BASE_URLS.cases}/v1/cases/referrals`, JSON.stringify({
        reporter_name: 'k6 Test',
        reporter_phone: '555-0100',
        child_name: 'Test Child',
        child_age: 5,
        allegation_type: 'neglect',
        narrative: 'k6 performance test referral',
        source: 'hotline',
    }), { headers, tags: { endpoint: 'create_referral' } });
    check(createRes, { 'create referral 200': (r) => r.status === 200 });
}

Step 3: Per-Service Scripts (craig-exchange, craig-financial, craig-reporting)

Files: tests/k6/services/exchange.js, tests/k6/services/financial.js, tests/k6/services/reporting.js

tests/k6/services/exchange.js:

  • GET /v1/exchange/partners — list partners

  • POST /v1/exchange/partners — create partner

  • GET /v1/exchange/agreements — list agreements

tests/k6/services/financial.js:

  • GET /v1/financial/payment-requests — list payment requests

  • POST /v1/financial/payment-requests — create payment request

  • GET /v1/financial/claims — list claims

tests/k6/services/reporting.js:

  • GET /v1/reporting/afcars — list AFCARS submissions

  • POST /v1/reporting/afcars — create AFCARS submission

  • GET /v1/reporting/data-quality/checks — list data quality checks

Step 4: Per-Service Scripts (craig-security, craig-intake, craig-web)

Files: tests/k6/services/security.js, tests/k6/services/intake.js, tests/k6/services/web.js

tests/k6/services/security.js:

  • GET /v1/security/audit-logs — list audit logs

  • GET /v1/security/admin-units — list admin units

  • GET /v1/security/consent-records — list consent records

tests/k6/services/intake.js:

  • GET /v1/intake/reports — list public reports

  • POST /v1/intake/reports — submit public report (unauthenticated)

  • GET /v1/intake/reports/{id} — get report detail

tests/k6/services/web.js (BFF — HTML responses):

  • GET /dashboard — dashboard page load

  • GET /cases — cases list page

  • GET /placement — placement list page

BFF tests require an authenticated session. Use the Keycloak code+PKCE flow or set a session cookie. If PKCE is complex for k6, test the underlying API calls instead and note the BFF overhead separately.

Step 5: VU Ramp Profiles

Files: tests/k6/scenarios/smoke.js, tests/k6/scenarios/load.js, tests/k6/scenarios/stress.js, tests/k6/scenarios/soak.js

Each scenario file imports service functions and configures k6 options.scenarios:

// tests/k6/scenarios/load.js
import { cases } from '../services/cases.js';
import { rules } from '../services/rules.js';

export const options = {
    scenarios: {
        cases_load: {
            executor: 'ramping-vus',
            exec: 'cases',
            startVUs: 0,
            stages: [
                { duration: '5m', target: 50 },   // ramp up
                { duration: '10m', target: 50 },   // sustained
                { duration: '2m', target: 0 },     // ramp down
            ],
        },
        rules_load: {
            executor: 'ramping-vus',
            exec: 'rules',
            startVUs: 0,
            stages: [
                { duration: '5m', target: 50 },
                { duration: '10m', target: 50 },
                { duration: '2m', target: 0 },
            ],
        },
    },
    thresholds: {
        'http_req_duration{endpoint:list_cases}': ['p(95)<100'],
        'http_req_duration{endpoint:create_referral}': ['p(95)<200'],
        'http_req_duration{endpoint:evaluate_rule}': ['p(95)<500'],
    },
};

Smoke profile uses shared-iterations executor with 1 VU and 1 iteration per endpoint.

Step 6: Threshold Definitions and Assertions

Files: tests/k6/helpers/thresholds.js, all scenario files

Define per-endpoint thresholds using k6 tags. Each service script tags requests with { tags: { endpoint: 'name' } }. Scenario files reference these tags in thresholds:

thresholds: {
    // Global
    http_req_failed: ['rate<0.01'],        // <1% error rate
    // Per-endpoint type
    'http_req_duration{endpoint:list_cases}': ['p(95)<100'],
    'http_req_duration{endpoint:get_case}': ['p(95)<50'],
    'http_req_duration{endpoint:create_referral}': ['p(95)<200'],
    'http_req_duration{endpoint:safety_assessment}': ['p(95)<500'],
},

k6 exits with non-zero status when thresholds are violated, which CI uses as a gate.

Step 7: cargo xtask perf Command

Files: xtask/src/main.rs

Add a perf subcommand to xtask:

// xtask/src/main.rs — add to command enum
Perf {
    /// Test profile: smoke, load, stress, soak (default: smoke)
    #[arg(long, default_value = "smoke")]
    profile: String,
    /// Specific service to test (default: all)
    #[arg(long)]
    service: Option<String>,
    /// Save results to baselines directory
    #[arg(long)]
    save_baseline: bool,
},

Implementation:

  1. Check that k6 is installed (run k6 version, error with install instructions if missing)

  2. Check that devstack is running (hit http://localhost:8001/healthz)

  3. Build the k6 command: k6 run tests/k6/scenarios/{profile}.js --out json=tests/k6/baselines/{date}.json

  4. If --service is specified, run only that service script: k6 run tests/k6/services/{service}.js

  5. If --save-baseline, copy JSON output to tests/k6/baselines/ with date-stamped filename

  6. Print summary table with pass/fail status per threshold

Step 8: CI Integration

Files: .gitlab-ci.yml

Add a manual CI job on main:

performance-smoke:
  stage: test
  image: grafana/k6:latest
  when: manual
  only:
    - main
  needs: ["build"]
  services:
    - name: docker:dind
  variables:
    OIDC_INTERNAL_URL: http://keycloak:9090
    CRAIG_RULES_URL: http://craig-rules:8001
    CRAIG_CASES_URL: http://craig-cases:8002
    CRAIG_PLACEMENT_URL: http://craig-placement:8003
    CRAIG_EXCHANGE_URL: http://craig-exchange:8004
    CRAIG_FINANCIAL_URL: http://craig-financial:8005
    CRAIG_REPORTING_URL: http://craig-reporting:8006
    CRAIG_SECURITY_URL: http://craig-security:8007
    CRAIG_INTAKE_URL: http://craig-intake:8008
    CRAIG_WEB_URL: http://craig-web:8080
  script:
    - k6 run tests/k6/scenarios/smoke.js --summary-export=summary.json
  artifacts:
    paths:
      - summary.json
    when: always
    expire_in: 30 days
Full load/stress/soak profiles require the complete devstack running. The CI smoke test validates endpoint availability and basic response times. Full performance characterization should run against a dedicated environment.

Baseline Documentation Format

Each baseline file (tests/k6/baselines/YYYY-MM-DD.json) is the raw k6 --summary-export JSON output. A companion tests/k6/baselines/README.md documents:

  • Date, commit hash, hardware specs

  • Profile used (smoke/load/stress/soak)

  • Summary table: endpoint, p50, p95, p99, error rate

  • Notable observations or regressions from previous baseline

Files Touched

File Change

tests/k6/helpers/auth.js

NEW: Keycloak ROPC token helper

tests/k6/helpers/config.js

NEW: Base URL resolution

tests/k6/helpers/thresholds.js

NEW: Shared threshold constants

tests/k6/services/rules.js

NEW: craig-rules load tests

tests/k6/services/cases.js

NEW: craig-cases load tests

tests/k6/services/placement.js

NEW: craig-placement load tests

tests/k6/services/exchange.js

NEW: craig-exchange load tests

tests/k6/services/financial.js

NEW: craig-financial load tests

tests/k6/services/reporting.js

NEW: craig-reporting load tests

tests/k6/services/security.js

NEW: craig-security load tests

tests/k6/services/intake.js

NEW: craig-intake load tests

tests/k6/services/web.js

NEW: craig-web BFF load tests

tests/k6/scenarios/smoke.js

NEW: 1 VU smoke profile

tests/k6/scenarios/load.js

NEW: 50 VU load profile

tests/k6/scenarios/stress.js

NEW: 100 VU stress profile

tests/k6/scenarios/soak.js

NEW: 10 VU / 30min soak profile

tests/k6/run-all.js

NEW: Orchestrator script

xtask/src/main.rs

Add perf subcommand

.gitlab-ci.yml

Add manual performance-smoke job

Verification

  1. k6 version — k6 CLI installed

  2. cargo xtask dev start — devstack running

  3. cargo xtask perf --profile smoke — smoke tests pass (all endpoints respond, thresholds met)

  4. cargo xtask perf --profile load — load tests complete, p95 latencies within thresholds

  5. cargo xtask perf --profile smoke --save-baseline — baseline JSON saved to tests/k6/baselines/

  6. Verify k6 exits non-zero when a threshold is violated (temporarily lower a threshold to test)

Documentation Updates

  • .claude/docs/testing.md — add performance testing section with xtask commands

  • CHANGELOG.adoc — entry under == Unreleased

  • docs/modules/ROOT/pages/developer-guide.adoc — reference k6 setup and cargo xtask perf

Edit this page · latest