catch performance regressions in CI before they reach production
Infrastructure performance testing with k6 and Gatling in CI/CD
13 min read
Monthly load tests miss regressions that land on every merge. Split smoke, scenario, and stress tiers with k6 and Gatling, gate CI on thresholds, and watch Kubernetes HPA, CPU throttling, and connection pools while the load runs.
Why late load tests miss infrastructure regressions
Many teams run performance testing as a pre-release checklist: one long staging run, a few fixes, then hope. Production pages still arrive when a merge quietly exhausts a database pool, tightens an ingress rate limit, or hits CPU throttling under HPA lag. Three failures stack: staging drifts from production topology for weeks; full load suites take half an hour so developers skip them; tests hit only application code and ignore DNS, TLS, connection pools, and autoscaling. You need gates that match pipeline speed—fast on every pull request, richer before release—and that observe the infrastructure while the load generator runs.
Three tiers: smoke, scenario, and stress
Split work by feedback speed. Smoke performance tests run under a minute on every PR with a handful of virtual users and hard thresholds—k6 exits non-zero (code 99 on threshold breach) so CI fails without custom parsing. Scenario tests model realistic journeys—login, browse, checkout—on a staging topology that mirrors production; Gatling shines for readable multi-step Scala or Java simulations with assertions. Stress and capacity tests ramp weekly or on release candidates to find breaking points, HPA reaction, and saturation. Prefer k6 for scriptable JS gates and cloud-native CI; prefer Gatling when complex protocol sequences and rich reports matter more than ultra-short PR jobs.
k6 smoke gate that blocks the merge
Keep smoke scripts tiny and deterministic. Point at a stable staging URL from secrets, not production. Put SLOs into options.thresholds so the process exit code is the gate—avoid reinventing pass/fail from raw JSON point streams. Export a summary artifact for trends; the threshold itself should already fail the job.
import http from "k6/http";
import { check, sleep } from "k6";
const BASE_URL = __ENV.BASE_URL || "https://staging-api.example.com";
export const options = {
vus: 5,
duration: "30s",
thresholds: {
http_req_duration: ["p(95)<300"],
http_req_failed: ["rate<0.01"],
checks: ["rate>0.99"],
},
};
export default function () {
const res = http.get(`${BASE_URL}/health`);
check(res, {
"status is 200": (r) => r.status === 200,
"body not empty": (r) => r.body && r.body.length > 0,
});
sleep(1);
}export const options = {
scenarios: {
stress: {
executor: "ramping-vus",
startVUs: 0,
stages: [
{ duration: "2m", target: 100 },
{ duration: "5m", target: 100 },
{ duration: "2m", target: 200 },
{ duration: "5m", target: 200 },
{ duration: "2m", target: 300 },
{ duration: "5m", target: 300 },
{ duration: "10m", target: 0 },
],
},
},
thresholds: {
http_req_duration: ["p(99)<1000"],
http_req_failed: ["rate<0.05"],
},
};Gatling scenarios for realistic journeys
Use Gatling when the gate must exercise multi-step flows with think time and percentile assertions across the whole journey. Run after deploy-to-staging on main or on a nightly schedule—not on every commit. Keep credentials in CI secrets, seed catalogs so product IDs exist, and fail the build on assertion breach the same way you fail unit tests.
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._
class CheckoutSimulation extends Simulation {
val httpProtocol = http
.baseUrl("https://staging-api.example.com")
.acceptHeader("application/json")
val checkoutScenario = scenario("Checkout Flow")
.exec(
http("Login")
.post("/auth/login")
.body(StringBody(s"""{"user":"loadtest","pass":"${sys.env.getOrElse("LOGIN_PASSWORD", "changeme")}"}"""))
.check(status.is(200))
)
.pause(2, 5)
.exec(http("Browse Catalog").get("/products").check(status.is(200)))
.pause(1, 3)
.exec(
http("Add to Cart")
.post("/cart/items")
.body(StringBody("""{"product_id":"sku-100","qty":1}"""))
.check(status.is(201))
)
.pause(1, 2)
.exec(http("Checkout").post("/orders").check(status.is(200)))
setUp(
checkoutScenario.inject(
rampUsers(50).during(30.seconds),
constantUsersPerSec(10).during(60.seconds),
)
).protocols(httpProtocol)
.assertions(
global.responseTime.percentile3.lt(500),
global.successfulRequests.percent.gt(99.0),
)
}Wire gates into GitHub Actions
Make smoke a required check on pull requests. Let k6 thresholds fail the step via exit code; upload --summary-export for history. Schedule Gatling after staging deploy on main. Never load-test production from CI without an explicit, approved window and traffic shaping.
jobs:
performance-smoke:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run k6 smoke gate
uses: grafana/[email protected]
env:
BASE_URL: ${{ secrets.STAGING_BASE_URL }}
with:
filename: tests/performance/smoke.js
flags: --summary-export=results/smoke-summary.json
- uses: actions/upload-artifact@v4
if: always()
with:
name: k6-smoke-summary
path: results/
performance-scenario:
if: github.ref == 'refs/heads/main'
needs: deploy-staging
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Gatling scenario
env:
LOGIN_PASSWORD: ${{ secrets.LOADTEST_PASSWORD }}
run: |
./gradlew gatlingRun -Dgatling.simulationClass=CheckoutSimulation
- name: Fail on Gatling assertion report
run: |
python scripts/validate-gatling.py \
--report build/reports/gatling \
--max-p95 500 \
--min-success 99Load-test Kubernetes infrastructure, not only the app
Run generators from a dedicated perf-testing namespace with ResourceQuota so staging tenants stay protected. Watch the target namespace during the run: HPA replica count, CPU throttling, OOMKilled events, ingress 429s, and pool wait times. On cgroup v2 read cpu.stat from /sys/fs/cgroup/cpu.stat inside the app pod. Correlate k6 or Gatling latency spikes with those signals—otherwise you optimize the wrong layer.
apiVersion: v1
kind: ResourceQuota
metadata:
name: perf-test-quota
namespace: perf-testing
spec:
hard:
requests.cpu: "4"
requests.memory: 8Gi
limits.cpu: "8"
limits.memory: 16Gi
pods: "20"
---
apiVersion: batch/v1
kind: Job
metadata:
name: perf-test-stress
namespace: perf-testing
spec:
backoffLimit: 1
template:
spec:
restartPolicy: Never
containers:
- name: k6
image: grafana/k6:0.54.0
args: ["run", "/scripts/stress.js"]
env:
- name: BASE_URL
valueFrom:
configMapKeyRef:
name: perf-config
key: target-url
resources:
requests:
cpu: 500m
memory: 256Mi
limits:
cpu: "1"
memory: 512Mi
volumeMounts:
- name: scripts
mountPath: /scripts
volumes:
- name: scripts
configMap:
name: k6-scripts#!/usr/bin/env bash
set -euo pipefail
kubectl get hpa -n production --watch &
kubectl top pods -n production --watch &
# cgroup v2 (Kubernetes 1.25+ nodes typically)
kubectl exec -n production deploy/api -- cat /sys/fs/cgroup/cpu.statOperational practices that keep gates trusted
Baseline every critical service before changing thresholds; store summaries in Prometheus, Grafana Cloud k6, or object storage so you spot a five-percent crawl across twenty merges. Align smoke p95 and error rate with published SLOs—not aspirational numbers. Use production-like payloads and skewed traffic, not uniform synthetic noise. Isolate external dependencies with stubs when flakiness would burn trust. Automate trend alerts separate from pass/fail. After stress finds a ceiling, feed capacity numbers into HPA and PodDisruptionBudget reviews. Performance in CI is a feedback loop at development speed: smoke on every PR, journeys before release, stress for capacity—and infrastructure metrics in the same window as the load.
Align p95 and error-rate gates with objectives from our SLO, SLI, and error budget guide for platform teams.
After a green stress gate, progressive delivery still needs automated canary analysis and SLO-based rollback.