rollback canaries when SLO burn rate exceeds policy
Automated canary analysis and SLO-based rollback in Kubernetes
14 min read
Canary deploys are easy to start; promoting or rolling back on evidence is not. Wire Flagger or Argo Rollouts to SLO burn-rate math so rollback happens when error budget consumption accelerates—not after a dashboard review at 3 AM.
Why canary deploys fail without SLO-driven decisions
Progressive delivery gives you the mechanism: shift traffic to a new version, observe, promote or revert. The decision logic is usually missing. Manual gating—someone watches a panel for fifteen minutes and clicks promote—does not scale and adds human latency. Static thresholds such as error rate above five percent for five minutes are disconnected from SLO targets and often too generous; by five percent errors you may have burned a large share of the monthly error budget. A canary can look healthy in isolation while aggregate SLO across all replicas slips because nobody compares versions against budget consumption. Rollback behavior is inconsistent: some services auto-revert, others wait for an operator. Teams either trust canaries blindly or add so many manual gates that the process is no faster than blue-green.
SLO burn rate as the promotion gate
Treat SLOs as the source of truth for promote and rollback—not gut-feel percentages. Define SLIs in Git with Sloth or equivalent, measure error budget burn rate during the canary window, and let Flagger or Argo Rollouts advance traffic only while burn stays within policy. When burn exceeds the multi-window threshold derived from your SLO objective, the controller reverts traffic to stable automatically. CI pushes image tags; the canary controller validates health. GitOps reconcilers apply desired state, but analysis gates prevent bad versions from receiving full traffic. Flow: build and push image, canary receives ten percent, Prometheus evaluates burn rate every thirty seconds, weight increases on pass or rollback on sustained fail.
Define SLOs in code with Sloth
Store PrometheusServiceLevel resources beside service manifests so objectives, queries, and alert names stay version-controlled. A 99.9% availability SLO means 0.1% error budget. Burn rate is current error rate divided by allowed error rate—14.4x over a one-hour window exhausts the monthly budget in about an hour; 72x over five minutes catches fast regressions. Sloth generates recording rules and multi-burn-rate alerts you can reuse in canary analysis templates.
apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
name: my-service-slo
namespace: production
spec:
service: "my-service"
labels:
team: "platform"
slos:
- name: "availability"
objective: 99.9
description: "99.9% of requests succeed"
sli:
events:
errorQuery: |
sum(rate(http_requests_total{
namespace="production",
service="my-service",
code=~"5.."
}[5m]))
totalQuery: |
sum(rate(http_requests_total{
namespace="production",
service="my-service"
}[5m]))
alerting:
name: "MyServiceHighErrorBudgetBurn"
labels:
severity: "critical"
annotations:
summary: "High error budget burn rate"
pageAlert:
labels:
severity: "page"Flagger canary analysis with custom burn-rate metric
Flagger increases canary weight on a schedule and queries Prometheus through MetricTemplate resources. Return a numeric burn rate from PromQL and compare it with thresholdRange on the Canary spec—not a boolean greater-than inside the query. Label metrics so canary and stable traffic are separable: version label, mesh subset, or pod label set by your ingress controller. Without that split, burn rate mixes versions and hides regressions in the canary slice.
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: my-service
namespace: production
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: my-service
service:
port: 8080
progressDeadlineSeconds: 600
analysis:
interval: 30s
threshold: 5
maxWeight: 50
stepWeight: 10
metrics:
- name: request-success-rate
thresholdRange:
min: 99
interval: 30s
- name: slo-burn-rate
templateRef:
name: slo-budget-burn
thresholdRange:
max: 14.4
interval: 30s
revertOnFailure: trueapiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
name: slo-budget-burn
namespace: production
spec:
provider:
type: prometheus
address: http://prometheus.monitoring:9090
query: |
(
sum(rate(http_requests_total{
namespace="production",
service="{{ target }}",
version="canary",
code=~"5.."
}[5m]))
/
sum(rate(http_requests_total{
namespace="production",
service="{{ target }}",
version="canary"
}[5m]))
)
/
(1 - 0.999)Argo Rollouts AnalysisTemplate alternative
Argo Rollouts runs AnalysisRuns against AnalysisTemplate metrics between canary steps. Use the same burn-rate PromQL with successCondition comparing result to your window threshold—14.4 for one-hour policy, higher for shorter windows. Pair with Istio, NGINX, or ALB traffic routing so stable and canary Services receive weighted traffic. Rollout status and AnalysisRun objects document each pass or fail for post-incident review.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: my-service
namespace: production
spec:
replicas: 3
strategy:
canary:
canaryService: my-service-canary
stableService: my-service-stable
trafficRouting:
istio:
virtualService:
name: my-service-vsvc
routes:
- primary
analysis:
templates:
- templateName: slo-check
startingStep: 2
args:
- name: service-name
value: my-service
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: slo-check
spec:
args:
- name: service-name
metrics:
- name: slo-budget-burn-rate
interval: 30s
count: 5
successCondition: result[0] < 14.4
failureLimit: 3
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
(
sum(rate(http_requests_total{
namespace="production",
service="{{args.service-name}}",
code=~"5.."
}[5m]))
/
sum(rate(http_requests_total{
namespace="production",
service="{{args.service-name}}"
}[5m]))
)
/
(1 - 0.999)End-to-end rollout, rollback trigger, and alerting
Trigger a rollout by updating the Deployment image; Flagger or Rollouts detects the change and starts analysis at ten percent traffic. Each interval queries error rate and burn rate; three consecutive failures initiate rollback to stable. Emit Prometheus alerts on elevated burn during canary windows so on-call sees why auto-rollback fired—silent reverts hide recurring build problems. Log metric snapshots and decision timestamps for every promote or abort.
kubectl set image deployment/my-service \
my-service=my-registry/my-service:v2.1.0 \
-n production
# Flagger or Rollouts: 90% stable / 10% canary, metrics every 30s(
sum(rate(http_requests_total{
namespace="production",
service="my-service",
code=~"5.."
}[5m]))
/
sum(rate(http_requests_total{
namespace="production",
service="my-service"
}[5m]))
)
/
(1 - 0.999)groups:
- name: canary-analysis
rules:
- alert: CanaryHighBudgetBurn
expr: |
(
sum(rate(http_requests_total{
namespace="production",
service="my-service",
version="canary",
code=~"5.."
}[5m]))
/
sum(rate(http_requests_total{
namespace="production",
service="my-service",
version="canary"
}[5m]))
) / (1 - 0.999) > 14.4
for: 2m
labels:
severity: warning
canary: "true"
annotations:
summary: "Canary burn rate exceeds 1h SLO policy; rollback may trigger"Operational practices: thresholds, separation, drills, and incident hooks
Derive every threshold from SLO objective and evaluation window—document the arithmetic in the repo README. Separate deploy triggers in CI from validation in Flagger or Rollouts so one tool pushes artifacts and another gates traffic. Start strict: false-positive rollbacks are safer than promoting a regression. Use multi-window burn rates—a five-minute window at 72x plus a one-hour window at 14.4x reduces noise from spikes while catching sustained damage. Test rollback quarterly with intentional bad builds or chaos experiments; an unexercised path fails during real incidents. When auto-rollback fires, open a ticket or page on-call—recurring silent aborts often mean flaky tests or mislabeled metrics. Keep canary and stable series separable in Prometheus, align gates with GitOps promotion rules, and treat SLO dashboards as deployment policy—not a wall display nobody reads during release hour.
Traffic splitting and feature flags are the foundation; this article adds SLO gates on top of progressive delivery with canary and feature flags in Kubernetes.
Burn-rate thresholds only make sense once SLIs and error budgets are defined, as in our SLO, SLI, and error budget guide for platform teams.