turn observability signals into reliable scaling actions
Observability-driven auto-scaling in Kubernetes: from metrics to action
14 min read
CPU-only autoscaling reacts after latency is already visible to users. Build a closed loop where Prometheus records saturation and queue pressure, KEDA applies bounded scaling behavior, and dashboards track scaling health as a first-class SRE signal.
Why resource-only autoscaling is not enough
Every production Kubernetes cluster faces the same trade-off: capacity must exist before demand arrives, but over-provisioning wastes budget while under-provisioning burns user trust. Most teams start with HPA on CPU and memory, which is useful but reactive. By the time utilization crosses a threshold and replicas appear, latency already spikes and queues are already backing up. The core gap is not missing data but missing control logic. Prometheus and Grafana show leading indicators—request rate growth, queue depth trend, p99 latency drift—yet autoscaling often consumes only a narrow resource slice. Observability-driven scaling closes that gap by converting service-level signals into bounded scaling decisions.
Reference architecture: collect, derive, act, and verify
Treat scaling as a closed-loop control system. Collect telemetry through OpenTelemetry Collector and Prometheus. Derive scaling signals with recording rules so KEDA does not execute expensive raw queries on each poll. Act through KEDA ScaledObjects that feed HPA behavior with separate scale-up and scale-down windows. Verify through dashboards and alerts that track scaling quality: time-to-scale, overshoot, replica churn, and SLO impact. This model decouples scaling from a single utilization metric and makes scaling behavior observable and reviewable in the same way as reliability policy.
Prometheus recording rules for scaling signals
Precompute normalized signals per service and namespace. Keep formulas robust against divide-by-zero and label mismatch. Use saturation ratio as a primary indicator (current request rate versus tested capacity per replica) and keep queue pressure and latency as secondary guards. For queue depth, use a gauge metric such as queue_depth and derive trend with delta or increase; avoid rate() over non-counter metrics.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: autoscaling-signals
namespace: monitoring
spec:
groups:
- name: scaling.signals
interval: 30s
rules:
- record: app:request_rps:2m
expr: |
sum by (namespace, service) (
rate(http_requests_total{job="app"}[2m])
)
- record: app:queue_depth_delta:2m
expr: |
sum by (namespace, service) (
delta(queue_depth{job="app"}[2m])
)
- record: app:latency_p99:5m
expr: |
histogram_quantile(
0.99,
sum by (le, namespace, service) (
rate(http_request_duration_seconds_bucket{job="app"}[5m])
)
)
- record: app:saturation_ratio
expr: |
app:request_rps:2m
/
clamp_min(app:capacity_rps, 1)KEDA ScaledObject with guarded HPA behavior
KEDA can combine multiple Prometheus triggers; scaling happens when any trigger exceeds its threshold. Use saturation ratio as primary scale-up input and queue depth delta as an accelerator for bursty workloads. Configure HPA behavior for fast scale-up and conservative scale-down to avoid flapping. Keep explicit minReplicaCount and maxReplicaCount to protect baseline reliability and cluster budget.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: api-gateway-scaler
namespace: production
spec:
scaleTargetRef:
name: api-gateway
minReplicaCount: 3
maxReplicaCount: 50
pollingInterval: 30
cooldownPeriod: 180
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleUp:
stabilizationWindowSeconds: 30
selectPolicy: Max
policies:
- type: Percent
value: 100
periodSeconds: 60
- type: Pods
value: 5
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
selectPolicy: Min
policies:
- type: Percent
value: 20
periodSeconds: 60
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus.monitoring:9090
metricName: saturation_ratio
threshold: "0.8"
query: |
app:saturation_ratio{namespace="production",service="api-gateway"}
- type: prometheus
metadata:
serverAddress: http://prometheus.monitoring:9090
metricName: queue_depth_delta
threshold: "20"
query: |
app:queue_depth_delta:2m{namespace="production",service="api-gateway"}Scaling health dashboard and alerting
Observability-driven scaling fails if teams cannot explain why replicas changed. Add a dedicated scaling health panel next to SLO dashboards: current replicas, request rate, saturation ratio, and replica changes. Alert on anti-patterns such as sustained high saturation with no scale-up, or frequent replica changes with flat traffic. This turns autoscaling from a black box into an auditable control loop.
# Current replicas
sum by (namespace, deployment) (
kube_deployment_status_replicas{namespace="production",deployment="api-gateway"}
)
# Replica change events in 10 minutes
changes(
kube_deployment_status_replicas{namespace="production",deployment="api-gateway"}[10m]
)
# Saturation versus threshold
app:saturation_ratio{namespace="production",service="api-gateway"}groups:
- name: autoscaling-health
rules:
- alert: ScalingLagUnderHighSaturation
expr: |
app:saturation_ratio{namespace="production",service="api-gateway"} > 0.9
and
avg_over_time(kube_deployment_status_replicas{namespace="production",deployment="api-gateway"}[5m])
==
min_over_time(kube_deployment_status_replicas{namespace="production",deployment="api-gateway"}[5m])
for: 5m
labels:
severity: warning
annotations:
summary: "Saturation is high but replicas stay flat"Operational practices: boundaries, load tests, and rollout safety
Start with one service that has clear traffic shape and known capacity_rps. Run load tests with k6, Locust, or Vegeta and measure time-to-scale, peak latency, replica overshoot, and recovery time. Keep separate scale-up and scale-down windows; aggressive downscaling often causes oscillation. Revisit min and max replica bounds monthly as workload seasonality changes. Version-control recording rules and scaler manifests in the same repository as service configuration. During canary or high-risk releases, combine scaling policies with SLO burn alerts so rollout and capacity controls reinforce each other. The goal is not frequent scaling events; the goal is stable user-facing performance under changing demand.
Start from a lean telemetry baseline before advanced scaling logic with our observability setup for small platform teams.
To ship service and infrastructure signals through one pipeline, reuse patterns from our production OpenTelemetry Collector guide.