scale batch and queue workloads from external events
KEDA event-driven autoscaling in Kubernetes: batch and queue workloads beyond CPU
14 min read
HPA scales on CPU and memory; queue consumers and batch jobs need queue depth, lag, and schedules. KEDA adds scale-to-zero, ScaledObject for long-lived workers, and ScaledJob for fan-out batches—without paying for idle pods between bursts.
Why HPA fails queue consumers and batch jobs
Horizontal Pod Autoscaler works for stateless APIs where CPU or memory tracks request load. It breaks for event-driven systems. Kafka consumers should scale on consumer group lag, not CPU—a worker processing one message per minute and one processing a thousand per second can look identical to HPA. Batch pipelines need scale-to-zero between runs; HPA minimum is one replica. S3 file drops, webhooks, and CDC events carry no meaningful resource signal. Cron windows need exact capacity for two hours, then zero. Teams that tune CPU thresholds for these cases either over-provision idle workers or under-provision growing backlogs and miss SLOs. HPA was built for request-driven services, not event-driven workloads.
How KEDA extends Kubernetes autoscaling
KEDA is a CNCF graduated operator with dozens of scalers for Kafka, RabbitMQ, SQS, Prometheus, cron, and more. You define ScaledObject or ScaledJob CRDs; the KEDA operator polls external sources, exposes metrics through the external.metrics.k8s.io API, and manages an HPA on your Deployment or Job template. ScaledObject attaches to Deployments and StatefulSets with optional scale-to-zero. ScaledJob creates ephemeral Jobs from queue depth—each unit of work runs and exits instead of holding long-lived pods. Multiple triggers on one ScaledObject scale when any trigger demands capacity. KEDA sits between external systems and native HPA behavior you still configure for scale-up aggressiveness and conservative scale-down.
Kafka consumer scaling with ScaledObject
For a consumer group reading topic orders, target roughly one pod per partition worth of lag. Set lagThreshold to the backlog per replica you accept, minReplicaCount to zero when idle, and maxReplicaCount to partition count or downstream capacity. Use TriggerAuthentication for SASL or TLS credentials instead of embedding secrets in the scaler. Pair cooldownPeriod with HPA scaleDown stabilization to avoid replica thrash when lag oscillates.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: order-processor-scaler
namespace: production
spec:
scaleTargetRef:
name: order-processor
pollingInterval: 15
cooldownPeriod: 120
minReplicaCount: 0
maxReplicaCount: 12
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 180
policies:
- type: Percent
value: 30
periodSeconds: 60
triggers:
- type: kafka
metadata:
bootstrapServers: kafka-cluster.kafka.svc:9092
consumerGroup: order-processor
topic: orders
lagThreshold: "100"
offsetResetPolicy: latest
authenticationRef:
name: kafka-authRabbitMQ fan-out with ScaledJob
When each message is independent and finite, ScaledJob beats a scaled Deployment. KEDA creates Jobs up to maxReplicaCount based on queue length; completed Jobs release CPU and memory immediately. Use accurate scaling strategy when you want roughly one Job per message. Store RabbitMQ credentials in TriggerAuthentication and reference them from the scaler—never hardcode connection strings in ScaledJob manifests.
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
name: email-sender-job
namespace: production
spec:
jobTargetRef:
template:
spec:
containers:
- name: email-worker
image: registry.example.com/email-worker:v2.4.1
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
restartPolicy: Never
pollingInterval: 10
successfulJobsHistoryLimit: 5
failedJobsHistoryLimit: 3
maxReplicaCount: 50
scalingStrategy:
strategy: accurate
triggers:
- type: rabbitmq
metadata:
protocol: amqp
queueName: email.delivery.high-priority
mode: QueueLength
value: "1"
authenticationRef:
name: rabbitmq-authapiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: rabbitmq-auth
namespace: production
spec:
secretTargetRef:
- parameter: host
name: rabbitmq-credentials
key: connection-stringCron windows plus Prometheus backlog triggers
Combine scheduled capacity with reactive scaling. A cron trigger guarantees replicas during nightly batch windows; a Prometheus trigger adds pods when report_queue_depth grows during the day. KEDA evaluates all triggers and scales to satisfy the highest demanded replica count. Keep timezone explicit and align desiredReplicas with downstream database connection limits.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: report-generator
namespace: analytics
spec:
scaleTargetRef:
name: report-generator
minReplicaCount: 0
maxReplicaCount: 20
pollingInterval: 30
triggers:
- type: cron
metadata:
timezone: Europe/Minsk
start: 0 2 * * 1-5
end: 0 5 * * 1-5
desiredReplicas: "10"
- type: prometheus
metadata:
serverAddress: http://prometheus.monitoring:9090
metricName: pending_reports
threshold: "5"
query: |
sum(report_queue_depth{service="report-generator"})Authentication separation and progressive HPA behavior
Keep TriggerAuthentication or ClusterTriggerAuthentication CRDs separate from ScaledObject so credential rotation never touches scaling logic. On AWS EKS use pod identity provider aws with IRSA instead of long-lived keys. Configure HPA behavior inside KEDA advanced block: immediate scale-up with Max selectPolicy, slow percentage-based scale-down to prevent flapping. When scaler reads fail, KEDA falls back to minReplicaCount—monitor scaler errors so silent fallback does not hide a growing backlog.
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: aws-sqs-auth
namespace: production
spec:
podIdentity:
provider: aws
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: sqs-consumer
namespace: production
spec:
scaleTargetRef:
name: sqs-consumer
minReplicaCount: 0
maxReplicaCount: 40
triggers:
- type: aws-sqs-queue
metadata:
queueURL: https://sqs.eu-west-1.amazonaws.com/123456789/orders
queueLength: "5"
awsRegion: eu-west-1
authenticationRef:
name: aws-sqs-authadvanced:
horizontalPodAutoscalerConfig:
behavior:
scaleUp:
stabilizationWindowSeconds: 0
selectPolicy: Max
policies:
- type: Percent
value: 100
periodSeconds: 60
- type: Pods
value: 10
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
selectPolicy: Min
policies:
- type: Percent
value: 25
periodSeconds: 120Operational practices: limits, monitoring, and tenant isolation
Always set minReplicaCount, maxReplicaCount, pollingInterval, and cooldownPeriod explicitly—defaults may surprise you in production. Cap Kafka maxReplicaCount at partition count; extra pods idle without partitions. Prefer ScaledJob for fan-out batch units with a clear end state. Alert on consumer lag above SLO, keda_scaler_errors_total increases, and HPA stuck at minReplicaCount while queue depth stays non-zero. Trend scaler metric values over time to right-size maxReplicaCount and node pools. If NetworkPolicy is enforced, allow KEDA operator pods to reach brokers, Prometheus, and the Kubernetes API. Restrict KEDA RBAC to namespaces it manages. KEDA turns Kubernetes into an event-driven platform: responsive scaling for queues and batches, scale-to-zero between bursts, and cost aligned with actual work in the queue.
Prometheus-based saturation scaling complements queue triggers in our observability-driven auto-scaling guide for Kubernetes.
Scale-to-zero savings still need namespace budgets from our namespace FinOps guide with ResourceQuotas.