eliminate deployment 502s with graceful pod termination
Graceful shutdown in Kubernetes: PodDisruptionBudgets, PreStop hooks, and connection draining
13 min read
Rolling updates and node drains still cause 502s when pods ignore SIGTERM. Combine application drain, PreStop delay for endpoint removal, PDBs for voluntary disruptions, and sidecar-aware termination so users never notice a rollout.
Why rolling updates still produce 502s
Users see 502s and connection resets during deploys even when the rolling strategy looks correct. Kubernetes can terminate pods at any time: rolling updates, node drains, Cluster Autoscaler scale-down, or preemption. Default termination sends SIGTERM, waits terminationGracePeriodSeconds (30s), then SIGKILL. Most apps never finish in-flight work in that window. The sequence matters: the pod enters Terminating and is removed from Endpoints and EndpointSlices asynchronously while PreStop runs, then SIGTERM reaches the container. If the app exits immediately, kube-proxy and cloud load balancers may still route to the dying IP. Service-mesh sidecars can stop before the app finishes outbound calls. PodDisruptionBudgets left unset allow a drain to eliminate every replica at once. Each failure looks small; during a rollout they stack into user-visible errors.
Drain at the application layer on SIGTERM
Platform knobs cannot save an app that ignores SIGTERM. On that signal, stop accepting new work, finish what is already open, then exit with zero. HTTP servers should close the listener, wait for active handlers, and shut down. gRPC servers should call GracefulStop so new RPCs are rejected while in-flight RPCs complete. Queue workers should finish the current message and refuse new ones. Database pools should close idle connections and wait for in-use ones instead of cutting sockets. Instrument the path: log SIGTERM received, active connection count, and shutdown complete. SIGKILL cannot be caught—everything useful must finish before it.
PreStop hooks bridge the endpoint removal race
PreStop runs before SIGTERM and counts against terminationGracePeriodSeconds. A short sleep gives kube-proxy, EndpointSlice consumers, and cloud load balancers time to stop sending new traffic before the process starts draining. Five seconds often works for ClusterIP Services; meshes and external LBs may need ten to fifteen. Keep the hook simple—if it hangs, it burns the whole grace window. Pair PreStop with readiness: once Terminating, probes should fail so the pod leaves Ready sooner. Formula for the grace period: PreStop delay plus longest expected in-flight request plus a safety buffer. Default thirty seconds is rarely enough for payment or long-poll APIs.
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- sleep 10PodDisruptionBudgets limit voluntary disruption
A PodDisruptionBudget constrains voluntary disruptions—drains, rolling node upgrades, voluntary eviction—not node crashes or OOM kills. Use minAvailable when you know the floor of healthy pods you must keep. Use maxUnavailable for percentage budgets across replica counts that change. Prefer an absolute minAvailable for critical user-facing APIs with a fixed replica floor. Always set a PDB on multi-replica traffic-serving workloads; without one, a single drain can remove every pod. Remember PDBs do not block Deployments from scaling down to zero if you ask for it, and forced drains can still override them. For unhealthy pods, Kubernetes 1.26+ unhealthyPodEvictionPolicy lets you choose whether PDB blocks eviction of pods that never became Ready.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-server-pdb
namespace: production
spec:
minAvailable: 2
selector:
matchLabels:
app: api-serverapiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-server-pdb
namespace: production
spec:
maxUnavailable: "25%"
selector:
matchLabels:
app: api-serverProduction pattern: rolling update, PreStop, and PDB together
Combine a conservative rolling strategy with PreStop delay, a realistic grace period, readiness probes, and a PDB. maxUnavailable zero with a small maxSurge keeps capacity during rollout while new pods become Ready. PreStop sleep covers endpoint propagation. terminationGracePeriodSeconds must exceed PreStop plus app drain. The PDB keeps voluntary drains from dropping below the availability floor. This pattern does not replace application SIGTERM handling—it only creates the time window the app needs.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-service
namespace: production
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: payment-service
template:
metadata:
labels:
app: payment-service
spec:
terminationGracePeriodSeconds: 90
containers:
- name: payment-service
image: registry.example.com/payment-service:v2.4.1
ports:
- containerPort: 8080
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 10"]
readinessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 5
failureThreshold: 2
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: "1"
memory: 512Mi
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: payment-service-pdb
namespace: production
spec:
minAvailable: 2
selector:
matchLabels:
app: payment-serviceSidecar-aware shutdown and native sidecars
Classic injected mesh proxies often received SIGTERM at the same time as the app and shut down first, cutting outbound calls mid-drain. Older Istio workarounds used longer PreStop sleeps and proxy.istio.io/config terminationDrainDuration so Envoy stayed up while the app finished. holdApplicationUntilProxyStarts fixes startup ordering, not shutdown—do not confuse the two. Prefer Kubernetes native sidecars (stable in recent releases, widely enabled by default): declare the proxy as an init container with restartPolicy Always. Native sidecars start before the app and stay until regular containers exit, which removes most proxy-died-first races. Linkerd and current Istio versions lean on this model; verify your cluster and mesh versions before dropping drain annotations. Keep shareProcessNamespace and wrapper scripts as last-resort bridges on older clusters only.
Test, measure, and align with your SLO
Prove the path in staging: kubectl drain a node that runs a replica, or send SIGTERM inside a pod, and confirm clients see no drop during the window. Align kubectl drain --grace-period with pod terminationGracePeriodSeconds so drains do not SIGKILL early. Watch 5xx and connection-reset rates during Deployments; correlate with pod Terminating events and your app drain logs. If p99 request latency is two seconds but grace is thirty with a ten-second PreStop, you may still SIGKILL long handlers—size the window to worst-case request duration plus PreStop. Headless Services and StatefulSets update endpoints differently; clients must retry and re-resolve DNS on failure. Graceful shutdown is quiet when it works: users never notice the rollout. When it fails, every deploy becomes a gamble against your error budget.
After pods drain cleanly, validate the rollout with automated canary analysis and SLO-based rollback.
Availability during disruptions belongs in the error budget from our SLO and SLI guide for platform teams.