enforce cluster standards before objects reach etcd
Kubernetes admission controllers: policy enforcement and configuration governance
13 min read
Bad manifests reach etcd when nothing sits in the admission path. Combine ValidatingAdmissionPolicy CEL, policy engines, and carefully operated mutating or validating webhooks—with Ignore-to-Fail rollout, namespace selectors, and cert-manager TLS—so defaults and denies land before persistence.
Why misconfigurations still reach the cluster
Most production incidents that look like “Kubernetes weirdness” start as manifests that should never have been admitted: no CPU or memory limits, root containers, hostPath mounts, images from random registries, missing cost or owner labels. Wikis and code review do not reject a create. Built-in controllers such as LimitRanger and NamespaceLifecycle help, but they do not encode your registry allowlist or probe requirements. Admission is the choke point between the API server and etcd—after authentication and authorization—where you mutate defaults and validate policy before persistence. Without a deliberate admission strategy, every namespace invents its own standards and audits become archaeology.
The admission chain in the right order
A request is authenticated and authorized through RBAC or another authorizer first. Only then does admission run. Mutating admission can change the object—inject defaults, labels, or sidecars. Object schema validation follows. Validating admission then accepts or rejects the final object; validating webhooks and ValidatingAdmissionPolicy run in this phase and must not mutate. If every validating check passes, the object is written to etcd. Treat mutating and validating as separate concerns: mutate for safe defaults, validate for hard denies. Never put business policy only in CI—GitOps and kubectl both hit the same admission path when configured correctly.
Choose the enforcement layer: CEL, engines, or custom webhooks
Prefer the lightest tool that fits. ValidatingAdmissionPolicy evaluates CEL in-process inside the API server—no webhook pod, lower latency, GA since Kubernetes 1.30. Use it for label requirements, replica ceilings, and simple securityContext checks. Prefer Kyverno or OPA Gatekeeper when you need rich mutation, generate rules, image verification, audit reports, or Rego shared outside the cluster—those engines still register as admission webhooks. Write a custom webhook only for logic CEL and policy engines cannot express, or when you must call an internal service during admission. Custom code owns TLS, availability, and upgrade risk; most platform teams should exhaust VAP and a policy engine before shipping Go handlers.
Native validation with ValidatingAdmissionPolicy
Define policy logic once, then bind it with ValidatingAdmissionPolicyBinding and validationActions such as Deny, Warn, or Audit. Start bindings in Audit or Warn on a labeled set of namespaces, fix charts, then switch to Deny. Keep failurePolicy Fail only after expressions are proven—misconfigured CEL with Fail blocks matching requests. Exclude kube-system and controller namespaces from matchConstraints.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: workload-baseline
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["pods"]
validations:
- expression: "object.metadata.labels.?team.hasValue()"
message: "Pod must have label team"
- expression: >-
object.spec.containers.all(c,
c.?securityContext.?runAsNonRoot == optional.of(true) ||
c.?securityContext.?runAsUser.orValue(0) > 0)
message: "Containers must not run as root"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: workload-baseline-apps
spec:
policyName: workload-baseline
validationActions: ["Deny"]
matchResources:
namespaceSelector:
matchLabels:
admission.policy/enforce: "true"Mutating webhooks for defaults—operate them like production services
When you need mutation beyond LimitRange or engine mutators, register a MutatingWebhookConfiguration with sideEffects None, explicit timeoutSeconds, and namespaceSelector opt-in. Serve TLS with cert-manager and inject the CA into caBundle via annotation so rotations do not strand the API server. Start failurePolicy Ignore during canary namespaces; switch to Fail only with dual replicas, probes, and dashboards. Set reinvocationPolicy IfNeeded if later mutators may require a second pass. Keep handlers idempotent—never append a second sidecar on UPDATE. Prefer one process exposing /mutate and /validate on a single port rather than two containers fighting for 8443.
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: config-defaults
annotations:
cert-manager.io/inject-ca-from: platform-system/webhook-tls
webhooks:
- name: defaults.platform.example.com
admissionReviewVersions: ["v1"]
sideEffects: None
timeoutSeconds: 5
failurePolicy: Ignore
matchPolicy: Equivalent
reinvocationPolicy: IfNeeded
clientConfig:
service:
name: admission-webhooks
namespace: platform-system
path: /mutate
port: 443
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE"]
resources: ["pods"]
scope: Namespaced
namespaceSelector:
matchLabels:
admission-config-defaults: "enabled"
objectSelector:
matchExpressions:
- key: admission.policy/skip
operator: DoesNotExistapiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: webhook-tls
namespace: platform-system
spec:
secretName: webhook-tls
duration: 2160h
renewBefore: 360h
issuerRef:
name: webhook-issuer
kind: Issuer
usages: ["server auth"]
dnsNames:
- admission-webhooks.platform-system.svc
- admission-webhooks.platform-system.svc.cluster.localValidating denies: registries, probes, and privileged mode
Express hard rules as validation—either CEL, Kyverno/Gatekeeper, or a validating webhook. Typical baseline: deny privileged containers and hostPath, require readiness probes on public Services’ pods, allow images only from your corporate registry. Do not treat docker.io/library as trusted by default in production. Test with kubectl --dry-run=server so admission runs without persisting. When a custom validating webhook is unavoidable, return Allowed false with a clear Message; log user, namespace, GVK, and decision for audit and SIEM.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: policy-validate
annotations:
cert-manager.io/inject-ca-from: platform-system/webhook-tls
webhooks:
- name: validate.platform.example.com
admissionReviewVersions: ["v1"]
sideEffects: None
timeoutSeconds: 5
failurePolicy: Ignore
clientConfig:
service:
name: admission-webhooks
namespace: platform-system
path: /validate
port: 443
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["pods"]
namespaceSelector:
matchLabels:
admission.policy/enforce: "true"#!/usr/bin/env bash
set -euo pipefail
kubectl label namespace app-team admission.policy/enforce=true --overwrite
kubectl apply --dry-run=server -f bad-pod-privileged.yaml
# Expect: admission webhook or ValidatingAdmissionPolicy denial
kubectl apply --dry-run=server -f good-pod.yamlOperate admission like a critical path service
Admission latency is API latency—alert on webhook P95 and error rate, and on API server apiserver_admission_webhook_* metrics. Size webhook Deployments with two or more replicas and PodDisruptionBudgets. Never apply cluster-wide Fail on day one; expand namespaceSelector gradually. Version policies in Git next to cluster config and review them like application code. Coordinate with multi-tenant quotas so LimitRange, ResourceQuota, and admission denies tell one story. Admission controllers are the last gate before cluster state—prefer native CEL and mature engines, reserve custom webhooks for true exceptions, and promote Ignore to Fail only when observability proves the path is safe.
Kyverno and Gatekeeper policy libraries build on the same admission path in our Policy as Code guide with OPA Gatekeeper and Kyverno.
Admission controls extend the baseline from our Kubernetes security hardening guide for production clusters.