Kubernetes security hardening for production clusters
Kubernetes Security Hardening: A Practical Guide for Production Clusters
14 min read
Default clusters are easy targets for RBAC sprawl, open APIs, and plaintext etcd. This guide walks through control plane flags, Pod Security Standards, default-deny networking, node sysctl hardening, and Vault-style secrets—with a phased rollout plan.
Why default Kubernetes configs fail in production
Kubernetes is the de facto standard for container orchestration, but default configurations rarely suit production workloads. Security misconfigurations consistently rank among the most common and impactful vulnerabilities in cloud-native environments. According to the NSA and CISA Kubernetes Hardening Guide, most successful cluster attacks trace to a handful of well-known patterns: overly permissive RBAC, exposed API server endpoints, container privilege escalation, permissive pod networking, and secrets stored without encryption at rest. The challenge is not missing security tooling — the CNCF ecosystem offers many options — but hardening a cluster correctly without breaking applications. Teams either harden as an afterthought or apply controls so aggressively that developer productivity and CI/CD pipelines suffer. A third failure mode is inconsistent controls across environments, leaving staging clusters with settings that would never pass a production security review. The consequences are severe: ransomware actors target misconfigured API servers exposed to the internet; customer data breaches from container escapes can trigger regulatory penalties under GDPR, PCI-DSS, or SOC 2. Even when attacks are indirect, kernel-level issues such as CVE-2020-8554 (man-in-the-middle via NodePort routing) or container escape implications around Log4Shell demand systematic mitigation, not one-off patches. This guide focuses on five pillars: control plane hardening, worker node security, network policies, RBAC, and secrets management.
Control plane hardening
The API server is the primary attack surface. Disable anonymous authentication, enable audit logging with retention, enforce TLS 1.2+ with strong cipher suites, and wire encryption for secrets at rest. The kubeadm snippet below shows representative apiserver flags; tune paths and admission plugins to your distribution. Enable etcd encryption at rest — unencrypted etcd stores secrets in plaintext, so any node compromise with etcd access exposes the entire cluster. Rotate etcd certificates before expiry with kubeadm certs check-expiration and kubeadm certs renew all in a rolling fashion.
# kube-apiserver flags (kubeadm configuration snippet)
apiServer:
extraArgs:
anonymous-auth: "false"
enable-admission-plugins: "NodeRestriction,PodSecurity,ServiceAccount"
profiling: "false"
audit-log-maxage: "30"
audit-log-maxbackup: "10"
audit-log-maxsize: "100"
audit-log-path: "/var/log/kubernetes/audit.log"
audit-policy-file: "/etc/kubernetes/audit-policy.yaml"
encryption-provider-config: "/etc/kubernetes/encryption-config.yaml"
tls-min-version: "VersionTLS12"
tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
client-ca-file: "/etc/kubernetes/pki/ca.crt"
request-timeout: "10s"# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
- configmaps
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-32-byte-key>
- identity: {}RBAC: least privilege is not optional
Overly permissive ClusterRoles are endemic. The built-in cluster-admin role should never be bound to users or service accounts in production. Use role aggregation to define scoped admin roles, audit grants with kubectl auth can-i --list --as=<user> before approval, and enforce continuous policy with OPA or Kyverno.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: namespace-reader
labels:
rbac.example.com/aggregation-role: "true"
rules:
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: dev-team-namespace-reader
subjects:
- kind: Group
name: "engineering-dev"
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: namespace-reader
apiGroup: rbac.authorization.k8s.ioPod Security Standards
Enforce the restricted Pod Security Standard across namespaces. It blocks privileged containers, risky hostPath mounts, excessive capabilities, and weak secret projections. Workloads that genuinely need elevated privileges — node agents, CNI plugins — belong in dedicated namespaces with explicit exemptions and change control.
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
pod-security.kubernetes.io/enforce: "restricted"
pod-security.kubernetes.io/enforce-version: "v1.29"
pod-security.kubernetes.io/warn: "restricted"
pod-security.kubernetes.io/warn-version: "v1.29"Network policies: zero trust inside the cluster
By default every pod can reach every other pod. Implement default-deny policies and allow only required flows. The manifest set below denies all traffic, permits DNS to kube-dns, then allows frontend-to-backend on port 8080. Use a CNI that enforces NetworkPolicy (Calico, Cilium, Weave Net) and validate enforcement in staging before production cutover.
# Default deny all ingress and egress in the namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
# Explicitly allow DNS resolution (required for all pods)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: production
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- port: 53
protocol: UDP
---
# Allow frontend to backend communication
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-backend
namespace: production
spec:
podSelector:
matchLabels:
app: backend
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- port: 8080Node hardening
Harden worker nodes at the OS layer: disable swap, block unused kernel modules, and apply sysctl hardening for redirects, bridge filtering, and ptrace restrictions. Deliver these settings through configuration management (Ansible, cloud-init, Talos MachineConfigs). Immutable images such as Bottlerocket or Flatcar Container Linux further shrink the attack surface.
# Disable swap (Kubernetes requires swap to be off or accounted for)
swapoff -a
# Edit /etc/fstab to make swap permanent
sed -i '/swap/d' /etc/fstab
# Disable unused kernel modules
cat > /etc/modprobe.d/k8s.conf << 'EOF'
install br_netfilter /bin/true
install ceph /bin/true
install rbd /bin/true
EOF
# Kernel hardening parameters
cat > /etc/sysctl.d/99-kubernetes-hardening.conf << 'EOF'
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
kernel.yama.ptrace_scope = 2
kernel.dmesg_restrict = 1
kernel.kptr_restrict = 1
EOF
sysctl --systemSecrets management
Never store secrets as base64-encoded ConfigMaps — they decode trivially. Combine etcd encryption with a dedicated secrets manager. HashiCorp Vault with the Agent Injector is a production-proven pattern for injecting short-lived material at runtime. Rotate secrets proactively; use Vault dynamic secrets for database credentials where possible.
# Vault Agent Injector annotation pattern on a pod
apiVersion: v1
kind: Pod
metadata:
name: myapp
namespace: production
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "myapp-readonly"
vault.hashicorp.com/agent-inject-secret-db: "secret/data/production/database"
vault.hashicorp.com/agent-inject-template-db: |
{{- with secret "secret/data/production/database" -}}
DATABASE_HOST={{ .Data.data.host }}
DATABASE_PORT={{ .Data.data.port }}
DATABASE_USER={{ .Data.data.username }}
DATABASE_PASSWORD={{ .Data.data.password }}
{{- end }}
spec:
containers:
- name: myapp
image: myapp:latest
volumeMounts:
- name: vault-secret
mountPath: /vault/secrets
readOnly: true
volumes:
- name: vault-secret
emptyDir:
medium: MemoryPhased migration example
Consider a platform team running a multi-tenant cluster for three engineering teams. An initial review finds pods running as root, hostPath volumes everywhere, no network policies, default service accounts bound to cluster-admin, and etcd encryption disabled. Phase 1 — immediate API server controls (zero downtime): set anonymous-auth=false, tighten admission webhooks, and enable etcd encryption during a maintenance window; confirm workloads start using service account tokens only. Phase 2 — RBAC remediation: audit ClusterRoleBindings with kubectl get clusterrolebindings -o json | jq '.items[] | select(.roleRef.name=="cluster-admin")', create scoped team roles, and add OPA Gatekeeper constraints against over-privileged bindings. Phase 3 — Pod Security Standards: roll out restricted in warn mode first, list violations with kubectl, then fix images for non-root users and remove hostPath dependencies. Phase 4 — network policies: deploy default-deny per namespace starting with production; monitor allow-listed flows; validate with cilium-cli connectivity test. Phase 5 — secrets migration: move static Secrets to Vault, inject via Agent, use dynamic database credentials with a one-hour TTL, and revoke legacy static credentials after a grace period. The result is roughly 80% lower attack surface coverage on the MITRE ATT&CK Kubernetes matrix while keeping CI/CD and developer workflows intact.
Best practices and continuous hardening
1. Automate hardening — never apply security controls manually. Codify settings in Terraform, Ansible, or Talos MachineConfigs and fail CI when kube-bench exceeds your severity threshold. 2. Treat the control plane as the most critical asset: encrypt etcd, require client certificates, restrict etcd to API server nodes, and test backups quarterly. 3. Store NetworkPolicy manifests in Git beside application code; require owners to approve new allow rules; use topology-aware routing where available. 4. Layer secrets defenses: etcd encryption, Vault integration, and chmod 600 on kubelet PKI; prefer dynamic secrets over static credentials. 5. Monitor failed API authentication, RBAC changes, new admission webhooks, relaxed pod security contexts, and unexpected DNS from system namespaces — kube-prometheus-stack ships useful alert rules. 6. Scan images in CI with Trivy, Grype, or Snyk; pin digests in production instead of mutable latest tags. 7. Maintain a Kubernetes incident runbook: isolate nodes, revoke service account tokens, rotate credentials, rebuild namespaces from GitOps; exercise with Litmus or Chaos Mesh in staging. 8. Keep staging and development comparable to production on security controls, with only deliberate relaxations for debugging; every exception needs documented justification and a remediation ticket. Hardening is continuous — new CVEs and changing workloads demand IaC pipelines, regular audits, benchmarking, and chaos experiments as operational rhythm.
Cluster hardening still depends on how secrets leave Git and reach pods, which we cover in secrets management for DevOps and CI/CD.
Declarative rollouts and policy hooks make controls stick; pair this baseline with GitOps with Argo CD and Flux on Kubernetes.