isolate tenant workloads with layered quota and network controls
Kubernetes multi-tenancy resource isolation: stopping noisy neighbors with quotas and network policies
14 min read
One batch job with unbounded requests can throttle latency-sensitive APIs across a shared cluster. Stack tiered namespaces, ResourceQuotas, LimitRanges, default-deny NetworkPolicy, and optional dedicated node pools into one isolation model with cost labels finance can reconcile.
Why shared clusters need layered tenant isolation
In a cluster shared by multiple product teams, the noisy neighbor problem is persistent: one misbehaving workload consumes schedulable capacity, triggers node pressure evictions, throttles CPU on unrelated pods, and cascades through service dependencies. A batch team spawning hundreds of pods with missing resource bounds can violate SLOs for an API team on the same nodes. Without isolation you get SLA breaches, unpredictable spend, and lateral movement across namespaces. Kubernetes offers ResourceQuota, LimitRange, NetworkPolicy, and node placement primitives, but most teams apply them in isolation. Production multi-tenancy needs a composed model: resource caps, network boundaries, optional dedicated nodes for gold tiers, and labels that tie consumption back to owners.
Four layers: tiered namespaces, quotas, network, and node placement
Treat isolation as four complementary layers. Tiered namespaces use one namespace per tenant and environment—team-alpha-prod, team-alpha-dev—linked by shared tenant, tier, and cost-center labels rather than a flat free-for-all. ResourceQuota plus LimitRange caps aggregate tenant consumption and bound every container even when developers omit requests. NetworkPolicy enforces default-deny ingress and egress with explicit allow paths for ingress controllers, approved peer tenants, and DNS. Node placement adds dedicated pools for high-priority tenants through taints, tolerations, and topology spread so gold-tier workloads do not share CPU with batch neighbors. ResourceQuota is per namespace; hierarchy is operational—consistent naming, shared labels, and Git-managed quota tiers—not automatic inheritance unless you adopt HNC or vCluster for larger estates.
Tenant namespaces with ResourceQuota and LimitRange
Label every tenant namespace with tenant, tier, cost-center, and environment. Set aggregate CPU, memory, pod, storage, and object-count quotas per namespace. Pair quotas with LimitRange defaults and max values so a single pod cannot request four CPU because staging worked once. Gold tiers receive higher caps; silver tiers stay tighter on shared nodes.
apiVersion: v1
kind: Namespace
metadata:
name: team-alpha-prod
labels:
tenant: team-alpha
tier: gold
cost-center: engineering
environment: production
pod-security.kubernetes.io/enforce: baseline
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-alpha-quota
namespace: team-alpha-prod
spec:
hard:
requests.cpu: "30"
requests.memory: 60Gi
limits.cpu: "60"
limits.memory: 120Gi
pods: "100"
services.loadbalancers: "2"
persistentvolumeclaims: "10"
requests.storage: 500Gi
---
apiVersion: v1
kind: LimitRange
metadata:
name: team-alpha-defaults
namespace: team-alpha-prod
spec:
limits:
- type: Container
default:
cpu: "500m"
memory: 512Mi
defaultRequest:
cpu: "100m"
memory: 128Mi
max:
cpu: "4"
memory: 8Gi
maxLimitRequestRatio:
cpu: "10"
memory: "4"NetworkPolicy segmentation between tenants
Start from default-deny ingress and egress in every tenant namespace. Allow north-south traffic only from the ingress controller namespace. Allow east-west traffic only to explicitly approved peer namespaces—never implicit full-mesh between tenants. Always add a dedicated DNS egress rule to kube-dns; without it pods fail name resolution silently under deny-all egress. Validate policies with flow observation before enforcement in production.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: team-alpha-prod
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-controller
namespace: team-alpha-prod
spec:
podSelector: {}
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-nginx
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: team-alpha-prod
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53Dedicated node pools for gold-tier tenants
For strict performance isolation, dedicate node pools to high-priority tenants. Label nodes, apply a NoSchedule taint, and require matching tolerations on tenant Deployments. Add topologySpreadConstraints so replicas spread across nodes in the pool instead of stacking on one noisy host. Silver-tier tenants stay on shared pools with quota and network controls only.
kubectl label node node-pool-alpha-01 tenant=team-alpha node-pool=alpha-dedicated
kubectl taint nodes node-pool-alpha-01 tenant=team-alpha:NoScheduleapiVersion: apps/v1
kind: Deployment
metadata:
name: alpha-api
namespace: team-alpha-prod
spec:
replicas: 3
selector:
matchLabels:
app: alpha-api
template:
metadata:
labels:
app: alpha-api
tenant: team-alpha
spec:
nodeSelector:
tenant: team-alpha
tolerations:
- key: tenant
operator: Equal
value: team-alpha
effect: NoSchedule
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: alpha-api
containers:
- name: api
image: registry.example.com/alpha-api:v1.4.0
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: "2"
memory: 2GiTwo-tenant example: gold dedicated versus silver shared
Team Alpha runs on gold tier with higher quotas and dedicated nodes. Team Beta runs on silver tier with smaller quotas on shared nodes. Both get default-deny networking and LimitRange guardrails; only Alpha receives node taints and spread constraints. This pattern keeps platform cost predictable while protecting latency-sensitive tenants.
apiVersion: v1
kind: Namespace
metadata:
name: team-beta-prod
labels:
tenant: team-beta
tier: silver
cost-center: engineering
environment: production
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-beta-quota
namespace: team-beta-prod
spec:
hard:
requests.cpu: "10"
requests.memory: 20Gi
limits.cpu: "20"
limits.memory: 40Gi
pods: "30"
---
apiVersion: v1
kind: LimitRange
metadata:
name: team-beta-defaults
namespace: team-beta-prod
spec:
limits:
- type: Container
default:
cpu: 500m
memory: 512Mi
defaultRequest:
cpu: 100m
memory: 128Mi
max:
cpu: "2"
memory: 4GiOperational practices: observe first, enforce, and review quarterly
Observe consumption for two to four weeks before tightening quotas—unexpected OOMKills erode developer trust. Use Kyverno or Gatekeeper in audit mode to catch missing resource requests and privileged pods before hard enforcement. Enforce Pod Security Standards at the namespace level: baseline for most tenants, restricted for sensitive workloads. Alert when namespace quota usage exceeds eighty percent. Label every namespace with cost-center and tenant for OpenCost or Kubecost showback. Combine L3/L4 NetworkPolicy with identity-aware Cilium or a service mesh when you need mTLS and path-level control. Review quota tiers quarterly from Prometheus history and right-size caps as teams grow. Multi-tenancy is not one feature—it is quotas, limits, network policy, node placement, and governance composed into predictable performance and clear accountability.
groups:
- name: tenant-quota
rules:
- alert: TenantQuotaNearLimit
expr: |
sum by (namespace, resource) (
kube_resourcequota{type="used"}
)
/
sum by (namespace, resource) (
kube_resourcequota{type="hard"}
) > 0.8
for: 15m
labels:
severity: warning
annotations:
summary: "Namespace {{ $labels.namespace }} above 80% of {{ $labels.resource }} quota"apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: audit-container-resource-requests
spec:
validationFailureAction: Audit
background: true
rules:
- name: require-cpu-memory-requests
match:
any:
- resources:
kinds: [Pod]
validate:
message: "Containers must define CPU and memory requests."
pattern:
spec:
containers:
- resources:
requests:
cpu: "?*"
memory: "?*"Namespace budgets and cost-center labels extend patterns from our namespace FinOps guide with ResourceQuotas and cost attribution.
Default-deny segmentation and identity-aware policies are covered in our zero-trust Kubernetes networking guide with Cilium.