organize GitOps repos for multi-environment Kubernetes fleets
GitOps repository structure: multi-environment Kubernetes configurations at scale
14 min read
Flat manifest folders break when you add clusters and teams. Separate application repos with Kustomize overlays, a platform infra repo for shared components, and optional environment control-plane config—then wire promotion, CODEOWNERS, and External Secrets so changes land where they belong.
Why flat GitOps folders fail as the fleet grows
A single repository with one directory of manifests works for one app and one cluster. It collapses when you add development, staging, production, and multiple teams. Staging changes leak into production because boundaries are social, not structural. Teams copy Deployment YAML into each environment folder; copies diverge and nobody knows which file is truth. Every team opens pull requests against the same monorepo, so reviews queue for days. Secrets land in Git history without clear ownership. Simple questions become hard: who owns payment-service in production, and which Helm chart version runs there? One layout mistake propagates through every deploy and every incident investigation.
Layered strategy: separate change rate and ownership
Separate what changes often from what changes rarely, and separate team-owned config from platform-owned config. Layer one is per-team application repositories: source, Dockerfile, Kubernetes manifests or charts, and CI. Layer two is a platform infrastructure repository for cluster add-ons, monitoring, ingress, and shared components. Layer three is optional for five or more clusters or regulated estates: an environment control-plane repository with cluster registration, AppProjects, namespace and RBAC templates, and policy CRDs. Application teams never edit platform-infra. Platform teams rarely edit application overlays. That ownership split prevents a resource-limit pull request from rewriting the production ingress controller.
Application repositories with Kustomize overlays
Keep the service close to its deploy config. Prefer a Kustomize base plus overlays for environment deltas when you own the manifests—patches are reviewable without rendering a full Helm template. Put shared fields in base; put replica counts, resource limits, ingress hosts, and environment ConfigMaps only in overlays. Helm remains appropriate when you consume third-party charts: pin chart versions in platform-infra or in the app repo and keep per-environment values files as the overlay equivalent. CI should update image tags or digests through a kustomize images transformer or Helm set, never by hand-editing production YAML after merge.
team-payment/
├── src/
├── Dockerfile
├── k8s/
│ ├── base/
│ │ ├── deployment.yaml
│ │ ├── service.yaml
│ │ ├── hpa.yaml
│ │ └── kustomization.yaml
│ ├── overlays/
│ │ ├── dev/
│ │ ├── staging/
│ │ └── production/
│ └── README.md
└── .github/workflows/
├── build.yaml
└── validate.yamlPlatform infra and optional environment control plane
The platform repository holds cluster-scoped and shared add-ons: cert-manager, ingress-nginx, monitoring values, network policies, External Secrets Operator, and the GitOps controller itself. Organize by cluster under clusters/ and share reusable component bases under components/ with overlays per environment. When fleets grow, move Argo CD Application and ApplicationSet definitions, Flux GitRepository and Kustomization objects, Kyverno or Gatekeeper policies, and namespace RBAC into a dedicated environment repo so operational control plane changes do not mix with application rollouts. Document promotion: change lands in the lower overlay first, then a pull request promotes the same image and config deltas upward.
platform-infra/
├── clusters/
│ ├── dev/
│ ├── staging/
│ └── production/
│ ├── monitoring/
│ ├── ingress-nginx/
│ └── network-policies/
└── components/
├── cert-manager/
├── external-secrets/
└── argocd/Kustomize overlays that make environment diffs reviewable
Base stays environment-agnostic. Production overlay sets namespace, labels, replica and resource patches, and adds production-only resources such as PodDisruptionBudget, Ingress, and NetworkPolicy. Run kustomize build in CI and locally before merge. A pull request that touches only overlays/production cannot change dev. Pin images by digest in higher environments when you need immutable promotion.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-service
labels:
app.kubernetes.io/name: payment-service
app.kubernetes.io/part-of: payments
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: payment-service
template:
metadata:
labels:
app.kubernetes.io/name: payment-service
spec:
containers:
- name: payment
image: registry.example.com/payment-service:placeholder
ports:
- containerPort: 8080
env:
- name: LOG_LEVEL
value: info
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512MiapiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: payment-prod
labels:
- pairs:
environment: production
app.kubernetes.io/name: payment-service
includeSelectors: false
resources:
- ../../base
- pdb.yaml
- ingress.yaml
- networkpolicy.yaml
images:
- name: registry.example.com/payment-service
newTag: "2.4.1"
digest: sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
patches:
- path: replicas-patch.yaml
- path: resource-limits-patch.yaml
- target:
kind: Deployment
name: payment-service
patch: |-
- op: replace
path: /spec/template/spec/containers/0/env/0/value
value: warnapiVersion: apps/v1
kind: Deployment
metadata:
name: payment-service
spec:
replicas: 5
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-service
spec:
template:
spec:
containers:
- name: payment
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: "1"
memory: 1Gi#!/usr/bin/env bash
set -euo pipefail
kustomize build k8s/overlays/production | kubeconform -strict -kubernetes-version 1.30.0
kubectl kustomize k8s/overlays/production | kubectl diff -f - || trueWire repos with ApplicationSets, CODEOWNERS, and CI validation
Do not invent one cluster URL from an overlay folder name. Map environments explicitly with a list or matrix generator so destination clusters and namespaces stay intentional. Use Argo CD AppProjects to bound which repos and clusters a team may target. Mirror the idea in Flux with GitRepository plus Kustomization per environment. CODEOWNERS force platform review on production overlays and platform-infra paths. Validate every overlay with kubeconform on pull request before merge—schema failures belong in CI, not in a failed sync on production.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: payment-service
namespace: argocd
spec:
generators:
- list:
elements:
- env: dev
cluster: https://kubernetes.default.svc
namespace: payment-dev
- env: staging
cluster: https://staging.cluster.example.com
namespace: payment-staging
- env: production
cluster: https://prod.cluster.example.com
namespace: payment-prod
template:
metadata:
name: payment-service-{{env}}
spec:
project: payments
source:
repoURL: https://github.com/myorg/team-payment
targetRevision: main
path: k8s/overlays/{{env}}
destination:
server: '{{cluster}}'
namespace: '{{namespace}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true# platform-infra
/clusters/production/ @platform-team-sre
/clusters/staging/ @platform-team-sre
/components/ @platform-team-core
# team-payment
/k8s/overlays/production/ @payment-team-lead @platform-team-sre
/k8s/overlays/dev/ @payment-teamname: Validate Kubernetes manifests
on:
pull_request:
paths:
- "k8s/**"
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install tools
run: |
curl -sL https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh | bash
sudo mv kustomize /usr/local/bin/
curl -sL https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | tar xz
sudo mv kubeconform /usr/local/bin/
- name: Build and validate overlays
run: |
for dir in k8s/overlays/*/; do
echo "=== Validating ${dir} ==="
kustomize build "${dir}" | kubeconform -strict -kubernetes-version 1.30.0
doneSecrets, drift, promotion, and documentation
Never commit secret values. Store ExternalSecret manifests that pull from AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault; use apiVersion external-secrets.io/v1 on current operators. Enable selfHeal for known-safe drift, and notify on sync failures so silent divergence is visible. Promote by pull request: merge image and config changes into staging, verify, then open a production overlay PR that copies the same digest—Git history is the audit trail. Keep a top-level README in every repo covering ownership, how to add an overlay, and the promotion path. For three to ten clusters, application repos plus one platform-infra repo is enough. Beyond that, or under strict audit requirements, add the environment control-plane repository and treat repository layout as architecture, not folder cosmetics.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: payment-db-credentials
namespace: payment-prod
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: payment-db-credentials
creationPolicy: Owner
data:
- secretKey: DB_PASSWORD
remoteRef:
key: /payment/prod/db-passwordOperator choice and sync semantics for Argo CD versus Flux are covered in our GitOps consistency and compliance guide.
Preview environments that fit the same overlay model appear in our ephemeral Kubernetes namespaces guide for pull request previews.