angri-tech markangri-tech
Request an audit

migrate cluster edge traffic from Ingress to Gateway API safely

Kubernetes Gateway API: migration patterns from Ingress

14 min read

Ingress packs routing, TLS, and policy into one object and leans on vendor annotations. Gateway API splits GatewayClass, Gateway, and HTTPRoute by ownership—migrate side by side with ingress2gateway, ReferenceGrant, and RequestMirror before you cut DNS.

Why Ingress becomes a platform bottleneck

Ingress remains widely deployed, but its model fights scaled platform ownership. One object mixes external listeners, TLS termination, and application routing. Non-trivial behavior—rate limits, header rewrites, canary weights—usually lives in controller annotations, so moving from nginx to another implementation is a rewrite. Ingress targets HTTP and HTTPS; TCP, UDP, and first-class gRPC need side CRDs. Platform and application teams edit the same objects, which creates RBAC friction. SIG-Network positions Gateway API as the successor: hierarchical resources, portable routing features, and controller-agnostic contracts. Migration is not find-and-replace—you introduce coexistence, convert routes, prove parity, then move DNS.

GatewayClass, Gateway, and HTTPRoute ownership

GatewayClass selects the controller implementation, similar to StorageClass; platform admins own it. Gateway represents an infrastructure edge—listeners, ports, TLS certificates, and which namespaces may attach routes; infrastructure or platform teams own it. HTTPRoute (and siblings such as GRPCRoute) carries hostnames, matches, filters, and weighted backendRefs; application teams own those objects and attach them through parentRefs. Keep one Gateway per environment or availability boundary, not one per service. Restrict listeners with allowedRoutes namespace selectors so only labeled teams can bind routes. Cross-namespace Secret or Service references need an explicit ReferenceGrant in the target namespace.

Migration pattern: side-by-side coexistence

Install Gateway API CRDs and a conforming controller such as Envoy Gateway, Istio, Cilium, or nginx Gateway Fabric. Create GatewayClass and a Gateway that mirrors ports and certificates of today’s Ingress controller. Translate Ingress rules into HTTPRoutes that parentRef the new Gateway while Ingress still serves production. Publish a second VIP or hostname for the Gateway, validate status Accepted and Programmed, compare latency and error rates, then shift DNS or load-balancer weight. Remove Ingress objects only after the Gateway path owns traffic. Do not expect HTTPRoute weights alone to split traffic between Ingress and Gateway—those weights balance backends behind one parent; cutover happens at DNS, CDN, or LB.

YAML · GatewayClass and production Gateway
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: production
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: production
  namespace: infra
spec:
  gatewayClassName: production
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              shared-gateway: "true"
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-tls
            kind: Secret
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              shared-gateway: "true"
YAML · HTTPRoute with weighted backends and header match
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-service
  namespace: app-team-a
spec:
  parentRefs:
    - name: production
      namespace: infra
  hostnames:
    - api.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1
      backendRefs:
        - name: api-v1
          port: 8080
          weight: 90
        - name: api-v2
          port: 8080
          weight: 10
    - matches:
        - headers:
            - name: X-Debug
              value: "true"
      backendRefs:
        - name: api-debug
          port: 8080

Convert with ingress2gateway and migrate namespace by namespace

Use the SIG-Network ingress2gateway tool to read Ingress objects and emit Gateway plus HTTPRoute YAML. Review the output: path and host rules usually translate cleanly; controller annotations often need manual mapping to HTTPRoute filters or implementation policies. Migrate one application namespace at a time—label it for Gateway access, apply routes, validate httproute status, then decommission that namespace’s Ingress. Store converted manifests in Git with the same promotion discipline you use for other platform config.

Bash · generate Gateway API manifests with ingress2gateway
#!/usr/bin/env bash
set -euo pipefail
# Install from upstream releases or: go install sigs.k8s.io/ingress2gateway@latest
ingress2gateway print --namespace app-team-a > ./gateway-manifests/app-team-a.yaml
kubectl get ingress -n app-team-a -o name
kubectl apply --dry-run=client -f ./gateway-manifests/app-team-a.yaml
kubectl get httproute -n app-team-a
kubectl describe gateway production -n infra

Replace annotations with filters, policies, and ReferenceGrant

Audit every Ingress annotation before cutover. Group them: native HTTPRoute filters (redirect, header modify, mirror, URL rewrite), controller policy attachments (Envoy Gateway BackendTrafficPolicy or ClientTrafficPolicy), and features that still need an extension. Prefer portable filters first. Use RequestMirror to shadow production requests to a Gateway backend while Ingress remains primary—compare logs and metrics before DNS moves. When a Gateway references a TLS Secret in another namespace, or a route targets a Service elsewhere, create ReferenceGrant in the owning namespace.

YAML · HTTPRoute filters and RequestMirror
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-shadow
  namespace: app-team-a
spec:
  parentRefs:
    - name: production
      namespace: infra
  hostnames:
    - api.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1
      filters:
        - type: RequestHeaderModifier
          requestHeaderModifier:
            add:
              - name: X-Edge
                value: gateway-api
            remove:
              - X-Internal-Debug
        - type: RequestMirror
          requestMirror:
            backendRef:
              name: api-shadow
              port: 8080
      backendRefs:
        - name: api-v1
          port: 8080
YAML · ReferenceGrant for cross-namespace Service
apiVersion: gateway.networking.k8s.io/v1
kind: ReferenceGrant
metadata:
  name: allow-app-routes
  namespace: shared-services
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: app-team-a
  to:
    - group: ""
      kind: Service
YAML · Envoy Gateway BackendTrafficPolicy retries
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: api-resilience
  namespace: app-team-a
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: api-service
  retry:
    numRetries: 3
    retryOn:
      triggers:
        - connect-failure
        - retriable-status-codes
      httpStatusCodes:
        - 502
        - 503

TCP and gRPC beyond HTTP Ingress

Gateway listeners can speak TCP and attach TCPRoute objects from the experimental channel when your controller supports them—useful for brokers or database proxies that previously needed ad-hoc CRDs. Prefer private networking and strong auth before exposing databases on a shared Gateway. GRPCRoute is the first-class path for gRPC matching by service and method once your CRD channel includes it. Keep protocol-specific routes on dedicated listeners with allowedRoutes kinds locked to TCPRoute or GRPCRoute so HTTP teams cannot attach the wrong kind.

YAML · TCP listener and TCPRoute
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: tcp-gateway
  namespace: infra
spec:
  gatewayClassName: production
  listeners:
    - name: redis
      protocol: TCP
      port: 6379
      allowedRoutes:
        kinds:
          - kind: TCPRoute
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
  name: redis
  namespace: cache
spec:
  parentRefs:
    - name: tcp-gateway
      namespace: infra
      sectionName: redis
  rules:
    - backendRefs:
        - name: redis-primary
          port: 6379

Operational practices for a durable cutover

Pin Gateway API CRD versions and upgrade deliberately—standard and experimental channels move at different speeds. Treat Gateway and HTTPRoute as GitOps config with CODEOWNERS matching the ownership model. Run dual dashboards with the same SLO definitions for Ingress and Gateway during coexistence. After a namespace is green on Gateway, delete its Ingress immediately to avoid two sources of truth. Document who may change GatewayClass, Gateway listeners, and application routes. Migration succeeds when ownership is clearer, annotations shrink, and traffic management features become portable—not when YAML files merely change kind names.

Weighted backends and canary cutovers on HTTPRoute pair with our progressive delivery guide for canary and feature flags.

After routes land in Git, keep overlays clean using our GitOps repository structure guide for multi-environment Kubernetes.

Tags:kubernetesgateway-apinetworkingplatform-engineering

You might also like

GitOps repository structure: multi-environment Kubernetes configurations at scale

Flat manifest folders break when you add clusters and teams. Separate application repos with Kustomize overlays, a platform infra repo for shared components, and optional environment control-plane config—then wire promotion, CODEOWNERS, and External Secrets so changes land where they belong.

Zero Trust networking in Kubernetes: network policies and mTLS with Cilium

Default Kubernetes networking lets any pod reach any other pod. Combine default-deny NetworkPolicy, SPIRE-backed mutual authentication, and Cilium eBPF enforcement to segment east-west traffic and prove service identity—without a sidecar on every pod.

Progressive delivery in Kubernetes: canary deployments and feature flags for controlled rollouts

Rolling updates alone still expose every user to risky changes at once. This guide combines Flagger-style canary traffic with feature flags so you can validate releases under real load and roll back fast without a full outage.

Discuss your infrastructure goals