What had to be replaced

The job was to get off the F5 appliances and onto something we ran ourselves. The requirements came back as a short list — the replacement had to cover:

  • Layer 4 and Layer 7 load balancing
  • TLS termination
  • Customizable backend server monitors
  • HA cluster mode, active/standby
  • Production ready throughput, more than 10 Gbit/s per cluster

Every one of those is phrased in F5 terms, which is fair enough — F5 is what we’d been running for years, and it’s the vocabulary everyone had for describing what the thing in front of the cluster does. That framing came with the job, and it quietly shapes every answer below. I’ll come back to it.

What follows is the first pass, built in a homelab over a weekend before anything went near production. A few of the obvious answers turn out to be wrong once you look at them properly, so I’ve put the corrections where the mistakes are rather than saving them for the end.

What the existing setup looks like

The starting point is a Kubernetes cluster with F5 appliances in front of it:

                 internet
                     |
            +--------+--------+
            |  R01        R02 |   edge routers
            +--------+--------+
                     |
            +--------+--------+
            |  core switches  |
            +--------+--------+
                     |
       +-------------+-------------+
       |   F5 BIG-IP pair          |   one VIP per virtual server
       |   LB-01  <-active/standby->   TLS terminate, re-encrypt
       |          LB-02            |   pool = the worker nodes
       +-------------+-------------+
                     |
          pool members hit NodePorts
                     |
     +---------------+---------------+
     |               |               |
+----+-----+   +-----+----+   +------+---+
| worker 1 |   | worker 2 |   | worker 3 |
+----------+   +----------+   +----------+

Every worker node is a member of an F5 pool. Each application gets a virtual server with its own VIP, and the pool members point at NodePorts on the workers. TLS terminates on the F5 and, in this config, gets re-encrypted on the way to the backend.

Hang onto that last part. It’s the one thing in the source config that no Ingress-based answer keeps by default, and nobody notices until much later.

Picking a replacement

I went with the NGINX Ingress controller plus MetalLB for the VIP.

That choice was about migration cost, not technical merit. It’s the boring, widely deployed option, it works with cert-manager, and ingressClassName is a per-Ingress field — so you can run the F5 ingress controller and the NGINX one side by side and move services over one at a time by editing a single line. The same line moves them back, which matters more than anything else on the list.

Two others I looked at and didn’t have time to test: Cilium Ingress, tempting because the cluster already runs Cilium as its CNI, and the Gateway API, which hit v1.0 earlier in the year but felt too new to put a production cutover on. There’s an update at the bottom about how well that call aged.

The lab

Six RKE2 nodes, three control plane and three workers, running Cilium as the CNI:

$ kubectl get nodes
NAME    STATUS   ROLES                       AGE   VERSION
k8sc1   Ready    control-plane,etcd,master   10d   v1.25.10+rke2r1
k8sc2   Ready    control-plane,etcd,master   10d   v1.25.10+rke2r1
k8sc3   Ready    control-plane,etcd,master   10d   v1.25.10+rke2r1
k8sc4   Ready    <none>                      10d   v1.25.10+rke2r1
k8sc5   Ready    <none>                      10d   v1.25.10+rke2r1
k8sc6   Ready    <none>                      10d   v1.25.10+rke2r1

Build instructions for this cluster are in an earlier post. Nodes sit on 192.168.88.0/24.

$ cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium             Desired: 6, Ready: 6/6, Available: 6/6
Deployment             cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2

NGINX Ingress controller

helm upgrade -i ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.metrics.enabled=true \
  --set controller.metrics.serviceMonitor.enabled=true \
  --set controller.metrics.serviceMonitor.additionalLabels.release="prometheus"

The serviceMonitor flags only do anything once the Prometheus operator is running, so in practice this ran twice — once without them, then again as an upgrade.

MetalLB for the VIP

On bare metal nothing hands out an external IP to a Service of type LoadBalancer, so it sits at <pending> forever. MetalLB is what fills that gap.

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb

Then an address pool and an advertisement:

---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: default
spec:
  addresses:
  - 192.168.88.20-192.168.88.30
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: first-l2advertisement
  namespace: default
spec:
  ipAddressPools:
    - "first-pool"

Pick that range from a subnet the nodes are actually on. L2 mode works by having a node answer ARP for the VIP, so if the nodes have no path to the address, no client ever reaches it. Nodes here are on 192.168.88.0/24, so the pool is too.

L2 mode has a second consequence that’s easy to miss, and it comes back to bite in the throughput section.

Monitoring, and why it does not answer the monitors requirement

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm upgrade -i prometheus prometheus-community/kube-prometheus-stack \
  --namespace prometheus --create-namespace \
  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

The ingress-nginx project publishes a Grafana dashboard that reads the controller’s metrics endpoint on port 10254. Import it, point it at the Prometheus datasource, and you get request rates, latency percentiles and upstream response times per ingress.

Grafana showing NGINX ingress metrics

This is where it’s tempting to tick off “customizable backend server monitors” and move on. Don’t — it’s answering a different question.

An F5 monitor is a health check. It probes a pool member on a schedule, and when the probe fails the member drops out of rotation. It’s a control loop, not a graph. The thing doing that job in Kubernetes is the pod readinessProbe, which pulls a failing pod out of the Endpoints list. That’s what actually keeps traffic off a sick backend, and Grafana has nothing to do with it.

It gets thinner from there. F5 monitors can be anything — send a string and require a specific response back, run an external script, check a database. A readinessProbe gives you an HTTP GET, a TCP connect, or a command inside the container, and that last one only ever tells you about that one pod. There’s no way to say “probe this backend from the load balancer’s point of view with this payload”, because community ingress-nginx has no active upstream health checking at all. That’s an NGINX Plus feature. So the honest answer for this stack is partially, through readiness probes, and you lose custom probe logic — worth writing down as a known gap instead of quietly checking the box.

One annotation makes the gap worse, and I was using it:

nginx.ingress.kubernetes.io/service-upstream: "true"

That tells NGINX to proxy to the Service ClusterIP instead of to the individual pod endpoints, which hands the load balancing to Cilium and eBPF. Useful if you want the CNI making those decisions. It also means NGINX has no per-endpoint view of the backends whatsoever, so if you care about the monitors requirement it’s working against you.

TLS termination with cert-manager

helm repo add jetstack https://charts.jetstack.io --force-update

helm upgrade -i cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true \
  --set 'extraArgs={--dns01-recursive-nameservers=1.1.1.1:53}'

A ClusterIssuer rather than an Issuer, so ingresses in any namespace can use it. DNS-01 through Cloudflare, since the lab has no inbound port 80 from the internet:

---
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret-cloudalbania-com
  namespace: cert-manager
type: Opaque
stringData:
  cloudflare-api-token-cloudalbania-com: <CLOUDFLARE_TOKEN_HERE>
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod-cloudalbania-com
spec:
  acme:
    email: <YOUR_EMAIL_HERE>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: example-issuer-account-key
    solvers:
    - dns01:
        cloudflare:
          email: <YOUR_EMAIL_HERE>
          apiTokenSecretRef:
            name: cloudflare-api-token-secret-cloudalbania-com
            key: cloudflare-api-token-cloudalbania-com

A demo app with a TLS ingress:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kuard
  namespace: ingress-demo
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod-cloudalbania-com"
    nginx.ingress.kubernetes.io/service-upstream: "true"
    external-dns.alpha.kubernetes.io/hostname: "nginx-test.k8s.cloudalbania.com"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - nginx-test.k8s.cloudalbania.com
    secretName: nginx-test-tls
  rules:
  - host: nginx-test.k8s.cloudalbania.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kuard
            port:
              number: 80

$ kubectl get ingress -n ingress-demo kuard
NAME    CLASS   HOSTS                             ADDRESS         PORTS     AGE
kuard   nginx   nginx-test.k8s.cloudalbania.com   192.168.88.20   80, 443   4m8s

$ curl -v https://nginx-test.k8s.cloudalbania.com
*   Trying 192.168.88.20:443...
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=nginx-test.k8s.cloudalbania.com
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.

That works. It’s also a quiet security downgrade, and it took me a while to spot it. The F5 config being replaced has both profiles on the HTTPS virtuals:

profiles {
    wildcard.cloudalbania.com-clientssl {
        context clientside
    }
    wildcard.cloudalbania.com-serverssl {
        context serverside
    }
}

clientside is TLS termination. serverside is re-encryption on the way to the pool member. The F5 decrypts, looks at the traffic, then opens a fresh TLS connection to the backend. The Ingress above terminates and then speaks plaintext HTTP to the pod for the rest of the trip.

On a cluster with an encrypting CNI that might be a fine trade — Cilium will do WireGuard or IPsec transparently. It’s still a change in the threat model between the old thing and the new one, and it belongs in the migration doc rather than in whatever the next audit turns up. Plain Ingress can re-encrypt with nginx.ingress.kubernetes.io/backend-protocol: "HTTPS", but actually verifying the backend certificate takes more annotations and there’s no portable way to write it down.

external-dns, and a way to delete production

DNS names come from the ingress objects themselves, via external-dns talking RFC 2136 to the authoritative servers. In the homelab those are Technitium, which I run as an HA pair and wrote about separately. It accepts dynamic updates signed with a TSIG key, which is all external-dns needs, so --rfc2136-host points at the primary and the secondary picks the records up through zone transfer. Any RFC 2136 capable server works here, BIND and Knot included, and nothing below is specific to Technitium.

        args:
        - --registry=noop
        - --provider=rfc2136
        - --rfc2136-host=192.168.178.2
        - --rfc2136-port=53
        - --rfc2136-zone=k8s.cloudalbania.com
        - --rfc2136-tsig-secret=<TSIG_SECRET_HERE>
        - --rfc2136-tsig-secret-alg=hmac-sha256
        - --rfc2136-tsig-keyname=external-dns
        - --rfc2136-tsig-axfr
        - --source=ingress
        - --domain-filter=k8s.cloudalbania.com

$ host nginx-test.k8s.cloudalbania.com
nginx-test.k8s.cloudalbania.com has address 192.168.88.20

Look hard at --registry=noop. The registry is how external-dns remembers which records belong to it, normally by writing a companion TXT record next to each one. With noop it keeps no ownership state at all, so every record in the zone is fair game. Point that at a zone that still holds the F5’s live A records mid-migration and it’ll happily overwrite or delete them, using a TSIG key you gave it precisely so it could.

For a lab zone with nothing else in it, noop is fine. For a real migration you want --registry=txt with a --txt-owner-id, plus --policy=upsert-only until the cutover is done so it can create and update records but never remove one. That’s the difference between a tool that adds DNS records and a tool that can empty a production zone while you’re at lunch.

High availability, active/standby, and what MetalLB actually gives you

The obvious move is to spread the controller across the workers, starting by keeping the pods off the control plane nodes:

---
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-role.kubernetes.io/control-plane
                operator: NotIn
                values:
                  - "true"

and scale to the number of workers:

$ WORKER_NODES=$(kubectl get nodes | grep '<none>' | awk '{print $1}' | wc -l)
$ kubectl scale deployment -n ingress-nginx ingress-nginx-controller --replicas $WORKER_NODES

Two problems here, and they’re different kinds of problem.

The first is mechanical. replicas: 3 on a three worker cluster does not give you one pod per worker — the scheduler is free to put two on k8sc4 and none on k8sc6, and after a drain it usually does exactly that. If you want one per node, ask for it: controller.kind: DaemonSet in the Helm values, or a topologySpreadConstraint on the hostname. And doing any of this with kubectl patch against a Helm-managed Deployment means it vanishes at the next helm upgrade, so put it in the values file.

The second is more interesting. We needed active/standby, this describes an active/active pod layout, and the actual active/standby behaviour was sitting in MetalLB the whole time.

In L2 mode MetalLB elects one speaker per service. That node answers ARP for the VIP, so every packet for that VIP arrives on that one node. From there Cilium spreads connections across Ingress pods anywhere in the cluster, so the pod layer really is active/active — but the front door is a single node. Kill it and another speaker wins the election and sends a gratuitous ARP. Failover takes however long the surrounding network needs to believe the new ARP, usually a few seconds, and anything in flight is dropped.

So we had our active/standby after all, just one layer lower than expected. It’s also what breaks the next thing on the list.

Getting to production throughput

I was expecting that horizontal scaling, deploying multiple replicas of the NGINX Ingress controller, would increase capacity, and that MetalLB would distribute traffic among all the Ingress pods.

MetalLB does no such thing. In L2 mode it doesn’t distribute anything. It isn’t in the data path at all — it’s a failover mechanism that makes one node claim an IP. Every packet for a VIP comes in through that one node’s NIC, and then Cilium fans it out to pods across the cluster. Adding Ingress replicas gives you more CPU to process requests with. It does nothing for how many bytes per second can get into the cluster, because that’s capped by a single node’s link. Six machines with 10 Gbit/s NICs still gives you a 10 Gbit/s ceiling per VIP, and buying a seventh machine won’t move it.

So you can’t tune your way out of this one. There are three options, and picking any of them changes the design rather than a config value:

  • BGP mode is the real fix. MetalLB peers with the edge routers and advertises the VIP from every node that has a ready endpoint. The routers install an ECMP route and hash flows across the next hops, so your ceiling becomes the sum of the participating NICs. It needs whoever runs the routers to agree, which is a meeting rather than a helm install — and that meeting wants to happen before anyone writes a throughput number into a requirements doc.
  • VIP sharding is the cheap version. Run several Ingress controller Service objects so different applications get different VIPs. Each VIP elects its own speaker and MetalLB spreads those elections across nodes, so the load lands on different NICs. Doesn’t help one application that needs more than a NIC’s worth on its own.
  • NGINX tuning moves the CPU ceiling, not the network one. worker_processes, worker_connections and an HPA are all worth having when the controller is CPU bound on TLS handshakes. They just don’t answer this particular question.

Converting F5 SCF files to Ingress objects

Next question in the migration: can the existing F5 config come across automatically? Nobody wants to retype a few hundred virtual servers by hand. Parts of it can.

An SCF file is F5’s flat text config export. Three object types matter here:

ltm node 192.168.88.20 {
    address 192.168.88.20
}

ltm pool WEB-SERVERS-1 {
    members {
        192.168.88.20:31002 {
            address 192.168.88.20
            session monitor-enabled
        }
        192.168.88.21:31002 {
            address 192.168.88.21
            session monitor-enabled
        }
    }
}

ltm virtual WEB_SERVER_1_443_HTTPS_VIP {
    destination 192.168.200.1:443
    description admin.k8s.cloudalbania.com
    ip-protocol tcp
    mask 255.255.255.255
    pool WEB-SERVERS-1
    profiles {
        wildcard.cloudalbania.com-clientssl {
            context clientside
        }
        wildcard.cloudalbania.com-serverssl {
            context serverside
        }
    }
    source 0.0.0.0/0
}

The mapping is reasonably clean. A pool is a named set of backends, so it becomes a Service. A virtual server is a listener that sends matching traffic to one of those sets, so it becomes an Ingress. The hostname lives in the virtual’s description field — that’s a local convention, not a guarantee, and it’s the first thing to check against a real config before you trust any of this.

So I wrote a Python script to read an SCF and spit out YAML. It’s in a gist, bugs and all, because the bugs are the interesting part. It produced perfectly plausible looking output on the sample file, with some errors that only show up once you try to apply the result.

Two things came out of writing it that are worth knowing before you write your own.

The port numbers aren’t in the SCF file. The pool members point at NodePorts, so that’s all the converter can see. But a NodePort is an artifact of how the F5 reached the pod, not a property of the pod, and once the Ingress controller runs inside the cluster there’s no reason for one to exist at all. Getting rid of them is one of the real wins of the migration. What you actually want is the container port, and the only place that lives is the running cluster:

$ kubectl get service -n ingress-demo -o json | jq '.items[] | {name: .metadata.name, ports: [.spec.ports[].targetPort]}'

So a converter has to read the live cluster alongside the SCF file. Working from the SCF alone gets you Services nobody can reach.

Virtuals and Ingresses don’t map one to one. On the F5, the HTTP and HTTPS versions of the same site are two separate virtual servers, one per port. In Ingress they’re a single object with a tls block. A converter that emits one Ingress per virtual will name both after the hostname and have the second overwrite the first, so it needs to group virtuals by hostname before it emits anything.

What is easy to migrate and what is not

Virtuals and pools map over more or less mechanically, with the port caveat above.

iRules don’t. An iRule is TCL running in the data path, and there’s no general translation into anything the Ingress API offers. The simple ones that rewrite a path or redirect a host turn into nginx.ingress.kubernetes.io/rewrite-target or a redirect annotation. Anything that inspects a payload, does arithmetic, or calls out somewhere else becomes a decision about where that logic lives now — usually the application or a dedicated proxy — and each one of those is its own small project.

Persistence profiles and custom monitors need the same case by case treatment. Realistically a converter gives you a first draft plus a list of everything it couldn’t express, and that second list is the more useful half.

Cutover difficulties

Nothing in the requirements asks how you actually switch over. Looking back, that is the biggest operational gap in the whole list.

The F5 virtuals sit on one set of addresses and MetalLB hands out another, so moving a service means repointing its DNS record. Sounds like a one line change. It’s the slowest and least reversible step in the whole migration.

The catch is that you don’t get to decide when clients stop using the old answer. A record’s TTL tells resolvers how long they may cache it, and plenty of them round it up, ignore it outright, or keep serving the stale answer while they refresh in the background. Worse, a lot of client libraries resolve once at startup and hold that result for the life of the process, so a long lived connection pool can keep hammering the F5 for days after you changed the record. Until the last straggler moves, both paths are live and you’re running two load balancers in production simultaneously.

Rollback has the same shape. Something breaks ten minutes after cutover, you change the record back, and now you wait all over again — while the clients that already moved stay moved until their own caches expire. A bad cutover costs you TTLs, not seconds.

The usual mitigations help but they don’t fix it:

  • Drop the TTL well ahead of the change, a day or more, so every cache has expired the long value before you touch anything.
  • Move one hostname at a time. Resist doing the whole zone in an afternoon.
  • Watch request rates on both sides and treat the F5 going quiet — not the DNS change — as the signal that a service has finished migrating. Put the TTL back up once it has.

The way to dodge all of this is to not touch DNS at all. If the F5’s existing VIP range can be routed to the cluster, put that range in the MetalLB pool and pin the address on the Ingress controller Service. Cutover becomes an ARP or routing change on a range the F5 no longer advertises, it takes seconds, and rolling back means putting the F5 back on the wire. DNS never moves and no client ever finds out anything happened. BGP mode is cleaner still, since the shift lives in routing policy and you can move traffic gradually.

It needs the network team to agree the range can follow the service, and it needs enough coordination that the old and new never advertise the same address at the same time. I’d still argue for it, because the rollback plan is “turn it off” instead of “wait for the internet to forget”.

What the requirements left out

Three things beyond the cutover.

  • There’s no real L4 answer here. We needed Layer 4 and Layer 7, and what this delivers is L7 with an L4 VIP parked in front of it. The Ingress API can’t express arbitrary TCP or UDP routing at all. ingress-nginx bolts it on through tcp-services and udp-services ConfigMaps, where every entry is a port mapping typed by hand — no per-service object, no RBAC boundary, no way for an app team to own their own line. If any virtual in the estate isn’t HTTP, and in a real estate some always are, you’re hand maintaining a ConfigMap forever. This is the one that eventually changed my mind about the whole approach.
  • The per-VIP model doesn’t survive. Every F5 virtual server gets its own IP. Ingress collapses everything behind one controller VIP and tells them apart by Host header, which quietly breaks anything keyed to an address: per service firewall rules, client allowlists, the partner who whitelisted an IP four years ago and hasn’t thought about it since, and any client that doesn’t send SNI. Finding this out mid-migration is expensive, so go collect the list of who depends on which VIP before you design anything.
  • Nobody mentioned a WAF. The list asks for a load balancer with TLS termination and says nothing about filtering, which usually means the appliance was quietly doing some and nobody wrote it down.

Update, 2026: I would use Gateway API now

I passed on Gateway API at the time for being too new. Wrong call, and not just because it’s since become the obvious default.

The trouble with the Ingress answer was never that NGINX is a bad proxy. It’s that the Ingress API is a poor fit for the F5 object model, and every gap above comes from the same place: nowhere to put L4, backend TLS demoted to an annotation, and one object trying to be both the listener and the routes. Gateway API splits exactly those apart, so the mapping from F5 gets a lot closer:

F5 BIG-IPIngressGateway API
Virtual server, with its own VIP, port and TLS profileImplicit, shared controller ServiceGateway, with addresses, listeners and per listener TLS
PoolServiceService, referenced by backendRefs
Pool memberEndpointEndpoint
HTTP routing, simple iRulesRules plus controller annotationsHTTPRoute, with matching, rewrites and weights as schema fields
Non HTTP virtual, TCP or UDPNot expressible, ConfigMap workaroundTCPRoute, UDPRoute
serverssl profile, re-encryption to the poolbackend-protocol annotationBackendTLSPolicy

Gateway is the piece Ingress never had and the F5 model badly needs. It is the virtual server: it owns an address, a set of listeners with their ports and protocols, and the TLS config for each one, and it belongs to the platform team. HTTPRoute objects attach to it from other namespaces and belong to the app teams. That’s the old “network team owns the VIP, app team owns the routing” split written into the API instead of into a wiki page nobody reads.

A few version details worth pinning down, since they’ve all moved since 2024.

BackendTLSPolicy is GA, and has been in the standard channel since Gateway API v1.4.0 as gateway.networking.k8s.io/v1. It attaches to a Service via targetRefs and carries the CA plus the hostname you expect back:

apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
  name: tls-upstream-auth
spec:
  targetRefs:
    - kind: Service
      name: auth
      group: ""
  validation:
    caCertificateRefs:
      - kind: ConfigMap
        name: auth-cert
        group: ""
    hostname: auth.example.com

That’s the serverssl profile from the SCF file, as a real object with a schema behind it. Exactly what was missing the first time round.

TCPRoute made it into the standard channel in Gateway API v1.6. Before that you needed the experimental CRD bundle, and back in December 2024 it was still v1alpha2 — a fair reason to be cautious then, not one now. On a current bundle the standard channel covers it:

kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.1/standard-install.yaml

Stuck on an older bundle and need TCPRoute? Swap standard-install.yaml for experimental-install.yaml.

For the implementation I’d start with kgateway, a Gateway API control plane for Envoy. Its README dates the project back to 2018, when Solo.io launched it as Gloo; it picked up the new name when it moved to the kgateway-dev org. What makes it interesting here isn’t conformance, it’s that Envoy does real active health checking — probing a backend on a schedule and ejecting it from the load balancing set when it fails — and kgateway hangs that off a Service through BackendConfigPolicy. That’s an F5 monitor, and it’s a far better answer to “customizable backend server monitors” than a Grafana dashboard was. The API is still v1alpha1, so check the field names against the kgateway docs before you write any of it. The same object does TLS origination to the backend, with SNI and SAN verification:

apiVersion: gateway.kgateway.dev/v1alpha1
kind: BackendConfigPolicy
metadata:
  name: backend-tls-policy
spec:
  targetRefs:
    - name: backend-service
      group: ""
      kind: Service
  tls:
    secretRef:
      name: client-tls-secret
    sni: test.example.com
    simpleTLS: true
    verifySubjectAltNames:
      - test.example.com
      - api.example.com

Envoy Gateway and Cilium’s own Gateway API support are the other two candidates. On a cluster already running Cilium, measure that one first — it takes a hop out of the path.

Dropping MetalLB for Cilium BGP

The other thing that changed is the part underneath. MetalLB is gone completely — not reconfigured for BGP mode, removed. The cluster was already running Cilium, and there was never a good reason for a second controller to own the VIPs.

Cilium covers both jobs on its own. LB IPAM hands out the external IPs, and the BGP control plane announces them to the routers. There is no L2 announcement anywhere in the setup now, and no ARP election deciding which node gets the traffic. Every node with a ready endpoint advertises the VIP, and the routers decide. That’s the fix for the single node ceiling from earlier rather than a workaround for it.

Turn the BGP control plane on:

helm upgrade cilium cilium/cilium --namespace kube-system --reuse-values \
  --set bgpControlPlane.enabled=true

Give it a pool to allocate from, which is the IPAddressPool equivalent:

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: ingress-pool
spec:
  blocks:
  - start: "192.168.88.20"
    stop: "192.168.88.30"

Then three objects instead of MetalLB’s two. The peering settings live on their own so several instances can share them:

apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: tor-peer
spec:
  timers:
    holdTimeSeconds: 9
    keepAliveTimeSeconds: 3
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 15
  families:
    - afi: ipv4
      safi: unicast
      advertisements:
        matchLabels:
          advertise: "bgp"

What to announce, and for which services:

apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: ingress-advertisement
  labels:
    advertise: "bgp"
spec:
  advertisements:
    - advertisementType: "Service"
      service:
        addresses:
          - LoadBalancerIP
      selector:
        matchExpressions:
          - { key: bgp, operator: In, values: [ ingress ] }

And which nodes peer with which routers:

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux
  bgpInstances:
  - name: "instance-65000"
    localASN: 65000
    peers:
    - name: "tor1"
      peerASN: 65001
      peerAddress: 192.168.88.1
      peerConfigRef:
        name: "tor-peer"
    - name: "tor2"
      peerASN: 65001
      peerAddress: 192.168.88.2
      peerConfigRef:
        name: "tor-peer"

Label the Ingress controller’s Service with bgp: ingress so the advertisement selector picks it up, and every node running a ready endpoint starts announcing the VIP. The routers see the same prefix from several next hops, install an ECMP route, and hash flows across them. The single node front door is gone, and the ceiling becomes the sum of the NICs actually participating rather than whichever node won an ARP election.

Two things to keep in mind. gracefulRestart matters more than it looks — without it, restarting an agent for an unrelated reason withdraws the route and drops traffic. And peering from every node means the routers hold more sessions, so check what your ToR switches are sized for before rolling it out fleet-wide. cilium bgp peers and cilium bgp routes are what you’ll be staring at while this comes up.

Where that leaves the L4 requirement

Worth joining up, because “Layer 4” was really two problems wearing one label.

The first was routing: a non-HTTP virtual had nowhere to go, since Ingress can’t express it and the tcp-services ConfigMap isn’t an API. TCPRoute and UDPRoute fix that. They attach to a Gateway with a TCP or UDP listener the same way an HTTPRoute does, so a non-HTTP service gets a real object that an app team can own.

The second was delivery: whatever the VIP belonged to, every packet for it still arrived on one node. That’s what BGP and ECMP fix, and it applies to a TCP listener exactly as it does to an HTTPS one.

Put together, an F5 virtual that was never HTTP in the first place now maps to a Gateway with a TCP listener, a TCPRoute pointing at the backend, and an address from LB IPAM announced from every node running a ready endpoint. That’s much closer to what the appliance was actually doing than anything in the original design.

It also brings back the per-VIP model. A Gateway can carry its own addresses, so instead of collapsing everything behind one controller VIP and splitting on Host header, you can give a service its own IP again — which matters for the firewall rules and client allowlists that were keyed to those addresses.

One thing to watch on Cilium: the TCPRoute and UDPRoute CRDs are optional. If you don’t install them, Cilium disables support for those routes rather than complaining, so you get a Gateway that quietly ignores half of what you meant.

None of that contradicts the analysis from 2024, it’s the conclusion of it. L2 announcement is a single node front door whoever implements it, so the fix was never to swap MetalLB for something else in L2 mode — it was to stop announcing at layer 2. BGP with ECMP is still the only real answer past one NIC.

And the converter still has to reconcile ports against the live cluster, because those numbers were never in the SCF file to begin with.

The one thing worth keeping from the original design is the reason I picked Ingress at all: ingressClassName let me move a service over one line at a time, and move it straight back. Gateway API keeps that and makes it explicit, since an HTTPRoute names its own parentRefs — moving a service from a legacy Gateway to a new one is a one field edit on one object. Right instinct. Wrong API.