Static NodePorts for Envoy Gateway

Today, I added a new TLS Secret reference to my Gateway object, instructing Envoy Gateway to use yet another certificiate for incoming connections. After applying the changes, I noticed that none of the HTTPRoutes could be served anymore, resulting in a connection timeout and a complete loss of connectivity for the cluster.

Turns out that when you alter the Gateway object, the corresponding Service might get updated in-place or recreated as well, resulting in a change of the used NodePorts. This does of course only apply, if you’re not using a LoadBalancer which is usually the case if you run Kubernetes bare metal or on a non-hyperscaler environment.

This is normal Kubernetes behaviour, because NodePorts are randomly assigned, unless explicity specified. As I didn’t expect the Service to change or to be recreated, I didn’t specify the NodePorts explicitly, resulting in the observed issue.

To resolve that problem, we need to make sure that the NodePorts are fixed. However, we don’t create the Service object ourselves. The Service is automatically generated by Envoy Gateway depending on the instructions in the Gateway object and its decendent EnvoyProxy objects. Fortunately, there’s a patch field allowing us to hook into the Service manifest that is going to be used to create it.

I already used an object of type EnvoyProxy when enabling the PROXY Protocol for Envoy Gateway. We can now extend said object with our NodePort instructions.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: envoy-proxy-config
  namespace: envoy-gateway-system
spec:
  logging:
    level:
      default: warn
  provider:
    type: Kubernetes
    kubernetes:
      envoyDaemonSet: {}
      envoyService:
        type: NodePort
        externalTrafficPolicy: Local
        patch:
          type: StrategicMerge
          value:
            spec:
              ports:
              - name: http
                nodePort: 30985
                port: 80
              - name: https
                nodePort: 30976
                port: 443

The highlighted part of the EnvoyProxy manifest is the patch instruction that is being applied over the Service object that is being created for the Gateway. It instructs the Service of type NodePort to listen on specific ports for HTTP and HTTPS traffic.

Changing the underyling Gateway object, will therefore no longer cause an accidental change of the NodePorts which would usually result in a complete loss of connectivity for all downstream HTTPRoutes.