GKE Ingress shows unhealthy backends while pods are ready

The test

On 10 November 2026 we ran this on an Autopilot cluster, GKE 1.36.4-gke.2377000 in europe-central2. One nginx Deployment listens on port 80 and has a TCP readiness probe on port 80. Two Services select the same pods:

• t8-lb-ok: port 80 to targetPort 80
• t8-lb-broken: port 80 to targetPort 8081, where nothing listens

One Ingress with no class sends /broken to t8-lb-broken and everything else to t8-lb-ok. We applied the manifests at 13:57:44Z; the cluster events show the forwarding rule and its IP address at 14:00:32Z.

Find the unhealthy backend with kubectl and gcloud

Start with the Ingress:

kubectl -n ostgate-t describe ingress t8-lb

Default backend:  t8-lb-ok:80 (10.100.1.59:80,10.100.1.60:80)
  *           /broken   t8-lb-broken:80 (10.100.1.59:8081,10.100.1.60:8081)
Annotations:  ingress.kubernetes.io/backends:
                {"k8s1-93fffa97-ostgate-t-t8-lb-broken-80-d64d9d48":"UNHEALTHY","k8s1-93fffa97-ostgate-t-t8-lb-ok-80-be5db818":"HEALTHY"}

The ingress.kubernetes.io/backends annotation gives Google's verdict per backend service, and each name carries the namespace, the Service and its port. t8-lb-broken:80 sends to both pods on 8081. For each endpoint, ask Compute Engine:

gcloud compute backend-services get-health \
  k8s1-93fffa97-ostgate-t-t8-lb-broken-80-d64d9d48 \
  --global --project PROJECT_ID

Both backend services, trimmed to state, IP address and port:

k8s1-93fffa97-ostgate-t-t8-lb-broken-80-d64d9d48: UNHEALTHY;UNHEALTHY 10.100.1.59;10.100.1.60 8081;8081
k8s1-93fffa97-ostgate-t-t8-lb-ok-80-be5db818:     HEALTHY;HEALTHY     10.100.1.59;10.100.1.60 80;80

Endpoints “marked as UNHEALTHY are either failing the configured health-check or not responding to it”, and backend services without a valid global forwarding rule “will not be health checked” (get-health reference), so a brand new Ingress shows no health until its forwarding rule exists. Match the IPs to pods with kubectl get pods -o wide.

We sent three requests to each path. / got 200 every time. /broken got 502 every time, with the page title “502 Server Error” and the text “The server encountered a temporary error and could not complete your request. Please try again in 30 seconds.” That page comes from the load balancer, not from nginx.

Why Ready pods fail Google's health check

The load balancer's health check “is different from a Kubernetes liveness or readiness probe” (container-native load balancing). The Ingress controller builds it from the pods' readiness probe only when it can. The request path comes from containers[].readinessProbe.httpGet.path, the default path is /, and two conditions apply to the port: “The readiness probe's port number must match the serving Pod's containers[].spec.ports.containerPort” and “The serving Pod's containerPort matches the Service's targetPort” (Ingress routing and security).

Our probe was TCP, so the controller had no HTTP probe to copy, and the broken Service's targetPort matched no container port.

GKE sets the NEG readiness gate, cloud.google.com/load-balancer-neg-ready, to True when “The backend service has a successful load balancer health check”, when the health check times out, or when “No load balancer health check data is available”, per the same routing page. If it never turns True, “the rollout stalls after attempting to create one new Pod”.

Fix the health check

Make targetPort match the container port

When the pods are Ready and every endpoint is UNHEALTHY, compare the port in the describe ingress endpoint list with the port your container listens on. Our t8-lb-ok Service, identical apart from targetPort: 80, was HEALTHY on the same pods:

ports:
- port: 80
  targetPort: 80

Give the health check a path that answers 200

The default health check asks for /. If your app answers something else there, add an HTTP readiness probe with the right path and port before you create the Ingress, or set the health check yourself. The probe is read once: the Pods must exist when the Ingress is created, and a later change to readinessProbe does not reach the health check (GKE Ingress overview). After a probe change, redeploy the Pods and the Ingress (Ingress health check troubleshooting).

A BackendConfig takes precedence over the readiness probe: “If the Service references a BackendConfig CRD with healthCheck information, GKE uses that to create the health check”. The fields and the Service annotation, from Ingress configuration:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: web-hc
spec:
  healthCheck:
    type: HTTP
    requestPath: /healthz
    port: 8080

metadata:
  annotations:
    cloud.google.com/backend-config: '{"default": "web-hc"}'

type must be HTTP, HTTPS or HTTP2, and port defaults to 80 when you leave it out.

Gateway: HealthCheckPolicy

A Gateway has no BackendConfig. “You can use a HealthCheckPolicy to control the load balancer health check settings” (Gateway policies). Without one, the request path “defaults to /” and the port specification defaults to USE_SERVING_PORT. The policy sits in the Service's namespace and points at the Service:

apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
  name: web-hc
  namespace: acme
spec:
  default:
    config:
      type: HTTP
      httpHealthCheck:
        portSpecification: USE_SERVING_PORT
        requestPath: /healthz
  targetRef:
    group: ""
    kind: Service
    name: web

The same get-health command reads the Gateway's backend services.

Firewall rules for Google's health checkers

For an external Ingress, the controller creates the firewall rules for the load balancer's proxies and health checkers itself, allowing 130.211.0.0/22 and 35.191.0.0/16 (GKE Ingress overview). The health check overview currently lists 35.191.0.0/16 and, for IPv6, 2600:2d00:1:b029::/64 as probe ranges for global external and classic Application Load Balancers.

If the service project's GKE service account may not manage firewall rules in a Shared VPC host project, the controller cannot create them. The Ingress events then carry “Firewall change required by security admin:” and the rule to create, with --source-ranges 130.211.0.0/22,35.191.0.0/16. An internal Ingress needs a rule for its proxy-only subnet in any VPC, because the controller “cannot create ingress allow firewall rules” for it.

How Ostgate does it

Ostgate's GKE inspector joins endpoint ports, health per IP and pod names in a Traffic path block for a Service with NEGs, an Ingress or a Gateway. It reads the forwarding rules, URL map, backend services and NEG health from Compute Engine with the Google account the cluster is opened with, without gcloud.

1. In GKE, open the cluster and select the Service, Ingress or Gateway.
2. Read the Traffic path block in the inspector, top to bottom.
3. Refresh Traffic Path reads Compute Engine again; the line under the block says when it last read.

For t8-lb-broken in the test above, the block showed this (the load balancer is deleted, so its address is swapped for a documentation one):

Forwarding rule  203.0.113.10:80  HTTP · EXTERNAL
  URL map
Service  ostgate-t/t8-lb-broken  :80 → 8081
  Backend service  Cloud Armor: none
    NEG  europe-central2-a · 0 of 2 healthy
      10.100.1.59:8081  Ready · UNHEALTHY
      10.100.1.60:8081  Ready · UNHEALTHY

t8-lb-ok showed the same pods on :80, 2 of 2 healthy, each Ready · HEALTHY, so the Service row's → 8081 is where to look. Without permission to read the load balancer, the block keeps the Kubernetes rows and says which read failed, with Compute Viewer (roles/compute.viewer) as the role to ask for. The GKE docs cover the inspector, and this guide reaches an internal Ingress or Gateway from a Mac.

Symptoms and causes

Questions

Why does a GKE Ingress show UNHEALTHY backends when the pods are Ready?

The readiness probe and Google's load balancer health check are separate checks. The health check runs from outside the cluster against each NEG endpoint, the pod IP on the Service's targetPort. In our test the readiness probe passed on port 80 while the Service's targetPort was 8081, where nothing listened, so both pods were Ready and UNHEALTHY at the same time.

How do I see which pod the load balancer marks unhealthy?

kubectl describe ingress shows the ingress.kubernetes.io/backends annotation with HEALTHY or UNHEALTHY for each backend service. Run gcloud compute backend-services get-health with that name and --global to get the state of every endpoint IP and port, then match the IPs to pods with kubectl get pods -o wide.

I changed the readinessProbe. Why didn't the Ingress health check change?

GKE copies the readiness probe into the health check when it creates the Ingress backend, and only if the Pods exist at that moment. Google's docs say GKE does not keep the two synchronized. Redeploy the Pods and the Ingress, or set the health check in a BackendConfig.

Which source ranges must the firewall allow for GKE Ingress health checks?

Google's GKE Ingress page names 130.211.0.0/22 and 35.191.0.0/16, and the Ingress controller creates those firewall rules itself. In a Shared VPC where the controller cannot manage firewall rules in the host project, it puts the rule it needs in an Ingress event, and an administrator of the host project creates it.