Reach any internal load balancer from a Mac through an IAP bastion

Every internal load balancer on Google Cloud, Application or Network, regional or cross-region, gets an address only inside your VPC network. From a laptop you reach it through a VM in that network: Identity-Aware Proxy carries SSH to the VM, and an SSH port forward carries your traffic from the VM to the load balancer. The same forward works whatever sits behind it: instance groups, a GKE Gateway, Ingress or Service, or Cloud Run and Cloud Run functions through a serverless NEG. This guide sets it up and covers what differs for HTTPS services: host names, the port in the Host header and TLS.

By Ostgate · Updated

Internal load balancer types and what they give you

Google Cloud has three families of internal load balancers. Each forwarding rule has an internal IPv4 address from a subnet of the VPC network, and that address and port are what you forward to. The facts below are from Google's load-balancing documentation.

Load balancerProtocolsPorts per rule
Internal Application Load Balancer, regional or cross-region HTTP, HTTPS, HTTP/2 One
Internal proxy Network Load Balancer, regional or cross-region TCP One
Internal passthrough Network Load Balancer TCP, UDP, ICMP and others Up to five, or all

The Application Load Balancer and the proxy Network Load Balancer are proxies: they end the client's connection and open their own to the backends from a proxy-only subnet. The passthrough Network Load Balancer is not a proxy: it delivers the client's packets to a backend with the client's source address. An internal Application Load Balancer requires HTTP/1.1 or later from clients; Google's documentation states that HTTP 1.0 is not supported.

Clients reach any of them from the same VPC network or from a network connected to it through VPC Network Peering, Cloud VPN or Cloud Interconnect. A Mac on the internet is none of those, so it needs a hop inside.

Why IAP cannot reach the load balancer directly

gcloud compute start-iap-tunnel connects to a port on a VM instance. A load balancer's forwarding rule is not an instance, so there is nothing to name in that command. What you can reach is any VM on the same network, and that VM can reach the load balancer. So the path is: your Mac, IAP, a bastion VM, the load balancer's internal address.

Where the bastion VM must be

The bastion must be in the load balancer's VPC network, or in a network peered with it. The region matters for regional load balancers:

  • Regional internal Application Load Balancer, regional internal proxy Network Load Balancer, internal passthrough Network Load Balancer: clients must be in the load balancer's region by default. With global access enabled on the forwarding rule (--allow-global-access), clients from any region of the network may connect.
  • Cross-region internal Application Load Balancer and cross-region internal proxy Network Load Balancer: clients from any region of the network may connect by default.
  • Regional internal Application Load Balancer with a serverless NEG: Google's setup guide states that the serverless NEG, the load balancer and any client VMs must be in the same region as the Cloud Run service.

A bastion in the load balancer's region satisfies every case, so pick one there.

Forward a local port through a bastion VM

The bastion needs no external IP. IAP needs a firewall rule from 35.235.240.0/20 on tcp:22 to the VM, roles/iap.tunnelResourceAccessor for you, and SSH access through OS Login or a metadata key; the full list is in the IAP firewall and IAM checklist. If the load balancer's address is 10.10.0.5 and it serves HTTPS on 443:

gcloud compute ssh bastion-1 --tunnel-through-iap \
    --zone=europe-west1-b --project=acme-prod \
    -- -N -L 8443:10.10.0.5:443

gcloud compute ssh runs OpenSSH with IAP as its proxy command, and everything after -- goes to ssh: -N runs no remote command, -L listens on 127.0.0.1:8443 on your Mac and opens a connection from the bastion to 10.10.0.5:443 for each client. Several -L options can share one session. The load balancer now answers on 127.0.0.1:8443.

If you use plain ssh rather than gcloud compute ssh, the same forward goes into ~/.ssh/config with gcloud's IAP tunnel as the proxy command:

Host bastion-1
    HostName bastion-1
    User alice
    ProxyCommand gcloud compute start-iap-tunnel bastion-1 %p --listen-on-stdin --zone=europe-west1-b --project=acme-prod
    LocalForward 8443 10.10.0.5:443

Then ssh -N bastion-1 opens the forward. Plain ssh needs a key that OS Login or the VM's metadata already accepts; gcloud compute ssh publishes one for you.

Host names and TLS for HTTPS services

Opening https://127.0.0.1:8443 usually fails, for two reasons:

  • The load balancer presents a certificate for the service's name, such as api.internal.example.com, not for 127.0.0.1. Google's documentation says the load balancer can use the SNI host name to pick the certificate when its target proxy has more than one, and falls back to the first certificate when none matches.
  • An internal Application Load Balancer picks the backend through its URL map, whose host rules match the request's Host header; the load balancer preserves the client's Host header. A request for 127.0.0.1 matches no host rule and goes to the default service.

Both are fixed by sending the real name to your local port. For a test from the terminal, curl's --connect-to sends SNI and Host for the real name on its default port, and only redirects the TCP connection to your forward:

curl --connect-to api.internal.example.com:443:127.0.0.1:8443 \
    https://api.internal.example.com/

--resolve api.internal.example.com:8443:127.0.0.1 with https://api.internal.example.com:8443/ also works, but it puts the port in the Host header, the same as a browser does (next paragraph).

For a browser, map the name to your Mac in /etc/hosts:

127.0.0.1   api.internal.example.com

Then open https://api.internal.example.com:8443. The certificate matches and the traffic goes through your forward. Remove the line when you are done: while it is there, this Mac sends that name to 127.0.0.1 whether the forward runs or not.

The port in the Host header. A browser sends Host: api.internal.example.com:8443 when the URL has a port. Google's URL map documentation says the classic Application Load Balancer ignores the port when matching a host rule, while other Application Load Balancers can match it, for example a host rule of example.net:8080. If browser requests by name land on the default service while curl with --connect-to reaches the right one, the port is the difference: add the host with that port to the host rule, or use a local port equal to the load balancer's port.

Private certificate authorities. If the load balancer's certificate is issued by a private CA, the Mac must trust that CA whether or not you use a tunnel. You set that trust in Keychain Access.

HTTP-only load balancers. An internal Application Load Balancer with an HTTP target proxy has no certificate, so only the host rules matter: forward to port 80 and send the name the same way, with http://.

TCP services behind a Network Load Balancer

A database, Redis or a message broker behind an internal proxy or passthrough Network Load Balancer needs no names: forward the port and point the client at 127.0.0.1. For Redis on 10.10.0.5:6379:

gcloud compute ssh bastion-1 --tunnel-through-iap \
    --zone=europe-west1-b --project=acme-prod \
    -- -N -L 6379:10.10.0.5:6379

Behind a passthrough Network Load Balancer the backends see the bastion's internal address as the client, so their firewall must admit it. Behind a proxy Network Load Balancer they see addresses from the proxy-only subnet instead. SSH port forwarding carries TCP only, so UDP services behind a passthrough Network Load Balancer cannot be reached this way.

Many host names: a SOCKS proxy

When several services share one internal domain, one forward per name gets tedious. OpenSSH's -D starts a SOCKS proxy on your Mac instead, and the bastion opens each connection to whatever the client names:

gcloud compute ssh bastion-1 --tunnel-through-iap \
    --zone=europe-west1-b --project=acme-prod \
    -- -N -D 1080
curl --socks5-hostname 127.0.0.1:1080 https://api.internal.example.com/

With --socks5-hostname the bastion resolves the name, so the name must resolve inside the VPC, for example in a Cloud DNS private zone, and the URL needs no local port. A browser can use the same proxy if it sends host names through SOCKS5 instead of resolving them locally.

What sits behind the load balancer

The client side, your Mac, the bastion and the forward, is the same for every backend type. What differs is where you find the address and what the backend side must allow.

Managed and unmanaged instance groups

Health checks and backend firewall rules are set on the backend side. Google's health check documentation lists 35.191.0.0/16 as the IPv4 probe range for internal Application, proxy Network and passthrough Network Load Balancers. The backends of an internal Application Load Balancer or proxy Network Load Balancer must also accept traffic from the proxy-only subnet. If the backends fail their health checks, the fix belongs on the backend side.

GKE: internal Gateway, Ingress and LoadBalancer Service

  • Gateway: the GatewayClass gke-l7-rilb creates a regional internal Application Load Balancer; gke-l7-rilb-mc and gke-l7-cross-regional-internal-managed-mc are the multi-cluster and cross-region multi-cluster variants. The address is in the Gateway's status.
  • Ingress: the annotation kubernetes.io/ingress.class: "gce-internal" creates a regional internal Application Load Balancer.
  • Service: type: LoadBalancer with networking.gke.io/load-balancer-type: "Internal" creates an internal passthrough Network Load Balancer. networking.gke.io/internal-load-balancer-allow-global-access: "true" enables global access on its forwarding rule.

Forward to the address GKE reports and the port the Gateway, Ingress or Service exposes, not to a Pod or node port. For a Gateway or Ingress, send the host names from its routes or rules, as described above.

To run kubectl against a private cluster's control plane through the same kind of bastion, see kubectl to a private GKE cluster from a Mac.

Serverless NEGs

Google's serverless NEG documentation states that serverless NEGs used with regional or cross-region internal Application Load Balancers can point only to Cloud Run or Cloud Run functions (2nd gen) resources; internal Application Load Balancers do not support Cloud Functions 1st gen or App Engine. The client talks to the load balancer's address and to the host name in its certificate, never to the service's run.app URL. The next section covers the Cloud Run side.

Cloud Run and Cloud Run functions behind an internal load balancer

Cloud Run's ingress setting decides which requests the service accepts. Its values are all, internal and internal-and-cloud-load-balancing. Requests through an internal Application Load Balancer count as internal, so both restricted settings accept them. Google's ingress documentation also states that IAM authentication still applies to requests on every ingress path: a service that requires the Cloud Run Invoker role answers 403 to a request without an identity token, load balancer or not.

From the Mac this is the HTTPS case above: forward to the load balancer's address on 443 and send the load balancer's host name. For a regional internal Application Load Balancer, put the bastion in the service's region.

A service with internal ingress and no load balancer in front is reached at its run.app URL from inside the VPC, which needs Private Google Access or a Private Service Connect endpoint on the bastion's subnet. That route is covered in internal Cloud Run services and functions from a Mac.

How Ostgate does it

Ostgate is a native macOS app that sets up the same path without gcloud.

  • Forward profiles (Connections › Forwards › New Forward Profile…) carry several ports through a bastion VM you pick by project and instance, over one SSH session through IAP, to any address the VM can reach. The HTTPS / Load balancer target fills in port 443; Custom (host + port) takes any TCP port, such as Redis on 6379. The bastion's host key is verified before the forward starts.
  • Each forward can have a Host alias. With Manage /etc/hosts aliases while running on, the profile adds each alias to /etc/hosts on start and removes it on stop. macOS asks for an administrator's password for that unless the Ostgate port helper is installed and your macOS user is an administrator. For an internal load balancer with its own host name, that is the forward to 10.10.0.5:443 plus the alias api.internal.example.com. On a local port such as 8443 the alias points to 127.0.0.1, the browser URL carries the port, and the Host header note above applies. The forward can also listen on 443 itself, on the alias's own loopback address, so the URL needs no port: see Forward ports 443 and 22 to your Mac.
  • Each profile's listeners admit Ostgate only, My processes (apps running as your macOS user) or Any local process, which asks for confirmation. Listeners bind 127.0.0.1, or for a port below 1024 the alias's own address between 127.0.0.2 and 127.0.0.254.
  • Cloud Run › Open Proxy… and Open in Browser reach a Cloud Run service with internal ingress at its run.app URL through a VM of the service's project, with a local HTTP proxy that speaks TLS to the service under its own name. This proxy accepts only *.run.app hosts, so it does not serve a load balancer's own host name; use a forward profile for that. Services that require IAM authentication cannot be opened this way yet.
Ostgate's New Forward Profile sheet over the Connections view: profile orders-db, bastion VM example-bastion in project acme-prod, zone europe-west1-b, and a PostgreSQL forward to remote host 10.0.8.3 port 5432 on local port 5432, with an optional host alias.
New Forward Profile through a bastion VM

When it does not connect

SymptomCheck
The SSH hop to the bastion fails Firewall from 35.235.240.0/20 on tcp:22, the IAP tunnel role, and OS Login or metadata key access to the VM.
Connection refused on 127.0.0.1 Nothing listens on that local port: the forward is not running, or uses another port.
The forward opens, requests time out or close at once The bastion is in another region without global access, or on a network that is not peered; ssh prints open failed when the bastion cannot connect.
502 or 503 from the load balancer The proxy cannot reach a healthy backend: health checks from 35.191.0.0/16 and traffic from the proxy-only subnet must be allowed to the backends. A 503 can also appear briefly after a configuration change.
TLS name mismatch You are browsing by 127.0.0.1; send the service's name with /etc/hosts or --connect-to.
404 or the wrong service answers The URL map routes by Host header; send the name, and mind the port in it.
403 from Cloud Run The service requires IAM authentication and the request has no identity token.

The bastion pattern for databases is in Cloud SQL private IP from a Mac, and single ports on a VM in port forwarding over IAP.

Questions

Can IAP TCP forwarding connect to the load balancer directly?

Not with a plain start-iap-tunnel: it connects to a port on a VM instance. To reach a load balancer's address, go through a VM that can route to it.

Does the bastion VM need a public IP?

No. IAP reaches the VM's internal address, so it needs only a firewall rule from 35.235.240.0/20 on tcp:22 and SSH access for you.

Which region should the bastion VM be in?

The load balancer's region. Regional internal load balancers accept clients from their own region only, unless global access is enabled on the forwarding rule; cross-region internal load balancers accept clients from any region of the network.

Why does https://127.0.0.1:8443 show a certificate warning?

The load balancer presents a certificate for the service's host name, and an internal Application Load Balancer routes by that name. Map the name to 127.0.0.1 in /etc/hosts and browse by the name instead.

Does Cloud Run behind an internal load balancer still check IAM?

Yes. Google's ingress documentation states that IAM authentication still applies on every ingress path, so a service that requires the invoker role needs an identity token in each request.

Internal services in your browser. Ostgate runs on Apple Silicon Macs with macOS 15 or later, with a 7-day trial and no sign-up. Setup is in the tunnels and forwards docs.