kubectl through the GKE Connect gateway

The test

We ran the commands below on 11 November 2026 from a Mac with kubectl 1.35.0 and Google Cloud SDK 590.0.0 (Homebrew cask gcloud-cli). The cluster is a Standard cluster on GKE 1.36.4-gke.2377000 in europe-central2-a, registered to its project's fleet with membership location europe-central2. Its public endpoint is off, the private endpoint is enforced, and the DNS endpoint has allowExternalTraffic: false, so kubectl on the Mac can't reach the control plane directly. The account holds the project's Owner role.

The tests with limited roles are from 2 and 6 November 2026 on the same cluster, with a second account and a temporary service account. We didn't repeat them, and we changed no IAM on the project for this page.

Is the Connect gateway free?

Yes. Google's release note of 2 September 2025 says “Features that were part of GKE Enterprise are now available as part of the standard GKE offering”, and Connect Gateway is in its list of features “included in the GKE offering at no additional cost” (GKE release notes). Posts that call it a paid Enterprise feature were written before that date.

The GKE pricing page says GKE includes “multi-cluster management features, like Fleets, Teams, Config Management, Policy Controller, and Fleet dashboard, at no extra cost”. Registering a cluster to a fleet doesn't waive the flat “cluster management fee of $0.10 per cluster per hour”, which “applies to all GKE clusters”. The same page prices Multi Cluster Gateway at $3 per backend pod per month; that is the multi-cluster load balancer, a different product from the Connect gateway.

Register the cluster to a fleet

“Only clusters registered to your project fleet can be accessed through the Connect gateway”, and GKE clusters on Google Cloud “must be registered separately” (Set up the Connect gateway). Enable the APIs the setup page lists, then register:

gcloud services enable --project=acme-prod \
    connectgateway.googleapis.com gkeconnect.googleapis.com \
    gkehub.googleapis.com cloudresourcemanager.googleapis.com
gcloud container clusters update acme-gke --enable-fleet \
    --location=europe-central2-a --project=acme-prod
gcloud container fleet memberships list --project=acme-prod

--enable-fleet registers the cluster to its own project's fleet, and the cluster name becomes the membership name. The gcloud CLI gives the membership “a regional membership location by default”, and --fleet-project registers it to a fleet in another project (Register a cluster to your fleet). In the Console it is Register in the cluster's Actions menu on the GKE Clusters page. We first registered our cluster with --enable-fleet on 2 November 2026 (exit 0, membership READY); the membership we used on 11 November was created on 9 November, and the project had the gkehub, gkeconnect and connectgateway APIs on:

NAME           UNIQUE_ID                             LOCATION
ostgate-t-gke  0e24d0e5-c21e-4943-9ee8-b80958eda991  europe-central2

Get a gateway kubeconfig

gcloud container fleet memberships get-credentials acme-gke \
    --location=europe-central2 --project=acme-prod
kubectl get namespaces

The command adds a context named connectgateway_acme-prod_europe-central2_acme-gke. Its server is a Google URL, not the cluster:

https://europe-central2-connectgateway.googleapis.com/v1/projects/123456789012/locations/europe-central2/gkeMemberships/acme-gke

The entry carries no certificate authority data, because kubectl talks to a Google API with a public certificate. Its user runs the bare command name gke-gcloud-auth-plugin. On our Mac that failed at once:

Unable to connect to the server: getting credentials: exec: executable gke-gcloud-auth-plugin not found

It looks like you are trying to use a client-go credential plugin that is not installed.

The plugin was installed. The Homebrew cask keeps it in the SDK's own bin folder, which is not on PATH, and gcloud container clusters get-credentials writes the plugin's full path where the fleet command writes only the name. Add the folder to PATH:

export PATH="$(gcloud info --format='value(installation.sdk_root)')/bin:$PATH"

Then kubectl get ns listed the cluster's 12 namespaces, kubectl version reported Server Version: v1.36.4-gke.2377000, and kubectl auth whoami returned the Google account with the group system:authenticated. Making the fix permanent, and apps started from the Dock that miss it, are in gke-gcloud-auth-plugin on a Mac.

To use a service account instead of your own account, Google sets gcloud config set auth/impersonate_service_account SA_EMAIL before get-credentials (Using the Connect gateway). We didn't test impersonation.

Roles: gatewayReader, gatewayEditor, gatewayAdmin

The gateway checks IAM on the fleet host project before a request reaches the cluster. The three predefined roles, read with gcloud iam roles describe on 11 November 2026 (each also holds the same 11 serviceusage permissions):

Google's setup page says a user needs one of the gateway roles “unless the user has roles/owner in the project”, offers the reader for “read-only access” and the editor for “read / write access”, and adds roles/gkehub.viewer to “retrieve cluster kubeconfigs” (Set up the Connect gateway). Grant them on the fleet host project:

gcloud projects add-iam-policy-binding acme-prod \
    --member=user:dana@example.com --role=roles/gkehub.gatewayReader

The gateway role only lets the request through. What the identity may do inside the cluster is still up to Kubernetes: RBAC, or on GKE the project's Kubernetes Engine IAM roles. In our tests an account with the project's Owner role needed no RBAC binding. On clusters outside Google Cloud the Connect agent also needs an impersonation policy, which gcloud container fleet memberships generate-gateway-rbac writes; GKE clusters on Google Cloud don't use the agent.

You don't need the whole admin role for streams. On 6 November 2026 a service account with a custom role made of the reader's three gkehub permissions plus gkehub.gateway.stream ran kubectl port-forward through the gateway, and a request through the forward got 200:

gcloud iam roles create gatewayStream --project=acme-prod \
    --title="Connect gateway with streams" \
    --permissions=gkehub.gateway.generateCredentials,gkehub.gateway.get,gkehub.memberships.get,gkehub.gateway.stream

That role lacks the editor's post, put, patch and delete; we didn't try a write with it. It still allows exec, attach and cp, so give the identity RBAC only for the subresources it should use.

exec, cp, attach and port-forward through the gateway

They work today. Google's usage page lists the requirements: “Clusters must be at version 1.30 or later for the attach, cp, and exec commands, and at version 1.31 or later for the port-forward command. The kubectl client must be at version 1.31 or later.” The identity needs gkehub.gateway.stream and a Role with pods/exec, pods/attach and pods/portforward (Using the Connect gateway). Older posts that say exec doesn't work through the gateway are out of date. Google's setup page still links to it as “Preview support for the commands”, while the usage page names no launch stage.

Our test pod runs a distroless image, so the first exec with /bin/sh passed the gateway and failed in the container runtime:

error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "7ad614c5…": OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory

Images without a shell need kubectl debug, covered in kubectl exec forbidden on GKE. We ran the image's own binary instead. It tried to bind the port its running copy already holds and exited, and its output and exit code came back through the gateway:

kubectl exec -n ostgate-t echo-58b48dd557-nl5vd -v=6 -- /hello-app

"Response" verb="GET" url="https://europe-central2-connectgateway.googleapis.com/v1/projects/123456789012/locations/europe-central2/gkeMemberships/ostgate-t-gke/api/v1/namespaces/ostgate-t/pods/echo-58b48dd557-nl5vd/exec?command=%2Fhello-app&container=echo&stderr=true&stdout=true" status="101 Switching Protocols" milliseconds=215
The subprotocol is v5.channel.k8s.io
2026/11/11 13:45:03 Server listening on port 8080
2026/11/11 13:45:03 listen tcp :8080: bind: address already in use
command terminated with exit code 1

The pod stayed Running with 0 restarts. port-forward through the same gateway answered 200 in our tests of 2 and 6 November; how long such a forward lasts and how to keep one open is in Port-forward to a GKE Service that survives pod restarts.

Without gkehub.gateway.stream the error hides the cause. A service account with roles/gkehub.gatewayReader and RBAC for the forward could run kubectl get pods, and its kubectl port-forward printed error: error upgrading connection: unable to upgrade connection: followed by a Google HTML page, Error 400 (Bad Request)!!1. With -v=4 or higher the line before it holds the refusal:

fallback to secondary dialer from primary dialer err: unable to upgrade streaming request: websocket: bad handshake (403 Forbidden): Permission 'gkehub.gateway.stream' denied on resource 'projects/…/gkeMemberships/ostgate-t-gke'

Google describes the same thing: the error “is often a generic 400 Bad Request”, and with kubectl 1.32 or later at verbosity 4 you look for RemoteCommand fallback: after exec, cp or attach, and for fallback to secondary dialer from primary dialer err: after port-forward. If that line reads failed to connect to the cluster's API Server with response (status=403 Forbidden, the gateway let you through and RBAC refused.

The 30-minute stream limit and the 10-stream quota

For the error generic::failed_precondition: connection to Agent failed/terminated, Google writes: “If you encounter this error after the session is active for about 20 to 30 minutes, it is an expected limitation due to security reasons. The connection needs to be re-established.” A shell, a long cp and a forward are all such sessions. Our port-forward through the gateway answered for 1798 seconds, then kubectl exited with error: lost connection to pod.

“There is a quota limit of 10 active streams per fleet host project”, the connectgateway.googleapis.com/active_streams quota, and the error when it runs out is Gateway's active_streams quota exhausted (Using the Connect gateway). The quota counts the fleet host project, so a team that shares one fleet shares the 10 streams. We didn't run into it.

Forbidden errors that name no permission

An account with only roles/container.clusterViewer and no gateway role got this from kubectl on 2 November 2026:

Error from server (Forbidden): unknown (get pods)

No permission, no resource name. The gateway's HTTP answer, read with a raw client, was a 403 whose JSON body named it:

"message": "Permission 'gkehub.gateway.get' denied on resource 'projects/…/gkeMemberships/ostgate-t-gke' (or it may not exist)",
"status": "PERMISSION_DENIED"

So a Forbidden that says unknown through the gateway means a missing gateway role on the fleet host project, and RBAC changes won't fix it. A Forbidden from RBAC names the user, the verb and the resource, as in kubectl exec forbidden on GKE. A yes from kubectl auth can-i comes from the cluster's authorizer and doesn't cover the gateway's IAM check. Google's troubleshooting suggests verbose mode (-v 10) for another gateway error; we didn't check whether it prints this body.

Tools that build the gateway URL themselves

Argo CD cluster secrets, CI jobs and other clients that don't run get-credentials need the server URL written out. Many examples use https://connectgateway.googleapis.com/v1/projects/NUMBER/locations/global/gkeMemberships/NAME, which fits a membership whose location is global, as in a Stack Overflow question about Argo CD and fleet clusters with no accepted answer; one answer there suggests /memberships/ instead. Our membership is regional. With the same credentials we tried five URLs:

The failures read Error from server (NotFound): the server could not find the requested resource. The location in the URL has to be the membership's location from gcloud container fleet memberships list, and the path segment is gkeMemberships. Copy the server from a kubeconfig that get-credentials wrote rather than building it: Google's troubleshooting says “Don't manually edit the kubeconfig file”. The client's identity needs a gateway role on the fleet host project, which is not necessarily the project the tool runs in.

How Ostgate does it

Ostgate 0.0.8 reads each cluster's endpoint settings and picks a path, shown in the Access column of the GKE area. For a cluster registered in a fleet, one of the paths is the Connect gateway: Google relays the requests to the control plane, and your account needs a gateway role such as roles/gkehub.gatewayReader to list and read. Port-forwards and a shell in a container also need gkehub.gateway.stream, which among the predefined roles only roles/gkehub.gatewayAdmin holds, and a custom role with the reader's permissions plus that one is enough.

Use with kubectl… on a cluster adds a context to ~/.kube/ostgate, with one Path row per path the cluster allows. kubectl, k9s and other tools then sign in by running Ostgate, which hands them a short-lived token of the account the context belongs to, without gcloud or gke-gcloud-auth-plugin, so the PATH problem above doesn't come up. Diagnose Access… checks each step of a cluster's path, including your IAM permissions in the project and your Kubernetes permissions, and names the setting that fails. When the Connect gateway ends a shell after about 30 minutes, the line under the terminal shows the reason. Details are in the GKE docs.

Symptoms and causes

Questions

Is the GKE Connect gateway free?

Yes, since 2 September 2025. Google's GKE release note for that day lists Connect Gateway among the features included in the GKE offering at no additional cost; before that it belonged to GKE Enterprise. Each cluster still pays GKE's cluster management fee of $0.10 per hour. Multi Cluster Gateway, priced per backend pod, is a load balancing feature and not the Connect gateway.

Does kubectl exec work through the Connect gateway?

Yes. Google requires a cluster on 1.30 or later for exec, cp and attach, 1.31 or later for port-forward, and kubectl 1.31 or later, plus the IAM permission gkehub.gateway.stream and Kubernetes permission for the pods/exec subresource. On 11 November 2026 kubectl 1.35.0 ran exec through the gateway on a GKE 1.36 cluster, and the command's output and exit code came back.

Which IAM role do I need for kubectl through the Connect gateway?

Grant it on the fleet host project: roles/gkehub.gatewayReader for read-only kubectl, roles/gkehub.gatewayEditor to change objects, and roles/gkehub.gatewayAdmin for exec, cp, attach and port-forward, because among the predefined roles only it holds gkehub.gateway.stream. A custom role with the reader's three gkehub permissions plus gkehub.gateway.stream ran port-forward in our test. Kubernetes permissions inside the cluster still apply on top.

Why does a kubectl stream through the Connect gateway stop after 30 minutes?

Google ends long gateway streams after about 20 to 30 minutes and calls it an expected limitation for security reasons. exec, attach, cp and port-forward are streams. Our port-forward through the gateway answered for 1798 seconds, then kubectl exited with error: lost connection to pod. Start the command again, or run kubectl port-forward in a loop.