kubectl exec on GKE: cannot create resource "pods/exec", the wrong container, no shell

The test

We ran this on 10 November 2026 on an Autopilot cluster, GKE 1.36.4-gke.2377000 in europe-central2, with kubectl 1.35.0 on a Mac through the cluster's DNS endpoint. In a namespace of its own (written as shop below) we created:

• two-plain, a pod with two containers: app runs busybox:1.37, worker runs the distroless image gcr.io/distroless/python3-debian12, which has Python and no shell
• two-annotated, the same pod with kubectl.kubernetes.io/default-container: worker
• crasher, a container that exits with code 1 and goes into CrashLoopBackOff
• Roles with different rules on pods/exec, each bound to an example.com user that we impersonated with --as

The impersonated users have no IAM role in the project (none of the project's 80 IAM policy members is allUsers, allAuthenticatedUsers, a domain or example.com), so what they could do came from RBAC alone. User names are shortened in the output.

The forbidden error

A user whose Role allows get and list on pods, and nothing else:

$ kubectl get pods -n shop --as=dev@example.com
NAME            READY   STATUS    RESTARTS      AGE
crasher         0/1     Error     3 (31s ago)   46s
two-annotated   2/2     Running   0             47s
two-plain       2/2     Running   0             47s

$ kubectl exec two-plain -c app -n shop --as=dev@example.com -- echo hello
Error from server (Forbidden): pods "two-plain" is forbidden: User "dev@example.com" cannot create resource "pods/exec" in API group "" in the namespace "shop"

Seeing pods doesn't let you exec into them. RBAC names a subresource with a slash, so the rule has to list pods/exec itself; pods doesn't cover it. The user in the message is the identity the cluster saw: your Google account, a service account, or the --as user. If it names something you didn't expect, such as a node's system:node:…, kubectl is signing in with other credentials than you think.

Check with kubectl auth can-i

kubectl auth can-i create pods \
    --subresource=exec -n NAMESPACE

The form many answers give, kubectl auth can-i create pods/exec, checks something else. kubectl reads the argument as TYPE/NAME: “VERB is a logical Kubernetes API verb like 'get', 'list', 'watch', 'delete', etc. TYPE is a Kubernetes resource. … NAME is the name of a particular Kubernetes resource”, and the subresource goes in --subresource (kubectl auth can-i). The request kubectl sent for can-i get pods/exec carried resource pods, an empty subresource and the name exec. Each row below is one user and its Role's rule on pods/exec; every Role also had get and list on pods. The slash form is kubectl auth can-i create pods/exec, the subresource form is kubectl auth can-i create pods --subresource=exec:

The slash form agreed with kubectl exec only for the users who couldn't run it either way. It said no to both users who could, and yes to the one who could create pods but not exec into them.

IAM or RBAC: what GKE needs

Google's pages word this differently. The RBAC page says “In GKE, IAM and Kubernetes RBAC are integrated to authorize users to perform actions if they have sufficient permissions according to either tool”, and “To authorize an action, GKE checks for an RBAC policy first. If there isn't an RBAC policy, GKE checks for IAM permissions” (Authorize actions in clusters using RBAC). The IAM page says “An entity must have sufficient RBAC and IAM permissions to work with resources in your cluster” (Create IAM policies). The access control overview sides with the first: “An entity must have sufficient permissions at either level” (Access control).

In our run, identities with no IAM role in the project opened exec sessions through a RoleBinding alone, so for authorization either one is enough. Where IAM is needed anyway is signing in: “The container.clusters.get permission is required for users to authenticate to the clusters in the project, but does not authorize them to perform any actions inside those clusters” (RBAC page). Impersonation with --as skips that sign-in step, so our test covers authorization only.

On the IAM side, exec is the permission container.pods.exec (Kubernetes Engine roles and permissions):

Google lists the project as the lowest level where Viewer, Developer and Admin can be granted, so Developer allows exec into every pod of every cluster in the project. A Role and RoleBinding keep it to one namespace. We didn't test an account that has only an IAM role; the table comes from Google's role list. Through the Connect gateway, exec also needs the gateway's own gkehub.gateway.stream permission (kubectl through the GKE Connect gateway).

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member=user:dev@example.com \
    --role=roles/container.developer

Grant exec with a Role

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-exec
  namespace: shop
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["get", "create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-exec-dev
  namespace: shop
subjects:
  - kind: User
    name: dev@example.com
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-exec
  apiGroup: rbac.authorization.k8s.io

On GKE the user name is the Google account's email; Google's RBAC page also shows Google Groups (kind: Group) and IAM service accounts as subjects.

Why both verbs: kubectl 1.35 first opens exec with a WebSocket, which is an HTTP GET. With only create, the cluster refused it, and kubectl logged RemoteCommand fallback: unable to upgrade streaming request: … cannot get resource "pods/exec", retried with SPDY over a POST and got in. With get and create the WebSocket answered 101 Switching Protocols at once. With only get, both attempts failed. That matches the AuthorizePodWebsocketUpgradeCreatePermission feature gate, beta and on by default since 1.35: “clients must be authorized to create Pod subresources even when triggering their creation using a WebSocket” (feature gates). kubectl copes with create alone thanks to the fallback; a client that only speaks WebSocket needs both.

Pick the container in a multi-container pod

Without -c, kubectl took the first container and said so:

$ kubectl exec two-plain -n shop -- echo hello
Defaulted container "app" out of: app, worker
hello

-c takes the container's name from the pod spec. A name that isn't in the pod, or a container ID copied from kubectl describe, gets this:

$ kubectl exec two-plain -c nope -n shop -- echo hello
Error from server (BadRequest): container nope is not valid for pod two-plain

kubectl get pod POD -n NAMESPACE \
    -o jsonpath='{.spec.containers[*].name}'

To change the default for everyone, set the annotation on the pod template: “The value of the annotation is the container name that is default for this Pod” (well-known annotations). In two-annotated, kubectl went straight to worker without the “Defaulted container” line, and -c app still reached the other one.

metadata:
  annotations:
    kubectl.kubernetes.io/default-container: app

kubectl also takes a Deployment in place of a pod name, as Google's Autopilot security page shows with kubectl exec -it deploy/YOUR_DEPLOYMENT -- sh (Autopilot security), so you don't need to copy the pod's generated suffix.

No shell in the image: kubectl debug

A distroless container accepts exec, but there is no shell to start. With sh and with /bin/sh the errors differ:

$ kubectl exec two-plain -c worker -n shop -- sh -c 'echo hello'
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "5946…": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

$ kubectl exec two-plain -c worker -n shop -- /bin/sh
error: … OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory

A binary that is in the image still runs: python3 -c 'print("python is here")' worked in the same container. For a shell, add an ephemeral container with its own tools next to it. --target joins the process namespace of the container you name:

kubectl debug -it POD -n NAMESPACE \
    --image=busybox:1.37 --target=worker -- sh

On our Autopilot cluster the debug container started, and ps in it showed the distroless container's process as PID 1. Its files were under /proc/1/root:

PID   USER     TIME  COMMAND
    1 root      0:00 /usr/bin/python3.11 -c import time; time.sleep(86400)
   13 root      0:00 sh -c sleep 3; echo; ps; …

$ ls /proc/1/root/usr/bin
python
python3
python3.11
$ ls /proc/1/root/bin/sh
ls: /proc/1/root/bin/sh: No such file or directory

kubectl 1.35 warned that the default --profile=legacy is deprecated and suggested naming one, such as --profile=general. An ephemeral container needs a permission of its own, which our user with get and create on pods/exec didn't have:

Error from server (Forbidden): pods "two-plain" is forbidden: User "dev@example.com" cannot patch resource "pods/ephemeralcontainers" in API group "" in the namespace "shop"

Kubernetes' docs grant it with patch and update on pods/ephemeralcontainers, because “Just being authorized to patch a Pod does not allow users to add ephemeral containers to that Pod” (Debug running pods).

Logs of a container in CrashLoopBackOff

A container in CrashLoopBackOff isn't running between restarts, so kubectl exec has nothing to attach to. We tested this on 12 November 2026 on the same cluster with crasher, a pod whose container app prints run started and the time, sleeps 20 seconds, writes config missing to stderr and exits with code 3:

$ kubectl exec crasher -c app -n shop -- echo hello
error: unable to upgrade connection: container not found ("app")

$ kubectl get pod crasher -n shop
NAME      READY   STATUS             RESTARTS      AGE
crasher   0/1     CrashLoopBackOff   5 (81s ago)   6m37s

While the container was running, the same exec printed hello. The exit code of the run that failed is under Last State:

kubectl describe pod POD -n NAMESPACE

State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       Error
  Exit Code:    3
  Started:      Thu, 12 Nov 2026 08:47:02 +0100
  Finished:     Thu, 12 Nov 2026 08:47:23 +0100
Ready:          False
Restart Count:  5

The gaps between an exit and the next start were 1, 13, 28, 55 and 90 seconds, and after the sixth run the container's waiting message read back-off 2m40s restarting failed container=app. Kubernetes restarts “with an exponential backoff delay (10s, 20s, 40s, …), that is capped at 300 seconds (5 minutes)” (Pod lifecycle).

The error itself is in the logs. --previous prints “the logs for the previous instance of the container in a pod if it exists” (kubectl logs). What each command printed at three points in the loop:

kubectl logs POD -c CONTAINER -n NAMESPACE \
    --previous

While the container waits, both commands show the run that just failed. Once it starts again, plain kubectl logs shows only the new run, which hasn't reached its error yet, so --previous is the one to type in both states. Before the first restart it fails with Error from server (BadRequest) and the message in the table. Older runs are gone from the node: “By default, if a container restarts, the kubelet keeps one terminated container with its logs” (Logging architecture).

Cloud Logging keeps all of them. Google's logging overview says “GKE container logs are removed when their host Pod is removed”, and “GKE logs sent to Cloud Logging are stored in a dedicated, persistent datastore” (About GKE logs). After we deleted the pod's namespace, gcloud logging read still returned all 12 lines of the six runs, with stdout and stderr as separate logs and the stderr line at severity ERROR. The _Default log bucket keeps entries for 30 days unless the project changes that (Cloud Logging quotas).

gcloud logging read 'resource.type="k8s_container"
  AND resource.labels.namespace_name="NAMESPACE"
  AND resource.labels.pod_name="POD"' \
    --project=PROJECT_ID --freshness=1h --limit=20 \
    --format='table(timestamp,severity,textPayload)'

TIMESTAMP                       SEVERITY  TEXT_PAYLOAD
2026-11-12T07:47:23.009192293Z  ERROR     config missing
2026-11-12T07:47:03.005298167Z  INFO      run started 07:47:03
2026-11-12T07:45:33.205710481Z  ERROR     config missing
2026-11-12T07:45:13.200414523Z  INFO      run started 07:45:13

To look around inside the image, start a copy of the pod whose container runs a command that doesn't exit, and exec into the copy. Autopilot allowed it:

kubectl debug POD -n NAMESPACE --copy-to=POD-debug \
    --container=app --profile=general \
    -- sh -c 'while true; do sleep 5; done'
kubectl exec -it POD-debug -c app -n NAMESPACE -- sh

The copy's container got ["sh","-c","while true; do sleep 5; done"] as its command, so its own logs stayed empty. It kept only the two topology.kubernetes.io labels and lost the pod's app label, so Services don't send it traffic. Delete it when you're done.

How Ostgate does it

Open Shell… in a pod's menu, or Shell… in its inspector, opens a terminal tab in one of the pod's containers, like kubectl exec -it. Before anything opens, Ostgate asks the cluster whether your account has get and create on pods/exec in that namespace, and says which one is missing when it doesn't. Then it asks you to confirm, naming the namespace, the cluster, its project and your account, with PROD or STAGING when the project ID says so.

In a pod with more than one container you pick the container in the same dialog, and the one the kubectl.kubernetes.io/default-container annotation names is chosen already. The tab starts bash, or /bin/sh if the image doesn't have it. A distroless image gets “No shell in” and the container's name instead of a terminal. A cluster that Ostgate reaches only through its public endpoint can't open a shell yet: turn on the cluster's DNS endpoint, or use kubectl exec. Details are in Shell in a container.

Pod logs open in a tab in Ostgate. The 0.0.8 release notes list “Pod logs in a tab: follow, pause, search, the previous run, and Save…”.

Symptoms and causes

Questions

What permission does kubectl exec need on GKE?

create on the pods/exec subresource in the pod's namespace. On GKE it can come from Kubernetes RBAC, a Role and RoleBinding, or from IAM, where container.pods.exec is in Kubernetes Engine Developer, Editor and Admin but not in Kubernetes Engine Viewer. In our test on GKE 1.36, kubectl 1.35 opened the session with create alone: its WebSocket request was refused for lack of get, and it fell back to SPDY. With get and create it stayed on WebSocket.

Why does kubectl auth can-i create pods/exec say yes when kubectl exec is forbidden?

kubectl auth can-i reads pods/exec as TYPE/NAME, so it asks about a pod named exec, not the exec subresource. In our test a user with create on pods and no rule for pods/exec got yes and was then refused by kubectl exec, and users who could exec got no. Ask kubectl auth can-i create pods --subresource=exec -n NAMESPACE instead.

Can a Kubernetes Engine Viewer exec into a pod on GKE?

Not through IAM: roles/container.viewer has container.pods.get, getStatus and list, and not container.pods.exec. GKE also authorizes through RBAC, so a RoleBinding with create on pods/exec in that namespace is enough on its own. In our test, identities with no IAM role in the project ran kubectl exec through RBAC alone. We didn't test an account that has only the Viewer role.

How do I exec into a container in CrashLoopBackOff?

You can't while the container waits to restart: kubectl exec answered error: unable to upgrade connection: container not found ("app"). Read the last run with kubectl logs --previous, or start a copy of the pod with a command that keeps running, using kubectl debug POD --copy-to=POD-debug --container=app, and exec into the copy. Our copy didn't keep the pod's labels, so Services don't send it traffic.

How do I see the logs of a container in CrashLoopBackOff?

Run kubectl logs POD -c CONTAINER --previous. It prints the run that crashed, both while the container waits and after it has started again; in our test, plain kubectl logs showed only the new run once the container restarted. Before the first restart, --previous fails with previous terminated container "app" in pod "crasher" not found. The kubelet keeps one terminated run per container. GKE also sends container output to Cloud Logging, where gcloud logging read still returned all six runs after we deleted the pod.

kubectl logs POD -n NAMESPACE --previous
kubectl debug POD -n NAMESPACE --copy-to=POD-debug \
    --container=app --profile=general \
    -- sh -c 'while true; do sleep 5; done'
kubectl exec -it POD-debug -c app -n NAMESPACE -- sh