OOMKilled and exit code 137 on GKE

The test

We ran this on 10 November 2026 on an Autopilot cluster, GKE 1.36.4-gke.2377000 in europe-central2. The node ran kernel 6.12.105+ on Container-Optimized OS with containerd 2.2.7. Two pods in namespace ostgate-t, applied at 14:54:14Z, both busybox:1.37 with a memory request and limit of 64Mi:

• t9-oom-limit runs tail /dev/zero as its only process
• t9-oom-child runs sh as PID 1 in a sleep loop, and after 20 seconds sh starts tail /dev/zero in the background

command: ["sh", "-c", "echo start; (sleep 20; tail /dev/zero) & while true; do sleep 5; done"]
resources:
  requests: {cpu: 50m, memory: 64Mi}
  limits: {memory: 64Mi}

Exit code 137 in kubectl describe pod

kubectl -n ostgate-t describe pod t9-oom-limit

    State:          Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Tue, 10 Nov 2026 15:54:35 +0100
      Finished:     Tue, 10 Nov 2026 15:54:36 +0100
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Tue, 10 Nov 2026 15:54:20 +0100
      Finished:     Tue, 10 Nov 2026 15:54:21 +0100
    Ready:          False
    Restart Count:  2
    Limits:
      ephemeral-storage:  1Gi
      memory:             64Mi

“An Exit Code of 137 also indicates an OOM kill” (Troubleshoot OOM events). The restart count reached 2 by 14:54:43Z and 6 by 15:01Z.

The child-process pod, read between two kills (its restart count reached 5 later):

    State:          Running
      Started:      Tue, 10 Nov 2026 15:54:40 +0100
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Tue, 10 Nov 2026 15:54:18 +0100
      Finished:     Tue, 10 Nov 2026 15:54:39 +0100
    Ready:          True
    Restart Count:  1

A child process over the limit restarts the whole container

sh used almost no memory and died anyway. The node's serial console for the kill at 14:54:39Z, trimmed:

[ 1261.090695] tail invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=999
[ 1261.759553] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-….scope,…
[ 1261.805340] Memory cgroup out of memory: Killed process 19573 (tail) total-vm:68740kB, anon-rss:64256kB, …
[ 1261.844794] Memory cgroup out of memory: Killed process 19561 (sh) total-vm:4560kB, anon-rss:0kB, …
[ 1262.027312] Memory cgroup out of memory: Killed process 19813 (sleep) total-vm:4428kB, anon-rss:128kB, …

The kill at 14:55:01Z looks the same: tail, sh and sleep gone within 0.3 s. Kubernetes' resource management docs expect one process to go: the kernel “typically, intervenes by stopping one of the processes in the container that tried to allocate memory”. On cgroup v2, though, “the OOM killer prevents partial kills” (Troubleshoot OOM events). The kubelet decides this per container cgroup: singleProcessOOMKill, “if true, will prevent the memory.oom.group flag from being set for container cgroups in cgroups v2”, and on cgroup v2 “The default value is false” (kubelet configuration reference). We didn't read the node's cgroup version; the kills match the group behaviour.

If your container runs workers under a supervisor or a shell script, one worker over the limit takes all of them down.

Container limit or node out of memory

When a container hits its limit, the kernel line contains memory cgroup, mem_cgroup or memcg. A node out of memory logs “a general message like Out of memory: Killed process... without mentioning a cgroup” (Troubleshoot OOM events). Our lines read Memory cgroup out of memory and constraint=CONSTRAINT_MEMCG.

We did not reproduce a node-wide OOM. In that case “the global OOM Killer assesses all processes on the node and terminates a process” (Troubleshoot OOM events), so the victim can be a pod that never reached its own limit. On a Standard node you can read the kernel log yourself:

gcloud compute ssh NODE_NAME --zone ZONE --project PROJECT_ID

sudo journalctl -k | grep -i "out of memory"

Find OOM kills in Cloud Logging

Node OOMKilling events

resource.type="k8s_node" AND jsonPayload.reason="OOMKilling"

On our test project this returned 200 entries for the last 7 days, all from one node. A long-running test pod there had restarted 1,594 times, each restart ending OOMKilled at its own 64Mi limit. Every entry is a Warning from kernel-monitor:

"reason": "OOMKilling",
"source": {"component": "kernel-monitor", …},
"type": "Warning",
"message": "(combined from similar events): Memory cgroup out of memory: Killed process 314564 (tail) total-vm:69132kB, anon-rss:64768kB, …"

“OOMKilling events are reported for the node (k8s_node resource type), so they don't have a namespace label”, and the message names a process and its PID. To reach the pod, Google pairs each event with the Started container event that follows it on the same node, and warns that “Not every Started container event is caused by an OOM kill” (Troubleshoot OOM events):

log_id("events")
resource.labels.cluster_name="CLUSTER_NAME"
(
  (
    resource.type="k8s_pod"
    resource.labels.namespace_name="NAMESPACE_NAME"
    jsonPayload.message:"Started container"
  ) OR (
    resource.type="k8s_node"
    jsonPayload.reason="OOMKilling"
  )
)

On one node the events came half an hour late. Our two test pods ran on a node created at 14:34Z and were first killed at 14:54Z. The serial console logged every kill at once, but the node's first OOMKilling event is stamped 15:24:01Z, and Cloud Logging received it four seconds later, so the delay was on the node, not in Logging. We don't know why. Until we do, an empty result for the last half hour doesn't prove nothing on a node was killed.

The node's serial console

resource.type="gce_instance"
log_id("serialconsole.googleapis.com/serial_port_1_output")

Both test pods' kernel lines arrived here, including those from the node that sent no OOMKilling event.

Size the memory limit

Google's first fix is “a higher resources.limits.memory value that accommodates the app's peak usage”, with resources.requests.memory “set to a realistic value for typical usage”. When many pods on a node sit near their limits, “the node itself might be too small” (Troubleshoot OOM events).

On Autopilot the request often sets the limit. In a cluster that doesn't burst, a pod with requests and no limits gets limits anyway: “GKE sets the limits equal to the requests”. Google's advice is to “set adequate resource requests and equal limits” (Autopilot resource requests).

To kill only the offending process on a Standard node pool, the kubelet setting singleProcessOOMKill “sets whether the processes in the container are OOMkilled individually or as a group”. It defaults to false on cgroupv2 nodes and needs GKE 1.32.4-gke.1132000, 1.33.0-gke.1748000 or later (node system configuration); that page doesn't say whether Autopilot accepts it. Turning it on brings back the cgroup v1 problem: “The main container process might continue to run, even if critical child processes have been terminated.”

How Ostgate does it

A cluster's Overview in Ostgate has a COST block, and Open Cost opens the cluster's Cost view. Its last section, OUT OF MEMORY, covers the last 14 days with three kinds of line:

• Each container whose current or last state ended OOMKilled: namespace, pod, container, how long ago, its memory limit (none without one) and its restarts. Only pods that still exist appear here.
• The nodes' kernel kills, read from Cloud Logging with the same k8s_node OOMKilling filter narrowed to the cluster: node, killed and the process name, its anon-rss and the age, with the kernel's full message on hover. It reads the newest 50.
• Nodes whose running pods' memory limits add up to more than the node's allocatable memory, with the percentage and the number of containers that have no limit.

A deleted pod's kills appear only as kernel lines, and while a node's events lag, as ours did, its kills show only in the pod lines. The Cloud Logging read uses the Google account the cluster is opened with. Without logging.logEntries.list the section keeps the pod lines and names Logs Viewer (roles/logging.viewer) as the role to ask for.

In the pods table, a pod in CrashLoopBackOff whose last end was OOMKilled shows Out of memory, and the inspector's explanation names the container and its limit. With Notify Me About Problems on, each OOMKilled restart posts a macOS notification (problem notifications). The RIGHTSIZING section of the Cost view suggests “memory limit = the 14-day maximum × 1.2”. A container that dies at its limit never shows more than the limit, so for that container treat the number as a floor.

Symptoms and causes

Questions

What does exit code 137 with OOMKilled mean on GKE?

The kernel's out-of-memory killer stopped the container. Google's GKE troubleshooting treats exit code 137 as an OOM kill as well. Usually a process in the container went over the container's resources.limits.memory; less often the whole node ran out of memory. kubectl describe pod shows Reason: OOMKilled and Exit Code: 137 under State or Last State.

Why did the whole container restart when only a child process used too much memory?

On cgroup v2 the kubelet sets memory.oom.group on each container's cgroup unless singleProcessOOMKill is true, and its default is false, so the kernel kills every process in the container together. In our test on GKE 1.36 a background tail went over a 64Mi limit and the kernel killed tail, the sh that was PID 1 and a sleep within 0.3 seconds. On Standard node pools, GKE 1.32.4-gke.1132000 and 1.33.0-gke.1748000 or later accept singleProcessOOMKill in the node system configuration.

Why doesn't the OOMKilling event in Cloud Logging name the pod?

OOMKilling events are reported on the k8s_node resource, so they carry no namespace, and the message names the killed process and its PID. Google's way to find the pod is the Started container event that follows on the same node, though not every Started container event comes from an OOM kill.

Does an OOMKilling event mean the node ran out of memory?

Not by itself. On our test cluster every OOMKilling event came from a container killed at its own memory limit, reported by kernel-monitor. Read the message: Memory cgroup out of memory points to a container limit, while a node that ran out logs Out of memory: Killed process without a cgroup. On one new node the first OOMKilling event came 30 minutes after the first kill, although its serial console logged the kills at once.