x509: certificate signed by unknown authority on the GKE DNS endpoint

Why the cluster CA fails on a gke.goog name

Google's network isolation page says which certificate each endpoint presents. For the DNS-based endpoint, “the Google Front End (GFE) service that hosts the DNS-based endpoint presents the certificate. The certificate is signed by a public, generally recognized Google CA.” For the IP-based endpoints, “the Kubernetes API server presents the certificate. The certificate is signed by the cluster root CA” (About network isolation in GKE).

The cluster root CA is the one in masterAuth.clusterCaCertificate, the certificate Terraform's GKE examples and Flux's GCP provider use. On our test cluster it is self-signed and valid until 2056:

gcloud container clusters describe acme-gke \
    --location=europe-central2 --project=acme-prod \
    --format='value(masterAuth.clusterCaCertificate)' | base64 -d \
  | openssl x509 -noout -subject -issuer

subject=CN=0f1e2d3c-…
issuer=CN=0f1e2d3c-…

The DNS endpoint's chain goes to Google Trust Services instead:

openssl s_client -connect gke-0a1b2c3d4e5f-123456789012.europe-central2.gke.goog:443 \
    -servername gke-0a1b2c3d4e5f-123456789012.europe-central2.gke.goog -showcerts </dev/null

 0 s:CN=*.europe-central2.gke.goog
   i:C=US, O=Google Trust Services, CN=WE2
 1 s:C=US, O=Google Trust Services, CN=WE2
   i:C=US, O=Google Trust Services LLC, CN=GTS Root R4
 2 s:C=US, O=Google Trust Services LLC, CN=GTS Root R4
   i:C=BE, O=GlobalSign nv-sa, OU=Root CA, CN=GlobalSign Root CA
Verification: OK

With the cluster CA as the only trusted root (-CAfile cluster-ca.pem -no-CApath -no-CAstore), the same handshake ends with Verify return code: 20 (unable to get local issuer certificate). gcloud allows for this: “because the Google Front End (GFE) service that hosts the DNS-based endpoint presents the certificate that's signed by a public certificate authority (CA), the kubeconfig entry omits the certificate-authority-data field” (Install kubectl and configure cluster access). Configurations written for the IP endpoint keep passing the cluster CA when only the host changes.

The error with kubectl

With a kubeconfig from get-credentials --dns-endpoint, we ran the same command three ways:

kubectl get ns
kubectl get ns --certificate-authority=cluster-ca.pem
kubectl get ns --certificate-authority=/etc/ssl/cert.pem

The first, with no CA set, and the third, with macOS's bundle of public roots, listed the namespaces. The second, with the cluster CA, failed:

Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority

helm list --kube-ca-file cluster-ca.pem failed the same way, with Error: kubernetes cluster unreachable: Get "https://gke-…gke.goog/version": tls: failed to verify certificate: x509: certificate signed by unknown authority, and listed releases without it.

kubectl: regenerate the entry instead of editing server:

A common way into this error is a kubeconfig written for the IP endpoint whose server: line was changed to the gke.goog name by hand. The entry still carries the cluster CA in certificate-authority-data. A user on Google's forum did exactly that, got the error, and got kubectl working with insecure-skip-tls-verify: true; the fix that held was a newer gcloud and --dns-endpoint (GKE DNS control plane endpoint failed to verify certificate: x509). Skipping verification gives up a check the certificate passes anyway. Write a new entry:

gcloud container clusters get-credentials acme-gke \
    --location=europe-central2 --project=acme-prod --dns-endpoint
kubectl config view --minify

The cluster entry then has a server: line with the gke.goog name and no certificate-authority-data line. Without --dns-endpoint, “The get-credentials command automatically uses the IP-based endpoint if the IP-based endpoint access is enabled” (Customize your network isolation). To fix an entry in place, remove the CA from it:

kubectl config unset clusters.gke_acme-prod_europe-central2_acme-gke.certificate-authority-data

To reach the name at all, the cluster must allow external traffic on its DNS endpoint, and your account needs container.clusters.connect, which roles/container.developer and roles/container.viewer include. Those errors are in kubectl times out on a GKE cluster and the table below.

Terraform kubernetes and helm providers

Google's “Using GKE with Terraform” guide and the helm provider's own GKE example both build the provider from .endpoint, the IP address, and pass cluster_ca_certificate from master_auth. Point host at the DNS endpoint and the CA has to go. terraform-provider-kubernetes#2637, open since November 2024, shows the result:

Error: Post "https://redacted-gke-dns-endpoint-hostname.us-east1.gke.goog/api/v1/namespaces": tls: failed to verify certificate: x509: certificate signed by unknown authority

Our plan with hashicorp/kubernetes 3.5.0 had two provider blocks for the same gke.goog host. The one with cluster_ca_certificate failed with Failed to read Namespace "kube-system" and the same x509 message; the one without it read the namespace. The working block:

data "google_container_cluster" "acme" {
  name     = "acme-gke"
  location = "europe-central2"
  project  = "acme-prod"
}

locals {
  dns_endpoint = data.google_container_cluster.acme.control_plane_endpoints_config[0].dns_endpoint_config[0].endpoint
}

provider "kubernetes" {
  host = "https://${local.dns_endpoint}"
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "gke-gcloud-auth-plugin"
  }
}

Where the runner has no gke-gcloud-auth-plugin, a token from google_client_config worked for commenters on the issue; we ran only the exec form. Google's guide says this token “is short lived (thus it expires) and it's stored in state”, and points to the plugin instead.

data "google_client_config" "default" {}

provider "kubernetes" {
  host  = "https://${local.dns_endpoint}"
  token = data.google_client_config.default.access_token
}

The helm provider takes the same settings, as a kubernetes attribute since version 3:

provider "helm" {
  kubernetes = {
    host  = "https://${local.dns_endpoint}"
    token = data.google_client_config.default.access_token
  }
}

For a cluster created in the same configuration, set dns_endpoint from the resource instead: google_container_cluster.acme.control_plane_endpoints_config[0].dns_endpoint_config[0].endpoint.

Argo CD: x509, or “the server has asked for the client to provide credentials”

Argo CD fails in two ways on a gke.goog server. With the cluster CA in the cluster's tlsClientConfig.caData, it gets the x509 error above. Adding the cluster with argocd cluster add fails differently, as in argo-cd#21318:

FATA[0006] rpc error: code = Unauthenticated desc = the server has asked for the client to provide credentials

argocd cluster add creates an argocd-manager ServiceAccount on the cluster for Argo CD to use, and the DNS endpoint refuses Kubernetes ServiceAccount tokens by default. Google lists them among authentication methods that “are disabled by default in GKE clusters” (About network isolation in GKE). On our cluster, with enableK8sTokensViaDns: false, a fresh token for a test ServiceAccount got 401 Unauthorized as Google's HTML error page, and kubectl printed:

error: You must be logged in to the server (the server has asked for the client to provide credentials (get pods))

A TokenReview of a token for the same ServiceAccount, sent through the IAM-authenticated context, returned authenticated=true. The API server accepts these tokens; the DNS endpoint's front end turns them away. Commenters on the issue got past it by switching the cluster back to the IP endpoint.

Add the cluster declaratively instead, signed in with Google credentials through argocd-k8s-auth gcp and with no cluster CA. Argo CD's declarative setup takes a cluster as a Secret in the argocd namespace labelled argocd.argoproj.io/secret-type: cluster, with the keys name, server and config. Set server to https://gke-0a1b2c3d4e5f-123456789012.europe-central2.gke.goog and config to the JSON a commenter on the issue reported working:

{
  "execProviderConfig": {
    "command": "argocd-k8s-auth",
    "args": ["gcp"],
    "apiVersion": "client.authentication.k8s.io/v1beta1"
  },
  "tlsClientConfig": {
    "insecure": false,
    "caData": ""
  }
}

The config holds no credential: argocd-k8s-auth fetches a Google token each time. Argo CD's docs add that you must enable Workload Identity, create a Google service account “and bind it to Kubernetes service account for argocd-application-controller and argocd-server”. That service account needs container.clusters.connect on the target cluster's project and RBAC inside the cluster. We did not run Argo CD itself.

The other way out is to let the DNS endpoint accept ServiceAccount tokens with gcloud container clusters update --enable-k8s-tokens-via-dns. Google's advice: “Unless your use case requires one of these authentication methods, use IAM credentials to authenticate to your control plane.”

Flux: the GCP provider reads the cluster CA

Flux's secret-less remote cluster access, .spec.kubeConfig.configMapRef with provider: gcp, takes an optional ca.crt; “If not set, the controller will use the CA certificate from the cluster resource” (Kustomization API). That is the cluster root CA, so a gke.goog address fails with the x509 error. fluxcd/flux2#5905, a feature request to use the system roots for gke.goog names, is open. A maintainer's description of the current behaviour: “The feature allows using a CA either set directly in the ConfigMap, or retrieved from the cluster resource in the GKE API, nothing else.”

Until that changes, you can set ca.crt to Google's published root bundle, pki.goog/roots.pem (21 certificates when we fetched it, including GTS Root R4). kubectl with that file as its only CA listed kube-system on our DNS endpoint; we did not run it through Flux.

What we tested on

On 10 November 2026, from a Mac on macOS 27: Google Cloud SDK 590.0.0, kubectl v1.35.0, Helm v4.1.0, Terraform 1.16.6 with hashicorp/kubernetes 3.5.0 and OpenSSL's s_client. The cluster was a regional Autopilot cluster on 1.36.4-gke.2377000 in europe-central2, with the DNS endpoint on and external traffic allowed, ServiceAccount tokens and client certificates via DNS off, and the public IP endpoint off. Apart from the ServiceAccount test, every command only read from the cluster; the test ServiceAccount and its namespace were deleted afterwards.

How Ostgate does it

Ostgate reads each cluster's endpoint settings and picks a path. One of them is the DNS endpoint: “the cluster's gke.goog name, reached over the internet and signed in with your Google account” (GKE docs). Use with kubectl… on a cluster adds a context for it to ~/.kube/ostgate, with Ostgate's pick of path marked “recommended”. kubectl signs in by running Ostgate, which hands it a short-lived token of the account the context belongs to, so neither gcloud nor gke-gcloud-auth-plugin is needed. The same file works for k9s and other tools that read a kubeconfig.

export KUBECONFIG="$HOME/.kube/config:$HOME/.kube/ostgate"
kubectl --context ostgate-acme-prod-acme-gke get namespaces

When a cluster can't be reached, Diagnose Access… checks each step of the path in order: the endpoint settings, authorized networks, your IAM permissions in the project and your Kubernetes permissions. Ostgate covers kubectl on your Mac; Terraform, Argo CD and Flux running elsewhere still need the settings above.

Symptoms and causes

The clusters.connect row comes from a forum thread where a project owner hit it; Google's page names the two roles. Other ways to reach a private control plane are in kubectl to a private GKE cluster from a Mac.

Questions

Why does the GKE DNS endpoint fail with x509: certificate signed by unknown authority?

The gke.goog name is served by Google Front End with a certificate from Google Trust Services, a public CA. The cluster's own root CA, from masterAuth.clusterCaCertificate, signs only the API server's certificate on the IP-based endpoints. A client told to trust only that CA rejects the gke.goog certificate. Leave the CA out for the DNS endpoint so the client checks it against the system roots.

Is insecure-skip-tls-verify a fix for the GKE DNS endpoint?

It makes the error go away by turning off the certificate check. The certificate chains to public roots that macOS ships, so remove the cluster CA instead and keep the check. With gcloud, get-credentials --dns-endpoint writes an entry without certificate-authority-data.

Why does Argo CD say the server has asked for the client to provide credentials on a GKE DNS endpoint?

argocd cluster add creates an argocd-manager ServiceAccount on the cluster for Argo CD to use. The DNS endpoint refuses Kubernetes ServiceAccount tokens unless the cluster has enable-k8s-tokens-via-dns. In our test a valid ServiceAccount token got a 401 from Google's front end, and kubectl printed the same message. Add the cluster with a declarative cluster Secret that signs in through argocd-k8s-auth gcp instead.

Do I still need the cluster CA certificate on GKE?

Only for the IP-based endpoints, where the API server presents a certificate signed by the cluster root CA. Clients of the DNS-based endpoint need no cluster CA. A tool that talks to both needs a different TLS setting for each.