IAP tunnel slow or disconnecting on a Mac

The tunnel drops after it sat idle

Google's IAP documentation states that IAP disconnects sessions after one hour of inactivity. In practice it can be sooner for the leg that matters: we have seen an idle Remote Desktop session's connection from Google to the VM closed after about 18 minutes without traffic, while the WebSocket between the Mac and Google stayed open. WebSocket pings keep only that first leg alive. What keeps the VM side alive is data inside the tunnel.

For SSH, including gcloud compute ssh, scp and editors that use your SSH config, make the client send a keepalive every minute. In ~/.ssh/config:

Host *
    ServerAliveInterval 60
    ServerAliveCountMax 3

For a database or HTTP tunnel, most clients have their own keepalive or "test connection every N seconds" setting. For RDP, the client has to send something on its own while you are away.

The tunnel drops after the Mac sleeps or changes network

Closing the lid, switching Wi-Fi or connecting a VPN breaks the WebSocket to Google. The relay protocol can resume a session: the client reconnects with the session id and the number of bytes it received, and resends what the server had not acknowledged. gcloud retries this for up to 15 minutes, with pauses of up to 20 seconds. If the Mac slept longer, or the program inside gave up first, the session is gone and you reconnect.

A start-iap-tunnel left running in a terminal keeps listening after a failed session, so the next connection to the local port starts a new one. An SSH session through gcloud compute ssh ends with its tunnel and has to be started again.

The tunnel drops when reauthentication is due

A tunnel is opened with an OAuth access token, and the relay can ask for a fresh one: close code 4004, reauthentication required. A client that refreshes the token and reconnects carries on. If your organisation sets a Google Cloud session length, the refresh itself stops working when that session ends, and every tunnel closes until you sign in again: gcloud auth login, or the account's sign-in in your client.

Transfers are slow

• gcloud's tunnel runs in Python. Google's documentation recommends installing NumPy into gcloud's Python to increase IAP TCP upload bandwidth:

  $(gcloud info --format="value(basic.python_location)") -m pip install numpy

• Round trips add up. Every tunnel goes from your Mac to Google and on to the VM. A tool that waits for each block to be acknowledged before sending the next is limited by that round trip, not by your bandwidth. We measured a single-request SFTP client taking 210 s to upload 50 MiB that a pipelined one moves in 10 s over the same tunnel; see measured transfer times.
• IAP is not a bulk transfer service. Google's documentation says TCP forwarding isn't intended for bulk data and may be rate-limited. For large datasets, copy through Cloud Storage instead.
• Distance. A VM in a region far from you adds latency to every round trip. A bastion in the VM's region does not help; the tunnel still ends at the VM.

How Ostgate keeps sessions up

Ostgate is a native macOS app that implements the IAP relay protocol itself, with no gcloud or Python in the path.

• Reconnects the same session after a network drop, with backoff, resending what the relay has not confirmed, and treats unknown close codes as worth a retry rather than an error.
• Refreshes the token on 4004 and reconnects. If an account needs a fresh sign-in, Sign In Again… asks for it while that account's open sessions keep running.
• Keeps Remote Desktop alive: an idle RDP session sends a tiny screen-refresh request every 60 seconds, which is real traffic to the VM.
• Tunnels come back by themselves after a network drop, keep their local port, and keep running when you close the window.
• Pipelined file transfers: the SFTP browser keeps 16 reads or 32 writes in flight per file.

Symptoms and fixes

Close codes are explained one by one in IAP tunnel errors.

Questions

How long can an IAP tunnel stay idle?

Google's documentation says IAP disconnects sessions after one hour of inactivity. We have also seen an idle RDP session's VM-side connection closed after about 18 minutes without traffic, so keep real traffic flowing rather than relying on the hour.

Do WebSocket pings keep the tunnel alive?

They keep the connection between your Mac and Google open, but not the connection from Google to the VM. Only data inside the tunnel does that, such as SSH keepalives or an RDP client's own traffic.

Does a tunnel survive the Mac going to sleep?

The relay can resume a session after a short break: gcloud retries for up to 15 minutes. Whether the program inside survives depends on it. An SSH session usually does after a short sleep; after a long one the server has often given up.

Is IAP TCP forwarding meant for large transfers?

No. Google's documentation says it isn't intended for bulk transfer of data and may rate-limit abuse. Copying files of a few hundred megabytes works; moving datasets belongs in Cloud Storage.