gcloud crashed (DatabaseError): file is not a database

The error and its traceback

We damaged each of gcloud's SQLite files in turn in a scratch config directory, on Google Cloud SDK 590.0.0 on macOS. gcloud auth list stopped with the same output each time:

ERROR: gcloud crashed (DatabaseError): file is not a database

If you would like to report this issue, please run the following command:
  gcloud feedback

To check gcloud for common problems, please run the following command:
  gcloud info --run-diagnostics

The console never says which file it was, even with --verbosity=debug. The traceback goes only to the command's log file under logs/ in the config directory. Its last frames, with default_configs.db broken:

  File ".../googlecloudsdk/core/credentials/creds.py", line 401, in __init__
    config_store = config_sqlite_store.GetConfigStore()
  File ".../googlecloudsdk/core/config_sqlite_store.py", line 84, in GetConfigStore
    return _GetSqliteStore(config_name)
  File ".../googlecloudsdk/core/config_sqlite_store.py", line 379, in _GetSqliteStore
    config_store = SqliteConfigStore(sqlite_config_file, config_name)
  File ".../googlecloudsdk/core/config_sqlite_store.py", line 138, in __init__
    self._Execute(
        'CREATE TABLE IF NOT EXISTS config '
        '(config_attr TEXT PRIMARY KEY, value BLOB)'
    )
  File ".../googlecloudsdk/core/config_sqlite_store.py", line 145, in _Execute
    return cur.Execute(*args)
  File ".../googlecloudsdk/core/config_sqlite_store.py", line 65, in Execute
    return self._cursor.execute(*args)
sqlite3.DatabaseError: file is not a database

Frames in config_sqlite_store.py point at a configuration's _configs.db. _InitializeSchema in creds.py points at access_tokens.db, and the credential store's __init__ at creds.py line 397 at credentials.db. The check in Find the broken file is quicker than reading the log.

gcloud config list kept working in all three cases: it reads properties from plain files in configurations/.

Which .db file holds what

gcloud 590.0.0 keeps these SQLite files in its config directory (googlecloudsdk/core/config.py). Its own comment on the first two: “Static credentials information, such as client ID and service account email, are stored in credentials.db. The short lived credentials tokens, such as access token, are cached in access_tokens.db.”

A service account key goes back in with gcloud auth activate-service-account --key-file=KEY_FILE. The configs DB holds flags such as client_arch and cred_token_store_formatted; your properties live in plain files under configurations/. A configuration named northwind got its own northwind_configs.db the first time we ran a command with --configuration=northwind. In our test gcloud created each deleted file again on the next command. That run had no account signed in, so the cost of losing access_tokens.db with a live account comes from gcloud's source above, not from a test.

Find the broken file

Print the config directory. This worked in our test with the configs DB broken:

gcloud info --format='value(config.paths.global_config_dir)'

Without CLOUDSDK_CONFIG it is ~/.config/gcloud on macOS and Linux and %APPDATA%\gcloud on Windows. Then check every .db file there with sqlite3, which macOS has in /usr/bin. -readonly keeps the check from writing:

cd ~/.config/gcloud
for f in *.db; do printf '%s: ' "$f"; sqlite3 -readonly "$f" 'PRAGMA quick_check;'; done

access_tokens.db: Parse error in 3rd command line argument: file is not a database (26)
credentials.db: ok
default_configs.db: ok
northwind_configs.db: ok

To see whether the IAP tunnel race did it, look at the first bytes of the broken file:

xxd -l 16 default_configs.db

00000000: 5351 4c69 7417 0303 0013 0ec7 a1ca c27f  SQLit...........

A healthy file starts with SQLite format 3. The race leaves SQLit and then 17 03 03 00 13 plus 19 more bytes, as the repro reports. Our 19 bytes were random.

Fix it

Move only the file sqlite3 named, then run any gcloud command:

mv default_configs.db default_configs.db.broken
gcloud auth list

For each of the three files, gcloud auth list worked again and a new file starting with SQLite format 3 was in place. Delete the .broken copy once gcloud works. If it was credentials.db, sign in again with gcloud auth login.

Moving all of ~/.config/gcloud away also clears the error, but it takes your configurations and properties with it.

Why IAP tunnels break it

A public repro, gcloud-iap-stale-fd-repro, traces the damage to tunnel shutdown: “When a tunnel closes, close() can run on one thread while the reader thread is still inside an SSL read. The socket's fd number is freed, gcloud reopens its configs DB on the same number, and OpenSSL, still holding that number, reads the DB header as a bad TLS record and writes an alert into the file.” Its author runs many gcloud compute start-iap-tunnel sessions with SSH connections opening and closing all day. The file that breaks is <config>_configs.db, holding one encrypted TLS alert record.

The repro runs gcloud 590.0.1 offline against a fake relay; we did not run it. The bytes in our test were written by hand in the shape it reports. Moving the file aside fixes the current damage, and the race can damage the new file the same way.

How Ostgate does it

Ostgate needs no gcloud (install docs). It opens its IAP tunnels without gcloud's tunnel code and stores each account's refresh token in the macOS Keychain, so its tunnels never write into gcloud's config files. The SSH Integration block lets ssh, scp and IDEs reach INSTANCE.ZONE.PROJECT.gcp through IAP using your Ostgate accounts. For GKE, Use with kubectl… lets kubectl and k9s sign in through Ostgate, without gcloud or gke-gcloud-auth-plugin (release notes).

gcloud itself stays as it is on your Mac. If you still run gcloud compute start-iap-tunnel, its files can break as above.

Symptoms and causes

Questions

Which file causes "gcloud crashed (DatabaseError): file is not a database"?

One of the SQLite files in gcloud's config directory, ~/.config/gcloud on macOS and Linux: credentials.db, access_tokens.db, or a configuration's NAME_configs.db such as default_configs.db. The message is the same for each of them. Run sqlite3 -readonly FILE 'PRAGMA quick_check;' on every .db file there; the broken one answers file is not a database and the others answer ok.

Is it safe to delete access_tokens.db or default_configs.db?

gcloud's source calls access_tokens.db a cache of short-lived tokens, and default_configs.db holds a few cached flags, not your properties, which live in configurations/config_default. In our test on gcloud 590.0.0 the next command created each file again. Deleting credentials.db is different: it holds every account you signed in with, so you run gcloud auth login again.

Why does default_configs.db break again when I use IAP tunnels?

A public repro traces it to a race when a tunnel closes: the socket's file descriptor number is freed while a reader thread is still inside an SSL read, gcloud reopens its configs DB on that number, and OpenSSL writes a TLS alert record into the file. The more tunnels open and close, the more often it happens. The damaged file starts with SQLit followed by the bytes 17 03 03 00 13.

Why does gcloud config list still work?

gcloud config list reads properties from the files in configurations/, not from the SQLite files. In our test it exited 0 with each of the three files broken, while gcloud auth list crashed every time.