How to enable GCP SDK debug logging to diagnose GCS backup deletion in YugabyteDB Anywhere

Problem

On universes that back up to Google Cloud Storage (GCS), backup deletions may appear to fail, stall, or succeed in the UI while objects remain in the bucket and the default YBA logs give no useful detail.

Error / alert observed: default application.log shows only a generic line with no HTTP-level cause:

Backup <uuid is successfully deleted

Impact: Retention policies do not free storage, creating a compliance risk if old backups must be removed. Operators cannot tell whether the cause is IAM, GCS-side, network/proxy, or stale YBA metadata.

Scope: This procedure covers GCS calls made by the YBA platform Java process (BackupGarbageCollector, storage-config validation, key-file deletion). It does not cover YB-Controller (YBC) node-side backup data paths see Limitations.

Cause

YBA uses the GCP Java SDK to talk to GCS. By default, the com.google.cloud and com.google.api loggers run at WARN, and the HTTP transport does not print request/response detail. A failure therefore collapses into a single application-level line with no visibility into:

• OAuth2 token acquisition (oauth2.googleapis.com)
• Permission denials (403 on storage.buckets.get or object APIs)
• Partial batch-delete failures (500 on individual objects in a batch)
• Proxy or service-mesh interception (502 on metadata or token endpoints)

Enabling SDK debug logging exposes each HTTP call so you can see exactly where the deletion flow stopped.

Diagnosis

Step 1 — Confirm the symptom in YBA

1. Open Universes → <universe → Backups.
2. Note the backup UUID and state (Expired, Deleting, or Failed).
3. Check Tasks for a Delete Backup / BackupGarbageCollector task and its error text.

If the UI shows success but GCS still holds objects, continue with SDK logging. YBA may have updated metadata while the GCS cleanup failed silently or partially.

Step 2 — Enable GCP SDK debug logging on the YBA node

Reason: GOOGLE_SDK_JAVA_LOGGING=true turns on the SDK HTTP logger; DEBUG on com.google.cloud and com.google.api ensures those lines reach application.log.

2a. Systemd environment — edit the YBA service file:

sudo vi /etc/systemd/system/yb-platform.service

Add under [Service]:

Environment="GOOGLE_SDK_JAVA_LOGGING=true"

2b. logback.xml — edit logback for your installed YBA version:

sudo vi /opt/yugabyte/software/<version/yb-platform/yugaware/conf/logback.xml

Add before the <root element:

<logger name="com.google.cloud" level="DEBUG" /
<logger name="com.google.api" level="DEBUG" /

2c. Restart YBA:

sudo systemctl daemon-reload
sudo yba-ctl restart

WARNING: Restarting YBA briefly interrupts the platform UI and in-flight tasks. Schedule this during a maintenance window on production controllers.

Step 3 — Reproduce and inspect

Delete an expired backup from the UI (or wait for the next garbage-collection cycle), then inspect the logs:

grep -E 'HttpTransport|backupGC|BackupGarbageCollector|successfully deleted' \
  /opt/yugabyte/yb-platform/yugaware/logs/application.log | tail -100

Expected (healthy) flow:

Step 4 — Map HTTP errors to fixes

Resolution

There is no single config change that fixes all deletion issues, use the diagnosis to choose the path.

1. IAM / permissions are the cause. Update the GCS service account on the backup config with list, get, and delete permission on the backup prefix.
   Reason: BackupGarbageCollector must list and batch-delete objects under the universe backup path; a missing storage.buckets.get or object-delete permission produces 403 in the SDK logs. Re-run deletion after IAM propagation (usually minutes).

2. Deletion succeeds in GCS but the UI still lists the backup. YBA metadata can retain rows when storage and DB state diverge.

   WARNING: Only proceed after confirming the objects are actually gone in GCS (SDK re-list shows an empty prefix). Work with Yugabyte Support for the metadata-cleanup SQL, do not run ad-hoc deletes on the YBA database without a support-approved runbook.

3. Only YBC / node-side files are involved. SDK logging on YBA will not show YBC GCS traffic. Collect YBC logs on the DB nodes:

   ls /home/yugabyte/controller/logs/

Verification

1. SDK logs show a full delete flow ending in an empty re-list for the backup prefix.
2. The YBA UI no longer lists the backup (or the task shows Completed).
3. Optional: the ybp_delete_backup_failure Prometheus metric does not increase after the fix.

Additional Information

Debug logging is a diagnostic aid, not a steady-state setting. After the investigation, disable it. It can generate GB/day of logs.

1. Remove Environment="GOOGLE_SDK_JAVA_LOGGING=true" from yb-platform.service.
2. Remove or comment the two DEBUG <logger lines in logback.xml.
3. Restart:

sudo systemctl daemon-reload
sudo yba-ctl restart

Limitations: This procedure is GCS-specific (the Java SDK). YBC uses a C++ client — use controller logs for those paths. Authorization headers are redacted in the logs, but still treat the logs as sensitive before sharing externally.