Universe Backup in YugabyteDB Anywhere fails with error 'storage.buckets.get' denied on resource

Environment

• YugabyteDB Anywhere - 2.20.x

Issue

After upgrading to version 2.20, backups fail with messages similar to the following appearing in the yb-controller logs:

E0425 19:06:38.739688 110768 cloud_store_google.cc:149] Unable to get bucket metadata: 
E0425 19:06:38.739766 110768 cloud_store_google.cc:150] Permanent error in GetBucketMetadata: yb-anywhere-cloud-provider@yb-support-298620.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist). 
E0425 19:06:38.739799 110768 cloud_store.cc:130] Invalid credentials E0425 19:06:38.739811 110768 cloud_store.cc:173] Verifying default cloud store failed!

Resolution

Overview

• This is due to a recent change in the yb-controller service used in YugabyteDB Anywhere aka YBA, where it now attempts to confirm the existence of the storage bucket before using it. This additional capability requires the permission storage.buckets.get.
• To test whether this permission is available to the service account or IAM role assigned to the instance:
  1. Log into the instance
  2. Authenticate using the service account (if using a service account)
  3. Run the following command to list the available accounts and confirm that the correct account is marked ACTIVE:
  

  gcloud auth list

  4. If the wrong account is ACTIVE, use the command gcloud config set account <account_name> to activate the correct account (where <account_name> is the name of the account that should be active).
  5. Run the following command to test whether the active account has the required storage.buckets.get permission:
  

  gsutil ls -L -b gs://bucketname

• If this command fails, the permissions on the account will need to be adjusted to add the storage.buckets.get permission. This change must be made in the GCP configuration. The Google Cloud Storage documentation describes how to Set and manage IAM policies on buckets.