Environment

• Yugabyte Platform - All versions 

Issue

Certain restore operations issued via Yugaware are failing with Python `code=77` and  `S3 error 403`

• Example:

java.lang.RuntimeException: java.lang.RuntimeException: {"error": "Exception: Command '['ssh', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'ControlMaster=auto', '-o', 'ControlPath=~/.ssh/ssh-%r@%h:%p', '-o', 'ControlPersist=1m', '-i', '/opt/yugabyte/yugaware/data/keys/$key.pem', '-p', '54422', '-q', 'yugabyte@10.1.8.200', \"cd / &&  bash -c 's3cmd --force --no-check-certificate --config=/tmp/yb_backup_xyz/cloud_cfg get s3://backup/yugabyte/univ-abc-123-efg-456/backup-2022-02-07T10:01:41-110558009/table-keyspace.table_settings_1-f6/SnapshotInfoPB.sha256 /tmp/yb_backup_jobgsjcxkyxucbtx/SnapshotInfoPB.sha256.downloaded'\"]' returned non-zero exit status 77."}.

Resolution

Overview

Yugaware restore operations interacting with S3, operate under the assumption the object from the S3 backup location has not been migrated off to Glacier storage. 

Steps

1. Using a GET request with the S3 command, review the ObjectClass in the response body to see if the object is has a Bucket Lifecycle Policy that would migrate the data to Glacier storage. 

    EX REQUEST

GET /?lifecycle HTTP/1.1
Host: examplebucket.s3.<Region>.amazonaws.com
x-amz-date: Thu, 15 Nov 2012 00:17:21 GMT
Authorization: signatureValue

    EX RESPONSE

<Status>Enabled</Status>  =====> IS IT ENABLED ? 
...
<Transition> <Days>365</Days> =======> Instruction to transition to Glacier after 365 days 
<StorageClass>GLACIER</StorageClass>
 </Transition> <Expiration> <Days>3650</Days>

2. If the response data reflects a LifecycleConfiguration policy that has migrated the object to Glacier storage, the object must first be restored from S3 Glacier storage to the S3 Bucket storage. 

Additional Information

Yugabyte is working to improve logging messages for Platform operations. In this particular instance, we see the code=77 as the return code from Python. This is the error that is reflected in the messages area in Platform, when selecting`View More` from platform task page. This error translates to Permission Denied . This is consistent with the S3 error 403 which is an S3 permissions issue..