Backup failure with invalid relnamespace oid – Yugabyte

Environment:

YugabyteDB using YSQL API

Affects all product versions where YSQL is available

Issue:

Table backup fails with the below error see in the Yugabyte Anywhere UI.

 Unable to export snapshot 3c5d909b-0365-4a38-8224-7a00ce83b030 to file /tmp/yb_backup_cjvxydvhkgfeooin/SnapshotInfoPB: Not found or invalid relnamespace oid for table oid 17938

Cause:

This issue occurs when a DDL operation fails to complete, because of CPU or Network contention. Thus, the problem is most commonly seen on database clusters under high load, that frequently do DDL operations, but it may appear on any system when a DDL is done, although with a much lower probability.

Solution:

We have an improvement lined up in the product to handle this.

Workaround:

The mitigation of the issue is to do a manual repair following the below steps:

Steps to mitigate:

List user tables registered in the docdb, by YSQL database
```
yb-admin -master_addresses x.x.x.x:7100,x.x.x.x:7100,x.x.x.x:7100 list_tables include_table_id include_db_type | grep ysql | grep -v catalog
```
The output of the above would be <DB>.<TableName> <tableID> . e.g. ysql.yugabyte.sql_test 0000703960000500070000000000031g
For each table, check in Postgres metadata if the table exists. The below will return a row if it exists
```
bin/ysqlsh -c "SELECT relname, oid FROM pg_class where relname = '<TableName>' -d <DB>
```
If the above returns ANY rows, then contact Support for assistance. If the above does NOT return any rows, then use the TableID from Step 1 to delete the table from DocDB. This is to make sure that the table which has a reference in the DocDB and the one which does not have a reference in the Postgres has to be removed. To remove we use the below yb-admin commands.
For an Index use the delete_index_by_id option. The UUID obtained in Step 1 should be provided as an argument. eg: 0000703960000500070000000000031g2 is the UUID obtained from Step 1
```
yb-admin --master_addresses <master-addresses> delete_index_by_id 0000703960000500070000000000031g2
```
For a table use the delete_table_by_id option. Same as above, use the UUID received in Step 1 as an argument
```
yb-admin --master_addresses <master-addresses> delete_table_by_id 0000703960000500070000000000031g2
```
Once the above is done, please trigger a backup.

Diagnostic:

The error says the metadata for a specific entity is unavailable in the backend Postgres table. This typically happens if the entity creation or deletion fails abruptly leaving it in a broken state. The entity can be Table, View, or Index. At the time of backup, these details are looked upon, and it fails as the required metadata is unavailable.

The concurrent creation of indexes is one possible scenario that can lead to this error where one index creation succeeded and the other failed in the middle leaving the index in a broken state.