Environment

• YugabyteDB - 2.6 and above.

Issue

Universe Backups fails, YBAnywhere Application log shows:

YW 2023-12-18T06:01:44.037Z [ERROR] eb8391b3-859d-4769-a0d7-27016f43c467 from CreateBackup in TaskPool-47 - Error executing task CreateBackup(21204a6d-2480-4bc9-9231-1e72f7cd9e23) 
with error='SubTaskGroup BackupTableYbc of type CreatingTableBackup at position 0: completed 1 out of 1 tasks failed.'.
java.lang.RuntimeException: SubTaskGroup BackupTableYbc of type CreatingTableBackup at position 0: completed 1 out of 1 tasks failed.
at com.yugabyte.yw.commissioner.TaskExecutor$RunnableTask.runSubTasks(TaskExecutor.java:1181)
at com.yugabyte.yw.commissioner.tasks.CreateBackup.run(CreateBackup.java:162)
at com.yugabyte.yw.commissioner.TaskExecutor$AbstractRunnableTask.run(TaskExecutor.java:839)
at com.yugabyte.yw.commissioner.TaskExecutor$RunnableTask.run(TaskExecutor.java:1052)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at com.yugabyte.yw.common.logging.MDCAwareRunnable.run(MDCAwareRunnable.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.yugabyte.yw.common.PlatformServiceException: Task id 4ab46cc6-40d6-4cee-b1f6-89e294c06a5b_PGSQL_TABLE_TYPE_uii_25b5745b-6f02-459d-b948-e656bf888f76 
status: Failed with error COMMAND_FAILED
at com.yugabyte.yw.commissioner.YbcTaskBase.handleTaskCompleteStage(YbcTaskBase.java:95)
at com.yugabyte.yw.commissioner.YbcTaskBase.pollTaskProgress(YbcTaskBase.java:66)
at com.yugabyte.yw.commissioner.tasks.subtasks.BackupTableYbc.run(BackupTableYbc.java:192)
at com.yugabyte.yw.commissioner.TaskExecutor$AbstractRunnableTask.run(TaskExecutor.java:839)
at com.yugabyte.yw.commissioner.TaskExecutor$RunnableSubTask.run(TaskExecutor.java:1251)

Corresponding YBController log shows:

/home/yugabyte/tserver/postgres/bin/ysqlsh --port=5433 --host=`ls /tmp/.yb.*/.s.PGSQL.*|grep -v 'lock'|head -n 1|xargs -L1 dirname` -d template1 -t -c "SELECT yb_catalog_version()" | xargs
+ xargs
ls /tmp/.yb.*/.s.PGSQL.*|grep -v 'lock'|head -n 1|xargs -L1 dirname
++ grep -v lock
++ ls /tmp/.yb.0.0.0.0:5433/.s.PGSQL.5433 /tmp/.yb.0.0.0.0:5433/.s.PGSQL.5433.lock
++ head -n 1
++ xargs -L1 dirname
+ /home/yugabyte/tserver/postgres/bin/ysqlsh --port=5433 --host=/tmp/.yb.0.0.0.0:5433 -d template1 -t -c 'SELECT yb_catalog_version()'
ERROR:  function yb_catalog_version() does not exist
LINE 1: SELECT yb_catalog_version()
               ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
check_error $? catalog_version
+ check_error 1 catalog_version
+ ret=1
+ shift
+ '[' 1 '!=' 0 ']'
+ echo 'catalog_version failed'
+ exit 1

• Additional Information: 
  
  • The Database was initially created in 2.4 or 2.6 YuygabyteDB Version.
  • The Database was subsequently upgraded to a newer version. (2.8 to 2.20).
  • The Backup is being scheduled for the first time.

Resolution

Overview

This issue is due to a bug in Clusters that were initially created in 2.4.x and 2.6.x versions. (Any older release < 2.4 (e.g., 2.2.7.0), or any newer release > 2.6 (e.g., 2.8.0.0) do not have this bug)

Due to the bug:

1. If pg_yb_catalog_version table isn’t empty
2. If pg_tablegroup table exists
3. If pg_stat_statements table exists and
4. If function jsonb_path_query exists

Then we skip running V1__3979__pg_yb_catalog_version.sql, V2__4525__pg_tablegroup.sql, V3__5478__pg_stat_statements.sql, V4__5408__jsonb_path.sql upgrade scripts as part of the Database Upgrade, this can be confirmed by running select * from pg_yb_migration on the Problematic Database(s).

In the output, any cluster/database that has the problem will not show the V1_, V2_, V3_, V4_ upgrade scripts getting applied, confirming the bug condition.

To fix the problem we need to applied the V1__3979__pg_yb_catalog_version.sql script to create the yb_catalog_version.

We dont have to run the V2__4525__pg_tablegroup.sql, V3__5478__pg_stat_statements.sql and V4__5408__jsonb_path.sql scripts as:

1. pg_tablegroup is obsolete and not needed anymore.
2. pg_stat_statements and jsonb_path_query already exist in the DBs. Executing the script will error out as the objects already exist.
   
   

Steps

Apply V1 Migration script: (needs to be done only once)

Things to Note:

• The path in Step 4 will be different based on your Installation location/version.
• Recommend running this first in Stage to ensure the issue is addressed. (Assuming the problem is reported in stage as well).
• Redo the steps in all the Databases that report the problem (By changing the DB Name in Step-2).

On the Master leader:

1. Start a YSQLSH Session. (~/tserver/bin/ysqlsh -h <hostname>)
2. Connect to the database. (\c template1)
3. Set ysql_upgrade_mode to true. (SET ysql_upgrade_mode TO true;)
4. Run the V1 Script. (\i /home/yugabyte/yb-software/yugabyte-2.18.3.0-b75-centos-x86_64/share/ysql_migrations/V1__3979__pg_yb_catalog_version.sql)

Additional Information

// Any information they can read about [Blog| Article | Doc page]