Environment

• YugabyteDB - Versions lower than 2024.x 

Issue

The YugabyteDB Anywhere fails to take a universe or table backup because a table enters DDL verification. The cluster uses the preview G-Flag ysql_ddl_rollback_enabled, which the current version may not fully support. This article explains the root cause, key observations, risks of using preview flags, and how to resolve the issue.

Issue Type 1: Backup Failure

Symptoms

During a schedule backup attempt, the following log message appears in the YB-Controller logs:

Table 00004b0000003000800000000000da2d is undergoing DDL verification, retry later

The corresponding YB-Master logs shows:

Operation failed. Try again (yb/master/ysql_ddl_handler.cc:380): Table is undergoing DDL transaction verification: <table name> [id=00004b0000003000800000000000da2d]

Cause

• A DDL operation triggered the table to enter the DDL verification state.
• The cluster used the preview G-Flag ysql_ddl_rollback_enabled.
• This preview flag may not function correctly without all associated fixes present in the current YugabyteDB version.

Note: Preview flags must be explicitly acknowledged before modification. Flags marked as “Preview” cannot be changed unless they are included in the allowed list. By adding them to this list, you accept the associated risks. Refer to the official CSV-formatted list of allowed preview flags: Preview Flags CSV

Background: DDL Verification Process

When a DDL operation is initiated:

1. The YB-Master starts a background task to poll the transaction status.
2. It compares the PostgreSQL catalog with the DocDB schema:
   • Match before change: The transaction is aborted.
   • Match after change: The transaction is committed.
3. Based on this, the system rolls forward or rolls back the changes.
4. Upon completion, the DDL verification state is cleared.

Important: If the background task fails or crashes, the verification state may persist indefinitely, blocking further DDL or restore operations.

Resolution

• The problematic table (00004b0000003000800000000000da2d) is manually deleted from the DocDB layer.
• This triggered the system to finalize the DDL verification and remove the stale state.
• A new backup was triggered successfully once the verification was cleared.

Permanent Resolution

Upgrade to the latest stable release in the 2024.x series to benefit from fully supported DDL atomicity and avoid issues related to incomplete DDL verification when using preview G-Flags.

Additional Information

This issue is not limited to backup operations. Any application or process performing similar DDL operations can also encounter the same error due to the DDL verification state. This includes:

• Application deployments with schema changes
• Automation pipelines triggering DDLs
• Manual DDL executions via client interfaces

To mitigate such failures:

• Avoid concurrent DDL and backup/restore tasks.
• Monitor for long-running or stuck DDL verifications.
• Refrain from using preview G-Flags in production unless fully tested.