Environment

Yugabyte database - 2.2 or newer.

Issue

DML queries fail with the following messages

ERROR: Query error: schema version mismatch for table <UUID>: expected 3, got 2

ERROR: Query error: Catalog Version Mismatch: A DDL occurred while processing this query. Try Again.

Resolution

• Users can safely retry the query when schema or catalog version mismatch occurs.
• Run DDL statements sequentially instead of running with multiple sessions.

Overview

A DML query in YSQL may touch multiple servers, and each server has a Catalog Version which is used to track schema changes. When a DDL statement runs in the middle of the DML query, the Catalog Version is changed and the running query has a mismatch, causing it to fail.

During SQL schema change operation, the catalog version is incremented on the master leader which gets applied to all the nodes by updating the Postgres layer system catalog cache. Since, the in-flight running DML operation is using an old schema version due to an outdated cache referring to the old catalog version, the operation errors out with version mismatch.

Note that we have a single global counter that gets incremented whenever there is a breaking change anywhere; while simple, this does not allow us to distinguish cases where the breaking change happened in a different database. So a DDL change in a different databae can also cause schema or catalog version mismatch errors.

We are actively working on reducing the impact of cross-database changes, such that the changes in one database would not impact operations in another database.

For the moment the workaround is to wait until the DDL changes finishes and try again.

Diagnostic Steps

Catalog Version Mismatch

## Session 1

yugabyte=# create table t1(a int primary key, b varchar);
CREATE TABLE
yugabyte=# create table t2(a int primary key, b varchar);
CREATE TABLE

## Session 2

yugabyte=# alter table t1 drop column b;
ALTER TABLE

## Session 1

yugabyte=# begin;
BEGIN
yugabyte=# insert into t2(a,b) values(630,'CHI');
ERROR: Catalog Version Mismatch: A DDL occurred while processing this query. Try again.
DETAIL: Internal error: column "b" of relation "t2" does not exist

#Retry
yugabyte=# begin;
BEGIN
yugabyte=# insert into t2(a,b) values(630,'CHI');
INSERT 0 1

Additional Notes

Please see existing ongoing work on the issue Eliminate Catalog Version Mismatch error and Catalog Version Mismatch should be a retryable.