Environment

• YugabyteDB < 2024.2

Issue

The tserver process encountered a fatal error due to a failed kTruncate operation. The logs indicate that a truncate operation was initiated on a table, followed by a delete operation. The truncate attempt failed with an Apply failed: Illegal state (yb/tablet/tablet.cc:3268): Tablet was shut down, request: error because the target tablet had already begun shutting down or already shutdown.

F0314 14:22:24.776016 2736739 operation_driver.cc:407] T b8672ffdcf834de3b7a5adc354fb0bda P 24e0c0068ba546778c98fbc2f38d20ba S RD-P Ts { physical: 1741962103910990 } kTruncate (0x00003551aff226c0): Apply failed: Illegal state (yb/tablet/tablet.cc:3268): Tablet was shut down, request:

Additional Information:

• This issue is caused by a race condition where a truncate operation is committed but fails to apply if the tablet shutdown has already started or completed.
• The result is a fatal error leading to a tserver crash.
• This issue is addressed in YugabyteDB version 2024.2 as the truncate operation is re-implemented as a table re-write.

Resolution

Overview

To resolve this issue, upgrading to YugabyteDB version 2024.2 is recommended. If an upgrade is not immediately possible, a workaround can help mitigate the issue.

Steps

Upgrade to YugabyteDB Version 2024.2

• The issue has been fixed in 2024.2, where truncate is re-implemented as a table rewrite operation.
• This prevents the race condition that causes the tserver crash.

Workaround (If Upgrade is Not Immediately Possible)

1. Avoid initiating truncate operations on tables that are being deleted.
   • Ensure that truncation does not occur on tables in the process of being dropped.
2. Avoid concurrent TRUNCATE and DROP table queries.

Sample Logs

Truncate Operation Initiated

I0314 14:22:08.386984 1949188 client-internal.cc:719] Truncated table(s) 00004016000030008000000000012c26

Table Deletion Initiated

I0314 14:22:09.540261 2560340 client-internal.cc:665] Deleted table 00004016000030008000000000012c26

Tablet Shutdown Begins

I0314 14:22:24.768342 2736739 db_impl.cc:800] T b8672ffdcf834de3b7a5adc354fb0bda P 24e0c0068ba546778c98fbc2f38d20ba [R]: Shutting down RocksDB at: /data/yb-data/tserver/data/rocksdb/table-00004016000030008000000000012c26/tablet-b8672ffdcf834de3b7a5adc354fb0bda

Truncate Apply Failure

F0314 14:22:24.776016 2736739 operation_driver.cc:407] T b8672ffdcf834de3b7a5adc354fb0bda P 24e0c0068ba546778c98fbc2f38d20ba S RD-P Ts { physical: 1741962103910990 } kTruncate (0x00003551aff226c0): Apply failed: Illegal state (yb/tablet/tablet.cc:3268): Tablet was shut down, request: 

Additional Information:

• Reference GitHub issues:
  • Potential issues with tablet Shutdown and Truncate operations
  • YSQL- Add support for ALTER TABLE ... ADD COLUMN ... with table rewrite