Environment

• YugabyteDB Anywhere - Versions greater than 2024.1.x.

  Note: In YBA 2024.1.x and later, this flag (use_memory_defaults_optimized_for_ysql) is enabled by default for new universes, but will remain disabled for existing  universes upgraded from older versions.

Issue

Customers using YCQL-only universes may experience failures or hangs during restore operations. The YugabyteDB Anywhere (YBA) UI will show the task as failed, and the logs will contain an IMPORT_SNAPSHOT_RPC_FAILED error, pointing to the cluster exceeding its safe tablet replica limit.

Resolution

Overview

The root cause is the use_memory_defaults_optimized_for_ysql g-flag, which is enabled by default in YugabyteDB Anywhere for all universe creations above 2024.1.z version. This flag adjusts memory allocation and tablet limits for YSQL-optimized performance. However, for YCQL-only universes, especially those with a large number of tables (a 'wide' schema), these YSQL-specific limits are too restrictive. An operation like a restore, which creates many new tablets, will breach these limits and fail. The solution is to disable this flag for YCQL universes, which reverts the tablet limit calculations to the older defaults suitable for YCQL workloads.

Steps

1. Verify the error in the YugabyteDB Anywhere (YBA) logs.

Navigate to the failed task in the YBA UI and inspect the error logs. Look for messages indicating that the tablet replica limit has been exceeded.

Caused by: java.lang.RuntimeException: Task id <task_id> status: Task failed due to failed RPC status IMPORT_SNAPSHOT_RPC_FAILED. Please check YB-Controller logs on node <node_ip> for more details

Further investigation into the YBC logs on the specified node will reveal the specific error:

I20250204 17:34:52.895344 1271 rpc_endpoint.h:72] The requested number of tablet replicas (216) would cause the total running tablet replica count (34200) to exceed the safe system maximum (33047)

E20250204 17:34:52.895401 1271 yb_api.h:83] Rpc: ImportSnapshot failed with 1

2. Navigate to the Universe Configuration in YBA.

• Select the universe experiencing the issue.
• Go to the Config tab.
• Click on Edit G-Flags.

3. Disable the YSQL-optimized memory flag.

• In the T-Server flags section, find the flag use_memory_defaults_optimized_for_ysql.
• Change its value from true to false.
• Click the Save button.

NOTE: Disabling this flag reverts the memory and tablet limit calculations to the defaults that are more suitable for YCQL workloads. This action does not remove tablet limits entirely but adjusts them to a more appropriate level for YCQL.

4. Re-run the restore task.

After saving the G-Flag change, the restore operation should now complete successfully.

Additional Information

An internal ticket (PLAT-16880) has been filed to change this default behavior in future releases for YCQL only universes. 

Relevant docs:

• https://docs.yugabyte.com/stable/architecture/docdb-sharding/tablet-splitting/#tablet-limits
  https://docs.yugabyte.com/stable/reference/configuration/yb-tserver/#memory-division-flags