Environment

• YugabyteDB Anywhere Affected Versions:

  • 2.18.X Branch: < 2.18.7.0-b30

  • 2.20.2 Branch: < 2.20.2.1-b2
  • 2.20.3.x Branch: < 2.20.3.0-b54

Issue

After an upgrade to 2.18, the user may notice hung tasks and that the tserver statuses on the Yugabyte Anywhere Nodes tab display a "-" for their uptime.

In the YB tserver WARNING logs, the user sees the message:

Received wrong universe_uuid <uuid1>, expected <uuid2>

In v2.18, we introduced a check using YB Master flag master_enable_universe_uuid_heartbeat_check. The error message above is a race condition that can occur relating to the flag, which prevents the nodes from heart-beating properly.

See TA-21491 for additional details.

Resolution

Overview

A race condition that can occur relating to master_enable_universe_uuid_heartbeat_check, which prevents the nodes from heart-beating properly.

As a workaround, we can disable master_enable_universe_uuid_heartbeat_check, and re-enable it once you upgrade to a fixed version.

Steps

Note: Follow these ONLY if you encounter the above issue as a result of the upgrade to 2.18

1. Set master_enable_universe_uuid_heartbeat_check to false

In the YBA UI, set the Master GFlag to master_enable_universe_uuid_heartbeat_check=false

When setting this, you can select the option on the GFlags UI that says Apply all changes which do not require a restart immediately; (a restart is not required for this GFlag)

Re-enabling the GFlag (once universe is upgraded to a release with the fix)

1. Run the following YB TS CLI command on all tervers to clear the UUID:

yb-ts-cli –server_address=<ip_address_or_hostname:9100> clear_universe_uuid

Expected output:

Universe UUID cleared from Instance Metadata

Note: If you see the following error, that is ok as it indicates the UUID is not set.

Unable to clear universe uuid on XXXX: Illegal state (yb/tserver/tablet_server.cc:1314): universe_uuid is not set in instance metadata: 0 vs 1

2. Re-enable the GFlag master_enable_universe_uuid_heartbeat_check by removing the GFlag override.

Additional Information

See TA-21491 for additional details.