Environment

• YugabyteDB Anywhere - 2.12 or newer

Issue

Starting in version 2.12, YugabyteDB Anywhere (YBA) introduced a new read / write health check for YSQL. This check connects to each tablet server, then inserts, selects, and deletes a row in a system table called read_write_test to verify that rows can be inserted and retrieved through the YSQL interface.

These checks may interfere with performance analysis, since these queries are included in performance analysis datasets such as those used by pg_stat_statements.

Resolution

Overview

The read / write health check can be disabled by setting the YBA runtime configuration setting yb.metrics.db_read_write_test to false. Currently, these settings can only be modified using the YugabyteDB Anywhere REST API. A UI for modifying these settings will be included in a future YugabyteDB Anywhere release.

Steps

1. Find the Customer UUID.

Log into YugabyteDB Anywhere. Make a note of the hostname or IP address and port number (if applicable) of the YugabyteDB Anywhere node.

Click the "User" icon in the upper-right corner of the YugabyteDB Anywhere UI, then select User Profile. Make a note of the UUID listed under Customer ID. You will need this in a later step.

2. Generate a REST API authentication token. If you already have a REST API authentication token, skip this step and use the existing value of this token.

In the User Profile section of the YugabyteDB Anywhere UI, select the Generate Key button, then the Copy button to copy the authentication token to the clipboard. 

Important: Generating a new REST API authentication token will invalidate any existing token for this user. Be sure to verify whether or not this token is in use before generating a new token.

Important: Store this authentication token securely. Anyone with access to this token can use it to make changes to the environment, including making configuration changes, creating or deleting database Universes, etc.

3a. To disable the read / write check for all Universes associated with a YBA instance, set the yb.metrics.db_read_write_test setting to false at the global scope:

curl --request PUT \
--url https://${YBA_HOST}/api/v1/customers/${CUUID}/runtime_config/00000000-0000-0000-0000-000000000000/key/yb.metrics.db_read_write_test \
--header 'Content-Type: text/plain' \
--header "X-AUTH-YW-API-TOKEN: $TOKEN" \
--data 'false'

where ${YBA_HOST} is the hostname or IP address of the YugabyteDB Anywhere host, ${CUUID} is the Customer UUID and  $TOKEN is the API token from step (1) above.

NOTE: Only the SuperAdmin user can modify runtime configuration variables at the global scope.

NOTE: The API will report the message Key yb.metrics.db_read_write_test is not defined in scope ${SCOPE} if this setting has not been set for the specified scope. This is expected behavior.

For example:

export YBA_HOST=$hostname
export CUUID=11111111-2222-3333-4444-777777777777
export TOKEN=<redacted>
curl --request PUT \
--url https://${YBA_HOST}/api/v1/customers/${CUUID}/runtime_config/00000000-0000-0000-0000-000000000000/key/yb.metrics.db_read_write_test \
--header 'Content-Type: text/plain' \
--header "X-AUTH-YW-API-TOKEN: $TOKEN" \
--data 'false'

3b. To disable the read / write check for a single Universe, set the yb.metrics.db_read_write_test setting to false at the Universe scope:

curl --request PUT \
--url https://${YBA_HOST}/api/v1/customers/${CUUID}/runtime_config/${UUUID}/key/yb.metrics.db_read_write_test \
--header 'Content-Type: text/plain' \
--header "X-AUTH-YW-API-TOKEN: $TOKEN" \
--data 'false'

where ${YBA_HOST} is the hostname or IP address of the YugabyteDB Anywhere host, ${CUUID} is the Customer UUID and  $TOKEN is the API token from step (1) above, and ${UUUID} is the UUID of the Universe for which you wish to disable the read / write check.

NOTE: The API will report the message Key yb.metrics.db_read_write_test is not defined in scope ${SCOPE} if this setting has not been set for the specified scope. This is expected behavior.

The UUID of a Universe can be found at the command line by using the List universes REST API call. Alternatively, the UUID can be found by browsing to the Universe using the YBA interface and examining the URI in the browser's address bar:

Example for disabling the read / write check for the Universe with UUID 4db59bfc-0000-0000-0000-17048f96cc74:

export YBA_HOST=$hostname
export CUUID=11111111-2222-3333-4444-777777777777
export UUUID=4db59bfc-0000-0000-0000-17048f96cc74
export TOKEN=<redacted>
curl --request PUT \
--url https://${YBA_HOST}/api/v1/customers/${CUUID}/runtime_config/${UUUID}/key/yb.metrics.db_read_write_test \
--header 'Content-Type: text/plain' \
--header "X-AUTH-YW-API-TOKEN: $TOKEN" \
--data 'false'

4. Run a GET to confirm that the yb.metrics.db_read_write_test setting has been applied as expected:

curl --request GET \
--url https://${YBA_HOST}/api/v1/customers/${CUUID}/runtime_config/${SCOPE}/key/yb.metrics.db_read_write_test \
--header 'Content-Type: application/json' \
--header "X-AUTH-YW-API-TOKEN: $TOKEN"

where ${SCOPE} is the scope at which the check was disabled (e.g. 00000000-0000-0000-0000-000000000000 for the global scope, the UUID of the corresponding Universe, etc.).

NOTE: The API will report the message Key yb.metrics.db_read_write_test is not defined in scope ${SCOPE} if this setting has not been set for the specified scope. This is expected behavior.

For example:

export YBA_HOST=$hostname
export CUUID=11111111-2222-3333-4444-777777777777
export SCOPE=4db59bfc-0000-0000-0000-17048f96cc74
export TOKEN=<redacted>
curl --request GET \
--url https://${YBA_HOST}/api/v1/customers/${CUUID}/runtime_config/${SCOPE}/key/yb.metrics.db_read_write_test \
--header 'Content-Type: application/json' \
--header "X-AUTH-YW-API-TOKEN: $TOKEN"
true

Additional Information

Note that runtime configuration settings for more specific scopes override settings for less specific scopes, meaning if you wish to disable the read / write health check for all Universes except one, this can be done by setting yb.metrics.db_read_write_test to false at the global scope, then setting yb.metrics.db_read_write_test to true at the Universe scope.

To revert the changes described above, follow the instructions in the "Revert a Setting Value to Default" section of the KB article How To Manage YugawareDB Anywhere Runtime Configuration Settings.