Environment

• YugabyteDB Anywhere - 2.14 and older
• YugabyteDB Anywhere - 2.20
• YugabyteDB Anywhere - 2024.1

Issue

Occasionally, the support or development teams may request a dump of raw metrics data for a Universe when analyzing an issue. This article describes how to export this data from the YugabyteDB Anywhere (YBA) Prometheus and upload it for analysis by the Yugabyte team.

Resolution

Overview

Yugabyte collects and stores metrics data in an instance of the open source Prometheus monitoring system running on the YugabyteDB Anywhere (YBA) node. The data can be extracted from the Prometheus database via API. The promdump binary described in this article can be used to easily export metrics data into JSON files that can later be imported into a compatible time series database for offline analysis.

You will need the following information:

• Hostname or IP address of the YBA node (default: localhost)
• Port number of the YBA Prometheus instance (default: 9090)
• Any two of (a) start time, (b) end time, and (c) period for which to collect data; if not overridden, the end time defaults to "now" and the period to 7 days
• A YBA API token and the name / UUID of the Universe for which to collect metrics --OR-- the "node prefix" value for the Universe if running promdump in legacy mode (example: yb-prod-appname)

Steps

1. Download the latest promdump binary (https://github.com/yugabyte/prometheus-remote-backfill/releases/latest) from the Yugabyte fork of the prometheus-remote-backfill project (or the attachment to this article) copy it to the YBA node (or another Linux system that can reach the YBA Prometheus instance over the network), extract it, and mark it executable:

gunzip promdump.gz
chmod +x promdump

The most recent binary is version 0.6.2 and was last updated on 2024-10-03. If you have a previous version of the binary, please download a fresh copy.

2. Collect information to be used in the promdump command. Examples are given below.

• The URL should be the hostname or IP address of the YBA node followed by a colon and the Prometheus port number.
• The name or UUID of the Universe. A list of available Universes will be printed if promdump is run with the --list_universes flag and a valid YBA API token.
• For legacy mode, the node prefix value can be found on the Nodes tab for the Universe. For example, if there is a node called yb-prod-appname-n1, the correct prefix value for this Universe is yb-prod-appname.
• Start or end times must be in RFC3339 format. For example, midnight Eastern Daylight Time on April 3, 2023 could be written as:

  2023-04-03T00:00:00-04:00

  It is strongly recommended to specify times in UTC. For example, the time above could also be written in UTC as:
  

  2023-04-03T04:00:00Z

• The period and batch values are normally written in hours or minutes. For example, to collect metrics for a single day, use a period of "24h".

NOTE: For systems managing large numbers of Universes and / or databases with many tables, a large batch size may cause Prometheus to run out of memory and crash while collecting tserver metrics. For busy systems, reduce the initial batch size. If too much data is returned for a query, promdump will automatically halve the batch size and retry, to a minimum of 1s batches and a maximum of 99,999 output files. If necessary, the --instances or --nodes flags can be used to limit collection to a particular set of Universe nodes. For example, specifying --nodes=1,3-7,14 would collect metrics for only nodes 1, 3, 4, 5, 6, 7, and 14. This may be useful for splitting up large datasets into more manageable pieces.

3. Run a command similar to the following to dump the Yugabyte metrics:

./promdump --url=http://yba.example.com:9090 \
 --yba_api_token=c340006e-0000-0000-0000-58c0000ee580 \
 --universe_name=myapp \
 --start_time=2023-04-03T04:00:00Z \
 --period=24h \
 --batch=15m

Be sure to update the parameters in the promdump command to match the information collected in step (2) above. The dumped metrics will be saved to files named node_export.00001, node_export.00002, master_export.00001, master_export.00002, etc. and the resulting files will be bundled into a compressed tar file at the end.

• If the YBA node is using a self-signed TLS certificate, the promdump command will fail with an error similar to "Get https://localhost/api/v1/customers: x509: certificate is not valid". Adding the flag --skip_yba_host_verification will disable certificate verification and allow promdump to connect. WARNING: This flag instructs promdump to accept the YBA node's TLS certificate regardless of validity, which may be a potential security risk. It is strongly recommended to use certificates issued by a certificate authority on production systems.
• If the command fails or produces only platform.* output files, check for typos in the command and confirm that the values of the supplied parameters are correct. The most common issue is an incorrect node_prefix value. Ensure that the system where promdump is running can reach the YBA Prometheus on port 9090.
• The "curl" utility can be used to test connectivity from the current host to the YBA API server and / or YBA Prometheus.

The following output shows a successful test connection to the YBA API server:

yugabyte@yba-host:~$ curl https://<yba_hostname> 
<!doctype html><html><head>...</html>

The following output shows a successful test connection to YBA Prometheus:

yugabyte@yba-host:~$ curl http://<prometheus_server_address>:9090
<a href="/graph">Found</a>.

The following shows an example of a failed test connection:

yugabyte@yba-host:~$ curl http://<prometheus_server_address>:9090
curl: (7) Failed to connect to yba-host port 9090: Connection refused

It is recommended to use the YBA API mode described above where possible. However, for systems with restricted access to YBA or limited YBA permissions, promdump can be run in "legacy" mode by specifying the --node_prefix value for the target Universe manually. For example:

./promdump --url=http://yba.example.com:9090 \
 --node_prefix=yb-prod-myapp \
 --start_time=2023-04-03T04:00:00Z \
 --period=24h \
 --batch=15m

3a. What to do if your prometheus is password protected

If your prometheus instance is password protected, it is now possible to add the password via a hidden file and an addition to the url. For example:

vi .prompw
export PROMPW="abc123"
(save and quit)
source .prompw
./promdump --url="https://promuser:${PROMPW}@ybahost.example.com:9090" \
--yba_api_token=c340006e-0000-0000-0000-58c0000ee580 \
--universe_name=myapp \
--start_time=2023-04-03T04:00:00Z \
--period=24h \
--batch=15m
rm .prompw

The PROMPW variable will be your password for prometheus - "abc123" is shown here for illustration. Note that the .prompw file should be deleted after use for security reasons.

4. Skip this step unless the Yugabyte support team has asked you to collect a custom metric. To collect a custom metric, use the --metric and --out parameters to specify the custom metric and output file prefix as shown in the example below:

./promdump --url=http://yba.example.com:9090 \
 --metric='{saved_name="follower_lag_ms",node_prefix="yb-prod-appname"}' \
 --out=custom_export \
 --start_time=2023-04-03T04:00:00Z \
 --period=24h \
 --batch=15m

Be sure to update the --metric parameter with the custom metric requested by support. Specifying both the --metric and --universe_nameflags in the promdump command will collect the standard Yugabyte metrics and the custom metric specified in the --metric parameter. For example:

./promdump --url=http://yba.example.com:9090 \
 --yba_api_token=c340006e-0000-0000-0000-58c0000ee580 \
 --universe_name=myapp \
 --metric='{saved_name="follower_lag_ms",node_prefix="yb-prod-appname"}' \
 --out=custom_export \
 --start_time=2023-04-03T04:00:00Z \
 --period=24h \
 --batch=15m

5. Upload the resulting tar file to your Zendesk ticket.