Environment

• YugabyteDB Anywhere - 2.8 and older
• YugabyteDB Anywhere - 2.12
• YugabyteDB Anywhere - 2.14
• YugabyteDB Anywhere - 2.16

Issue

Occasionally, the support or development teams may request a dump of raw metrics data for a Universe when analyzing an issue. This article describes how to export this data from the YugabyteDB Anywhere (YBA) Prometheus and upload it for analysis by the Yugabyte team.

Resolution

Overview

Yugabyte collects and stores metrics data in an instance of the open source Prometheus monitoring system running in a container in YBA. The data can be extracted from the Prometheus database via API. The attached promdump binary can be used to easily export metrics data into JSON files that can later be imported into a compatible time series database for offline analysis.

You will need the following information:

• Hostname or IP address of the YugabyteDB Anywhere (YBA) node (default: localhost)
• Port number of the YBA Prometheus instance (default: 9090)
• The "node prefix" value for the Universe (example: yb-prod-appname)
• Any two of (a) start time, (b) end time, and (c) period for which to collect data

Steps

1. Download the latest promdump binary (https://github.com/yugabyte/prometheus-remote-backfill/releases/latest) from the Yugabyte fork of the prometheus-remote-backfill project, copy it to the YBA node (or another Linux system that can reach the YBA Prometheus instance over the network), extract it, and mark it executable:

gunzip promdump.gz
chmod +x promdump

The most recent binary is version 0.1.3 and was last updated on 2023-04-20. If you have a previous version of the binary, please download a fresh copy.

2. Collect information to be used in the promdump command. Examples are given below.

• The URL should be the hostname or IP address of the YBA node followed by a colon and the Prometheus port number.
• The node prefix value can be found on the Nodes tab for the Universe. For example, if there is a node called yb-prod-appname-n1, the correct prefix value for this Universe is yb-prod-appname.
• Start or end times must be in RFC3339 format. For example, midnight Eastern Daylight Time on April 3, 2023 could be written as:

  2023-04-03T00:00:00-04:00

  It is strongly recommended to specify times in UTC. For example, the time above could also be written in UTC as:
  

  2023-04-03T04:00:00Z

• The period and batch values are normally written in hours or minutes. For example, to collect metrics for a single day, use a period of "24h".

NOTE: For systems managing large numbers of Universes, a large batch size may cause Prometheus to run out of memory and crash while collecting tserver metrics. For busy systems, reduce the initial batch size. If too much data is returned for a query, promdump will automatically halve the batch size and retry, to a minimum of 1s batches and a maximum of 99,999 output files.

3. Run a command similar to the following to dump the Yugabyte metrics:

./promdump --url=http://yba.example.com:9090 \
 --node_prefix=yb-prod-appname \
 --start_time=2023-04-03T04:00:00Z \
 --period=24h \
 --batch=15m

Be sure to update the parameters in the promdump command to match the information collected in step (2) above. The dumped metrics will be saved to files named node_export.00001, node_export.00002, master_export.00001, master_export.00002, etc.

• If the command fails or produces zero files, check for typos in the command and confirm that the values of the supplied parameters are correct. The most common issue is an incorrect node_prefix value. Ensure that the system where promdump is running can reach the YBA Prometheus on port 9090.
• The "curl" utility can be used to test connectivity from the current host to YBA Prometheus.

The following output shows a successful test connection:

yugabyte@yba-host:~$ curl http://prometheus_server_address>:9090
<a href="/graph">Found</a>.

The following shows an example of a failed test connection:

yugabyte@yba-host:~$ curl http://prometheus_server_address>:9090
curl: (7) Failed to connect to yba-host port 9090: Connection refused

4. Skip this step unless the Yugabyte support team has asked you to collect a custom metric. To collect a custom metric, use the --metric and --out parameters to specify the custom metric and output file prefix as shown in the example below:

./promdump --url=http://yba.example.com:9090 \
 --metric='{saved_name="follower_lag_ms",node_prefix="yb-prod-appname"}' \
 --out=custom_export \
 --start_time=2023-04-03T04:00:00Z \
 --period=24h \
 --batch=15m

Be sure to update the --metric parameter with the custom metric requested by support. Specifying the --node_prefix parameter to the promdump command will collect both the standard Yugabyte metrics and the custom metric specified in the --metric parameter. For example:

./promdump --url=http://yba.example.com:9090 \
 --node_prefix=yb-prod-appname \
 --metric='{saved_name="follower_lag_ms",node_prefix="yb-prod-appname"}' \
 --out=custom_export \
 --start_time=2023-04-03T04:00:00Z \
 --period=24h \
 --batch=15m

5. Compress the resulting dump files:

tar -cvzf promdumps.tar.gz *_export.*

6. Upload the resulting tarball to your Zendesk ticket.