Environment

• YugabyteDB Anywhere - All versions

Issue

• In the Yugabyte Anywhere Prometheus 

Resolution

Overview

The prometheus targets i.e http://<YBA_Host>:9090/targets shows the error in the UI "expected equal, got INVALID" which can also mark the Tserver/Master state bad in YBA > Universe > Node page. Though the DB nodes will keep functioning in the background. 

As a consequence of above this can mark the subsequent service unhealthy. 

Cause: 

This issue appears when you have some metrics which is not being correctly parsed by the prometheus and in return it marks the targets as bad. 

A very common example is if you have a non-conventional table name with double quotes like below:

yugabyte=# create table """yo"""(id int, full_name text);
CREATE TABLE
yugabyte=# \dt+ """yo""";
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------+-------+----------+---------+-------------
public | "yo" | table | yugabyte | 1024 kB |
(1 row)

yugabyte=#

This table will not be able to parsed by Prometheus produce the error attached in screenshot above in description section. 

How to verify the issue with metrics? 

The prometheus comes with an builtin tool called promtool which can validate the /metrics endpoint output. 

Here are easy steps to validate /metrics endpoint with promtool. 

• Copy the promtool from prometheus docker container to YBA host or if you have on local machine that would also work.

ubuntu@yba-host:~$ sudo docker cp prometheus:/bin/promtool .
ubuntu@yba-host:~$ ls -l promtool
-rwxr-xr-x 1 root root 110468077 Feb 1 07:58 promtool

•  Run the validation with endpoint. The endpoint in this case can be taken from http://<YBA_Host>:9090/targets . In this example the endpoint is: http://<YBA_Host>:9000/prometheus-metrics . So the command to validate would be: 

ubuntu@yba-host:~$ curl -ks https://HOST-IP:9000/prometheus-metrics | ./promtool check metrics
error while linting: text format parsing error in line 1765: unexpected end of label value ""

• As per output the error exist in the line no 1765 of the output where the "" is not parsable. 
• Redirect the output into some file and check the line no 1765. For example: 

ubuntu@yba-host:~$curl -ks https://HOST-IP:9000/prometheus-metrics > metrics.out

ubuntu@yba-host:~$ sed -n 1765p metrics.out
rocksdb_bytes_per_read_sum{metric_type="table",exported_instance="yb-dev-agangwar-test-n1",namespace_name="yugabyte",table_name=""yo"",table_id="000033f1000030008000000000004215"} 0 1687787115433

• You can see the culprit table which was made to produce the issue. 

Tip: The sed command syntax to show a particular line in the file. 

sed -n 'LINE_NUMBERp' FILE_PATH 

Resolution:

• Drop the table with unsupported name from database. 
• Rename the table and remove double quotes from name.