Environment:

Product: Platform 

Version: All

Issue:

On the universe home page, the overview metrics page found nothing to render 

Root Cause:

This issue can occur due to the below reasons:

• During Platform install using Replicated, the Prometheus container on the Yugaware host didn’t start.
• Firewall rules aren’t letting  the port 11000,12000,13000,9000,9300 and 7000 through

This may present as status unreachable as in the following screenshot:

Resolution:

• Confirm on the platform server the Prometheus container is running or not by running the below command

ubuntu@ip-10-9-107-81:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
646bcd50d627 10.9.107.81:9874/prom-prometheus:v2.2.1 "/bin/sh -c 'rm -rf …" 2 days ago Up 2 days 0.0.0.0:9090->9090/tcp prometheus

• If the Prometheus container is not running, see the steps in Solution 2 section and Solution 3 section in here to resolve this.
• Confirm that port 9090 is listening via the below command

ubuntu@ip-10-9-107-81:~$ netstat -tupln
tcp6 0 0 :::9090 :::* LISTEN - 

To troubleshoot if the port is causing the issue,  we need to check which port is unavailable using the below mentioned steps:

 Step 1: SSH tunnelling  using the below command:

ssh -i ~/.ssh/<pem key> centos@<platform Public IP> -L localhost:9090:<platform private IP>:9090

Step 2: Go to localhost:9090 and you will see the below screen as shown in the screenshot

Step3: Go to the tab " Status "   and from sub-task go to "Targets " as shown in the Screenshot

Step4: The Targets page will show  the port state (Up or Down) and also some other details as shown in the screenshot

Note: we can also troubleshoot the Prometheus state by using the below command

On platform server:

curl <private IP of DB node>:9300/metrics # node exporter metrics to confirm node exporter is running and port open

If no response from curl <ip>:9300.metrics then log onto the DB nodes and check if the node exporter process is running or not

The above curl command  can be also use to check for other endpoints:

port 7000 # master metrics
port 9000 # tserver
port 12000 # csql
port 13000 # ysql
port 11000 # redis