Environment

• YugabyteDB- 2.16.0 

Issue

Yugabyte Anywhere reports health check failures with YB-Controller server error - "Error calling ping; Error: UNAVAILABLE". 

Resolution

Overview

The YB-Controller is a service that is running on each node of the cluster that communicates over gRPC to perform backup/restore operations.The above error could happen if the YB-Controller service is not running and the YB-Platform trigger a health check failure alert. 

Steps

1. Check if the YB-Controller is running or not

ps -ef | grep yb-controller

2. If the service is not running, start the YB-Controller using below command

/home/yugabyte/controller/bin/yb-controller-server --flagfile /home/yugabyte/controller/conf/server.conf &

This will mitigate the issue.

In 2.16.0 there is no cron check for yb-controller. So if the yb-controller stops, it is not getting restarted. This can be fixed by manually adding the cron check for yb-controller in the crontab

Steps to add the cron check for yb-controller:

1. To list the existing cron jobs, use the below command:

crontab -l

2. To add the cron check use the below commands: 

crontab -e

*/1 * * * * /home/yugabyte/bin/yb-server-ctl.sh controller cron-check || /home/yugabyte/bin/yb-server-ctl.sh controller start

This is fixed 2.16.1 and wont require manually adding the cron check for yb-controller.