Node-Agent Health Check failure in YugabyteDB Anywhere with error CERTIFICATE_VERIFY_FAILED – Yugabyte

Environment

YugabyteDB Anywhere - All Versions below 2025.x

Issue

During a node check from YugabyteDB Anywhere (YBA), the following error was encountered:

Node check failed: java.lang.RuntimeException: Error occurred. Code: 137.
...
Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.

Symptoms

Node-Agent logs include the following entries:

2025-03-10 07:36:42,532 INFO: Using RPC connection to 10.1.1.1:9070
2025-03-10 07:36:42,532 INFO: RPC time-out is set to 1799 secs
E0310 07:36:42.768028538  490250 ssl_transport_security.cc:1421] Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
E0310 07:36:43.550401412  490250 ssl_transport_security.cc:1421] Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.

This indicates a failure in the SSL handshake due to certificate verification errors.

Check Certificate Validity:

To verify the certificate's expiry date, run:

openssl x509 -in /opt/yugabyte/node-agent/certs/<uuid>/node-agent.crt -noout -enddate

This helps confirm whether the certificate has expired or is nearing expiration.

Cause

This issue is caused by the expiration of Node-Agent certificates. By default, Node-Agent certificates are valid for one year, and earlier versions of YBA do not provide a UI option to rotate them.

Note: This behavior has been addressed in YBA version 2025.x and later under Jira ID PLAT-17056, where certificate rotation is handled automatically in the background before expiration.

Resolution

1. Verify Node-Agent Binary

On each affected DB node, confirm the Node-Agent binary exists:

ls -l /opt/yugabyte/node-agent/pkg/bin

Expected output:

node-agent
node-agent-installer.sh

2. Stop Node-Agent Service

Run the following command as a sudo or root user:

systemctl stop yb-node-agent.service

3. Re-register Node-Agent

Use the commands below to re-register the Node-Agent with YBA:

node-agent node unregister -t <api_token> -u <https://yba-ip-address>
node-agent node register -t <api_token> -u <https://yba-ip-address>

Replace <api_token> and <https://yba-ip-address> with your actual API token and YBA IP address.

4. Start Node-Agent Service

Once registered, restart the service:

systemctl start yb-node-agent.service

Validation

Verify that the Node-Agent service is running without errors.
Confirm from the YBA UI or by checking the Node-Agent logs.