• YB Anywhere - 2.6

Issue

When following the procedure "How to Rotate Custom CA-Signed Certificates" in the Yugabyte 2.6 documentation, the procedure fails at Step 4: "Edit the Universe to use the new certificate" with a NullPointerException. This issue happens when the existing certificate for the Yugabyte Universe is self-signed and the new certificate is CA-signed.

The following error message appears in the YB Anywhere logs:

2022-06-01 12:00:00,xxx [ERROR] from UpgradeUniverse in TaskPool-6 - Error executing task UpgradeUniverse(xxxxx-xxxx-xxxx-xxx-xxxxxxxx) with error={}.
java.lang.NullPointerException: null
    at com.yugabyte.yw.common.CertificateHelper.arePathsSame(CertificateHelper.java:439)
    at com.yugabyte.yw.commissioner.tasks.UpgradeUniverse.verifyParams(UpgradeUniverse.java:133)
    at com.yugabyte.yw.commissioner.tasks.UpgradeUniverse.run(UpgradeUniverse.java:180)
    at com.yugabyte.yw.commissioner.TaskRunner.run(TaskRunner.java:146)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
2022-06-01 12:00:00,xxx  [ERROR] from TaskRunner in TaskPool-6 - Error running task
java.lang.NullPointerException: null
    at com.yugabyte.yw.common.CertificateHelper.arePathsSame(CertificateHelper.java:439)
    at com.yugabyte.yw.commissioner.tasks.UpgradeUniverse.verifyParams(UpgradeUniverse.java:133)
    at com.yugabyte.yw.commissioner.tasks.UpgradeUniverse.run(UpgradeUniverse.java:180)
    at com.yugabyte.yw.commissioner.TaskRunner.run(TaskRunner.java:146)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
2022-05-24 11:07:00,748 [INFO] from TaskRunner in TaskPool-6 - Updating task [taskType : UpgradeUniverse, taskState: Running] to new state Failure

Root Cause

This issue is caused by a check in YB Anywhere that enforces unique custom filesystem paths for node certificates and node keys.  Self-signed certificates do not use custom paths, so this check fails with NullPointerException if the existing certificate is self-signed. The check is no longer required and has been removed in newer versions of the product.

Resolution

Permanent Fix

Upgrade YB Anywhere to version 2.8 or newer.

Workaround

If it is not possible to upgrade, use the following workaround to set the nodeCertPath and nodeKeyPath values in the YB Anywhere Postgres database entry for the affected certificate and bypass the affected check.

Important: Incorrect edits made to the YB Anywhere Postgres database may lead to serious issues. This procedure should only be used when recommended by the Yugabyte support team. Create a YB Anywhere backup before attempting to use this procedure.

Steps

1. Find the name of the certificate that is currently in use for the affected Universe.

Open the YB Anywhere UI, select the affected Universe from the Universe list, then select Actions > Edit Security. The name of the certificate that is currently in use will be displayed in the Select a root certificate field of the TLS Configuration dialog.

2. Log into the YB Anywhere node using SSH, then connect to the yugaware Postgres container:

sudo docker exec -it postgres psql -U postgres yugaware

3. Find the certificate information for the certificate. The following query will display the certificate details (where <certificate_name> is the certificate name from Step 1 above):

SELECT * FROM certificate_info WHERE label='<certificate_name>';

Confirm that this is the correct certificate, that the value of the cert_type column for this certificate is SelfSigned, and that the custom_cert_info column is empty. Do not proceed unless these values are as expected.

4. Update the custom_cert_info column for the old self-signed certificate to non-empty values to bypass the NullPointerError:

UPDATE certificate_info 
SET custom_cert_info = '{"nodeCertPath":"/opt/yugabyte/certs/bogus.crt","nodeKeyPath":"/opt/yugabyte/certs/bogus.key"}' 
WHERE label='<certificate_name>';

5. Confirm that the update has been applied:

SELECT * FROM certificate_info WHERE label='<certificate_name>';

The custom_cert_info column should now be populated for this certificate.

6. Reload the YB Anywhere UI and try to rotate the certificate again.