Skip to main content

How to rotate certs with zero downtime (Hot Reload)

Joshua Martin
Updated

NOTE: Functionality for YBA to generate a new CA certificate in YBA UI is currently under review in feature request IDEA-2564. If implemented, certs will not need to be generated outside of YBA or through the REST API, and can be auto generated and downloaded from YBA UI instead. 

Environment

  • Yugabyte Anywhere: 2.16.6 and later (Recommended: 2.20.x or higher)
  • YugabyteDB Version: - N/A (Dependent on YBA version for management features)
     

Question

Suppose we need to rotate the Root Certificate Authority (CA) and/or Server Certificates for a YugabyteDB universe without incurring any downtime for the database nodes or the connected clients.

Standard certificate rotation (especially for YBA-generated certificates) often implies a rolling restart or a hot reload. While hot reloads can generate new CA certificates on the fly, there is still a problem where we have to import that CA to the client's trust store after the hot reload has already commenced, leading to potential connection issues for clients, who do not yet trust the new CA. 

Given this info, how can we achieve a true “zero downtime” rotation? 

Answer

This procedure allows us to generate the certificates ahead of time so they can be pre-staged on clients, Thus achieving a true "zero downtime" rotation

Overview

To achieve zero downtime during certificate rotation, two conditions must be met:

  1. Node Level: The database nodes must reload the new certificates without restarting the process. This is known as "Hot Reload".
  2. Client Level: Clients must trust the new Certificate Authority (CA) before the server switches to it, ensuring no connection failures occur during the transition.

Prerequisites for Hot Reload:

  • YugabyteDB Anywhere (YBA) version 2.16.6 or later.
  • Global Configuration (YBA runtime flag) yb.features.cert_reload.enabled must be true.
  • Universe Configuration (YBA runtime flag)  cert_hot_reloadable must be true.

General Approach:

  1. Verify Prerequisites: Ensure the universe is capable of Hot Reload.
  2. Generate New CA: Create a new self-signed CA and Private Key externally or with internal API.
  3. Pre-provision Clients: Add the new CA to the client's trust store (concatenated with the old CA) so they trust both.
  4. Import & Rotate in YBA: Upload the new certs to YBA and trigger the rotation using the "Hot Reload" (Non-Rolling) option.
  5. Cleanup: Remove the old CA from clients after the rotation is confirmed.

Steps

1. Verify Hot Reload Prerequisites

Before attempting a rotation without restarts, you must verify that the feature is enabled.

  • Check Global Flag: Navigate to Admin > Advanced > Global Config and search for yb.features.cert_reload.enabled. It must be set to true.
  • Check Universe Flag: You can verify if the specific universe is ready for hot reload via the API or by checking the config.
curl -X GET 'https://<YBA_HOST>/api/v1/customers/<CUSTOMER_UUID>/universes/<UNIVERSE_UUID>' \
  -H 'X-AUTH-TOKEN: <API_TOKEN>' | jq '.universeConfig.cert_hot_reloadable'

If this returns false or null, you must perform one Rolling Upgrade rotation first to enable this capability for future rotations.


Note: If cert_hot_reloadable is false, you cannot perform a hot reload immediately. You must schedule a maintenance window for one rolling restart rotation.

2. Generate New Self-Signed Certificates with openssql

To avoid client downtime, you need the new CA certificate before YBA rotates it. Generate it manually using openssl.

# Generate a new private key and a self-signed Root CA valid for 365 days
openssl req -newkey rsa:2048 -nodes -keyout yugabyte_private_key.pem -x509 -days 365 -out yugabyte_cert.pem
  • Common Name (CN): Enter your Organization or Universe identifier.
  • Organization: Enter your Organization name (e.g., Example Inc).

 

3. Update Clients to Trust the New CA

Clients (YCQL or YSQL) need to trust the new CA before the servers start using it.

  • Locate the current CA on the client machine (e.g., /etc/yugabyte/certs/root.crt).
  • Transfer the new CA (yugabyte_cert.pem) generated in Step 2 to the client machine.
  • Concatenate the certs to create a bundle trusting both:
# Example for YCQL or YSQL clients
cat /path/to/old/root.crt /path/to/new/yugabyte_cert.pem > /path/to/new/root_bundle.crt
chmod 600 /path/to/new/root_bundle.crt


# Update environment variable or config to point to the bundle
export SSL_CERTFILE=/path/to/new/root_bundle.crt
  • Restart/Reload Clients: Ensure application clients reload their configuration to pick up the new trust bundle.

4. Import Certificates to YBA

  1. In YBA, navigate to Universes > [Your Universe] > Actions > Edit Security > Encryption in-Transit.
  2. Select Add Certificate.
  3. Upload the yugabyte_cert.pem (Root CA) and yugabyte_private_key.pem (Private Key) generated in Step 2.
  4. Validate and Save.

5. Perform the Hot Rotation

  1. In the Encryption in-Transit settings, ensure the new certificate is selected.
  2. Click Select Upgrade Option and Apply.
  3. Deselect Rolling Upgrade. (Note: if cert_hot_reloadable is true, YBA will perform a hot reload regardless of this checkbox selection. Deselecting it avoids confusion in the task logs.)
    • Alternative UI Wording: Select Apply all changes which do not require a restart immediately.
  4. Click Apply.

YBA will now push the new certificates to the nodes and reload them without killing the database processes.

6. Cleanup Client Trust Store

Once the rotation is complete and verified:

  1. Update the client configuration to point solely to the new certificate (yugabyte_cert.pem).
  2. Remove the old certificate and the temporary bundle.
  3. Reload clients.
     
cp /path/to/new/yugabyte_cert.pem /etc/yugabyte/certs/root.crt
export SSL_CERTFILE=/etc/yugabyte/certs/root.crt

Additional Information

Related to

Related articles

Comments

0 comments

Please sign in to leave a comment.

Powered by Zendesk