Best practices for KMS provider migration and KMS high availability (Encryption at Rest) in YugabyteDB Anywhere

Overview

This guide explains how to migrate Encryption at Rest (EAR) between KMS providers in YugabyteDB Anywhere (for example, Azure Key Vault → HashiCorp Vault) and how to plan KMS high availability (HA). It is written for DBAs and Platform/SRE engineers who operate encrypted universes.

Prerequisites

• EAR is enabled. Confirm with:

  yb-admin -master_addresses $M is_encryption_enabled (prints Encryption status: ENABLED with key id <key_id)

• A full universe backup taken before migration
• The target KMS is reachable from YBA (transit engine, policies, tokens configured)
• The new KMS config validated in a non-production universe first
• No in-flight upgrade or restore on the target universes

Background

EAR uses envelope encryption. An external Master Key / Customer Master Key (CMK) held in the KMS wraps the Universe Keys stored in cluster metadata; the universe keys encrypt the data files on disk.

The KMS is contacted only at specific moments, master restart, universe-key rotation, KMS provider migration, and restore and not during steady-state reads and writes, because the universe keys are already loaded in master memory.

Recommendations

1. Migrate the KMS provider through YBA (re-wrap universe keys)

What to do:

1. Integrations → Security → Encryption At Rest → Create New Config for the target KMS.
2. Universe → Edit Security → Encryption at Rest → select the new config → Apply.

Why: YBA runs reEncryptActiveUniverseKeys  it decrypts each universe key with the old CMK and re-encrypts it with the new CMK, writing new kms_history rows. No data is re-written; only the key wrapping changes.

Risk / Trade-off: A wrong target KMS config, or losing access to the old CMK, makes old backups unrestorable. The earliest signal of trouble is a failed post-migration master-restart test.

Rollback: Keep the old KMS config in place; if needed, restore the universe from the pre-migration backup.

2. Retain old KMS configs until every backup that used them has expired

What to do: Do not delete the previous KMS config until all backups taken under it are past their retention period.

Why: A restore requires the exact KMS/CMK that was active when the backup was taken. Deleting the old config strands those backups.

Risk / Trade-off: Deleting the config too early blocks restore of historical backups.

3. Design KMS for high availability per provider

What to do:

Why: A master restart fails if the KMS is unreachable. HA keeps the KMS available for the restart/rotation/restore moments that actually call it.

Risk / Trade-off: For Vault specifically, a short-lived token expiring produces an "EAR config expiry" alert and can cause master-restart failures. Set a comfortable token TTL and monitor it.

4. Never rotate the master key and the universe key at the same time

What to do: Complete a master-key (KMS config) migration or a universe-key rotation one operation at a time, verified before starting the other.

Why: The platform does not support both operations together; overlapping them can leave key state inconsistent.

Quick Reference Table

Verification

• One master restart succeeds against the new KMS
• A universe-key rotation succeeds
• A new full backup completes after migration
• No EAR expiry alerts (HashiCorp Vault token path)

# 1. Confirm encryption is enabled and get the current key id
tserver/bin/yb-admin -master_addresses $M is_encryption_enabled

# 2. Confirm every master actually has that key loaded in memory
#    (replace <key_id with the id printed by step 1)
tserver/bin/yb-admin -master_addresses $M all_masters_have_universe_key_in_memory <key_id

These are the actual yb-admin commands and their real output:

• is_encryption_enabled prints: Encryption status: ENABLED with key id <key_id (or DISABLED).
• all_masters_have_universe_key_in_memory <key_id prints one line per master: Node <host:port has universe key in memory: true, and returns an error for any master that does not yet have it — run this against every master address, not just the leader.

Expected signal: ENABLED with the new config's key id, every master reporting true, and a successful restart test against the new provider.