Environment

• Yugabyte Platform - All versions
  

Issue

If an internal security policy mandates rotation of the SSH keys for an On-Premises Provider, this process can be followed to manually rotate the SSH keys.

Resolution

Overview

The overall steps to manually rotate SSH keys are:

1. Generate new SSH keys
2. Move new keys to the appropriate places in the Platform container and nodes
3. Update the Platform database with the new key path
4. Confirm the new keys work successfully
5. Delete the old keys and remove them from the Platform database

Note: Key rotation through the Platform UI is planned for a future release.

Steps

Gather information from Platform postgresql Container

1. From the Platform node, enter the postgresql container and connect to the postgresql database

sudo docker exec -it postgres psql -U postgres yugaware

2. Retrieve Customer UUID

select * from customer;

3. Retrieve Provider UUID from YB Platform, where <customer_uuid> is the Customer UUID from step (2) above:

select uuid, code, name from provider where customer_uuid = '<customer_uuid>';

4. Retrieve the Region details from YB platform, where <provider_uuid> is the Provider UUID from step (3) above:
Note: You will use the region code later

select * from region where provider_uuid = '<provider_uuid>';

5. Retrieve location of current SSH keys in YB Platform container:

Note: Take note of the sshUser, sshPort, key paths, and other values returned. Later on, you will insert a new record with the same values, but with the new key paths

select * from access_key where provider_uuid = '<provider_uuid>';

Generate New SSH keys and Add Them to Platform DB

Note: If you have already generated your own keys outside of the Platform node, use a secure copy tool such as scp to move the keys to the platform node and docker cp to move them into the Platform container in the paths shown in step (3), then continue the procedure at step (4). Otherwise, follow the steps here to generate new keys inside the container

1. Enter the Platform container

sudo docker exec -it yugaware /bin/bash

2. Generate new keys

ssh-keygen -b 2048 -t rsa

3. Move the new keys to the appropriate locations:

mv <new_key_name> /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pem
mv <new_key_name>.pub /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pub

4. Set ownership and permissions on both keys:

chmod 400 /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pem
chmod 400 /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pub
chown root /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pem
chown root /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pub

5. For any region create vault and vault_password files:

Note: You retrieved the region code earlier in step 4 of the previous section

/opt/yugabyte/devops/bin/ybcloud.sh onprem --region <region_code> access create-vault --private_key_file /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pem

6. From the Platform node, enter the postgresql container and connect to the postgresql database

sudo docker exec -it postgres psql -U postgres yugaware

7. Add database entry for the new keys:

Note: Refer to the record returned from the access_key information gathering steps. Match values, as needed (sshUser, sshPort, airGapInstall, etc.)

insert into access_key (key_code, provider_uuid, key_info) values ('<new_key_name>', '<provider_uuid>', '{"publicKey":"/opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pub","privateKey":"/opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pem","vaultPasswordFile":"/opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.vault_password","vaultFile":"/opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.vault","sshUser":null,"sshPort":54422,"airGapInstall":false,"passwordlessSudoAccess":true,"provisionInstanceScript":"","installNodeExporter":true,"nodeExporterPort":9300,"nodeExporterUser":"prometheus","skipProvisioning":false}');

Update DB nodes to use both old and new keys

1. Generate authorized_keys file with both public keys

Note: old key path may be different than shown below. See access_key information gathering step above.

cd /tmp
rm authorized_keys
cat /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<old_key_name>.pub <(echo) /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pub > authorized_keys

2. For each node, copy keys from the platform container to the node

Note: This will completely overwrite your current authorized_keys file with just the old key and the new key. If you have additional custom keys, please adjust the command accordingly.

scp -i /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<old_key_name>.pem -P <ssh_port> authorized_keys <ssh_user>@<node_ip>:/home/<ssh_user>/.ssh
scp -i /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<old_key_name>.pem -P <ssh_port> authorized_keys yugabyte@<node_ip>:/home/yugabyte/.ssh

3. For each node, run a command such as the "date" command to confirm you can connect to the node using the old keys

ssh -o "IdentitiesOnly=yes" -i /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<old_key_name>.pem -ostricthostkeychecking=no -p <ssh_port> yugabyte@<node_ip> 'date'
ssh -o "IdentitiesOnly=yes" -i /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<old_key_name>.pem -ostricthostkeychecking=no -p <ssh_port> <ssh_user>@<node_ip> 'date'

4. For each node, run a command such as the "date" command to confirm you can connect to the node using the new keys

ssh -o "IdentitiesOnly=yes" -i /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pem -ostricthostkeychecking=no -p <ssh_port> yugabyte@<node_ip> 'date'
ssh -o "IdentitiesOnly=yes" -i /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pem -ostricthostkeychecking=no -p <ssh_port> <ssh_user>@<node_ip> 'date'

Update Platform DB to use new keys

1. From the Platform node, enter the postgresql container and connect to the postgresql database

sudo docker exec -it postgres psql -U postgres yugaware

2. Update the Universe Details to use the new key

update universe set universe_details_json = replace(universe_details_json, '"accessKeyCode":"<old_key_name>"', '"accessKeyCode":"<new_key_name>"') where universe_details_json like '%"provider":"<provider_uuid>"%';

3. Delete the old key from the access key record

delete from access_key where provider_uuid = '<provider_uuid>' and key_code = '<old_key_name>';

4. Make sure next health checks are successful for affected universes

5. From the Platform node, make sure the new key is used for health checks

sudo docker logs yugaware

6. Make sure provider properties in Platform UI show the new key:

Update DB nodes to use only new keys

1. For each node, remove the old key from the authorized_keys file

scp -i /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pem -P <ssh_port> /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pub <ssh_user>@<node_ip>:/home/<ssh_user>/.ssh/authorized_keys
scp -i /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pem -P <ssh_port> /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pub yugabyte@<node_ip>:/home/yugabyte/.ssh/authorized_keys

Note: This copies over the authorized_ keys file with ONLY the new key.

2. Make sure you can NOT connect the node using the old keys

# This should fail because the old key is used

ssh -o "IdentitiesOnly=yes" -i /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<old_key_name>.pem -ostricthostkeychecking=no -p <ssh_port> yugabyte@<node_ip> 'date'
ssh -o "IdentitiesOnly=yes" -i /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<old_key_name>.pem -ostricthostkeychecking=no -p <ssh_port> <ssh_user>@<node_ip> 'date'

3. Make sure you CAN connect the node using the new keys

# This should succeed because the new key is used

ssh -o "IdentitiesOnly=yes" -i /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pem -ostricthostkeychecking=no -p <ssh_port> yugabyte@<node_ip> 'date'
ssh -o "IdentitiesOnly=yes" -i /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<new_key_name>.pem -ostricthostkeychecking=no -p <ssh_port> <ssh_user>@<node_ip> 'date'

4. Make sure the next health check for the universe is successful:

Delete old keys from Platform node

1. Remove the old keys

Note: old key paths may be different than shown below. Use old key locations identified in access_key information gathering step.

rm /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<old_key_name>.pub
rm /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<old_key_name>.pem
rm /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<old_key_name>.vault
rm /opt/yugabyte/yugaware/data/keys/<provider_uuid>/<old_key_name>.vault_password

For Providers that use the pre-provisioning script to manually provision nodes

1. Enter the Platform container

sudo docker exec -it yugaware /bin/bash

2. Create a copy of the manual provisioning script

cp /opt/yugabyte/yugaware/data/provision/<provider_uuid>/provision_instance.py /opt/yugabyte/yugaware/data/provision/<provider_uuid>/old_provision_instance.py.bak

3. Edit /opt/yugabyte/yugaware/data/provision/<provider_uuid>/provision_instance.py, updating references to the ssh and vault files (Some references highlighed in screenshot below):