Environment

• YugabyteDB - All versions

Issue

Enabling the Disk encryption agent like Vormetric Agent or similar software on YBDB nodes without stopping the Cluster could cause data to be corrupt or crash the YugabyteDB Master/T-Server process. 

Resolution

While enabling such agents please plan a downtime for cluster and stop the cluster before performing this activity. 

Follow the below sequence. 

1. Stop the YugabyteDB Master and T-Server process on all the nodes. 

NOTE: This is cluster down situation, hence you need a full downtime. 

2. To stop the process on all nodes, do SSH into all DB nodes and run below command.

/home/yugabyte/bin/yb-server-ctl.sh master stop
/home/yugabyte/bin/yb-server-ctl.sh tserver stop

3. After that enable the encryption agent and start the encryption agent service. Also perform any other action related to the encryption agent if needed. 

4. Lastly once the encryption agent is enabled and running fine. Start the Master service first and T-Server process on all the DB nodes. 

/home/yugabyte/bin/yb-server-ctl.sh master start
/home/yugabyte/bin/yb-server-ctl.sh tserver start

IMPORTANT

• If the cluster is systemd managed, please use `systemctl --user` command. Do not use `yb-server-ctl.sh`. For example below:

systemctl --user stop yb-master
systemctl --user start yb-master

• If the new node needs to be added. Make sure the agent is up and running before starting the YB processes.