Environment
- YugabyteDB Anywhere (All Versions)
- AWS/Azure/GCP
Issue
- One or more nodes in a universe are low on disk space
Resolution
Overview
If one or more nodes in a universe are low on disk space and need to be expanded, there are two options available:
- Edit the universe in YugabyteDB Anywhere - YBA and select a new instance type. The disk size for the new instances can be configured here.
- If the volumes are able to be expanded manually (such as an AWS EBS-backed volume) then this can be done outside of YBA. A manual edit to the YBA database will be needed afterward to update the universe details to the new correct value for the disk size.
Option 1: Change the instance type in YugabyteDB Anywhere's "Edit Universe" page
In the "Edit Universe" page, change the instance type to the new desired type. At the time of the change, a new disk size can be set. YBA will spin up new nodes alongside the old ones, migrate the data, and then remove the old nodes from the cluster. There should be no downtime during the migration process. However, the new nodes are stood up with new IP addresses, so anything that points to the database (load balancer, application, etc) will need to be updated with the IPs.
Procedure
- Navigate to "Universes" and select the applicable Universe. In the top right, click "Actions -> Edit Universe"
- Select the new instance type, along with the new disk configuration. Click "Save" at the bottom of the page to apply the new configuration.
Option 2: Expand the volume outside of YugabyteDB Anywhere and update it's database
If these volumes expandable (such as an AWS EBS-backed volume) then they can be expanded manually outside of YBA. The database cluster itself will benefit from the expanded disk immediately, but Platform will display the wrong information. You can update the YBA database manually with the new disk details, or simply ignore the incorrect value if you'd prefer.
Procedure
- Expand the volume manually (see links to each cloud provider's documentation below)
Optional: Update the YugabyteDB Anywhere database with the new volume size
- If the YBA database is going to be manually updated afterward, then a backup of the YBA database should be taken before making any changes. Be sure to use the
--exclude_prometheus
flag to keep the backup size small, since Prometheus data is not needed for this task. Do not attempt this procedure without a valid backup of the YBA database!
Backup procedure: https://docs.yugabyte.com/preview/yugabyte-platform/administer-yugabyte-platform/back-up-restore-yp/
- Modify the YBA database
We are specifically going to be modifying a column called "universe_details_json". As the name implies, this is a JSON blob with all of the universe details in it. We'll be replacing the "volumeSize" key with the new volume size.
- From the YBA node, enter the database:
$ sudo docker exec -it postgres psql -U postgres -d yugaware
- From the YBA node, enter the database:
-
-
The current value of "volumeSize" can be viewed with the following "select" statement:
yugaware=# select jsonb_pretty(universe_details_json::jsonb)
from universe
where name='universe-name';
-
The current value of "volumeSize" can be viewed with the following "select" statement:
-
-
Run the following command to update "volumeSize". In the command add a new value where it say <value> . Be sure to replace "universe_name" with the name of the universe that's being updated.
yugaware=#Begin;
begin
yugaware=# UPDATE universe
SET universe_details_json = jsonb_set(universe_details_json::jsonb,
'{clusters,0,userIntent,deviceInfo,volumeSize}', '<value>')
where name='universe-name';
UPDATE 1
yugaware=# Commit;
Commit
-
Run the following command to update "volumeSize". In the command add a new value where it say <value> . Be sure to replace "universe_name" with the name of the universe that's being updated.
-
-
Re-run the "select" statement to make sure that the new value has been applied:
yugaware=# select jsonb_pretty(universe_details_json::jsonb)
from universe
where name='universe-name';
-
Re-run the "select" statement to make sure that the new value has been applied:
Additional Information
Cloud provider documentation for expanding disks.
AWS
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/requesting-ebs-volume-modifications.html
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html
Azure
https://docs.microsoft.com/en-us/azure/virtual-machines/linux/expand-disks
GCP
https://cloud.google.com/compute/docs/disks/resize-persistent-disk
Comments
0 comments
Please sign in to leave a comment.