Environment

• Yugabyte Platform (All Versions)
• AWS/Azure/GCP

Issue

• One or more nodes in a universe are low on disk space

Resolution

Overview

If one or more nodes in a universe are low on disk space and need to be expanded, there are two options available:

1. Edit the universe in Yugabyte Platform and select a new instance type. The disk size for the new instances can be configured here.
2. If the volumes are able to be expanded manually (such as an AWS EBS-backed volume) then this can be done outside of Platform. A manual edit to the Platform database will be needed afterward to update the universe details to the new correct value for the disk size.

Option 1: Change the instance type in Platform's "Edit Universe" page

In the "Edit Universe" page, change the instance type to the new desired type. At the time of the change, a new disk size can be set. Platform will spin up new nodes alongside the old ones, migrate the data, and then remove the old nodes from the cluster. There should be no downtime during the migration process. However, the new nodes are stood up with new IP addresses, so anything that points to the database (load balancer, application, etc) will need to be updated with the IPs.

Procedure

• Navigate to "Universes" and select the applicable Universe. In the top right, click "Actions -> Edit Universe"
  
  

• Select the new instance type, along with the new disk configuration. Click "Save" at the bottom of the page to apply the new configuration.

Option 2: Expand the volume outside of Platform and update Platform database

If these volumes expandable (such as an AWS EBS-backed volume) then they can be expanded manually outside of Platform. The database cluster itself will benefit from the expanded disk immediately, but Platform will display the wrong information. You can update the Platform database manually with the new disk details, or simply ignore the incorrect value if you'd prefer.

Procedure

• Expand the volume manually (see links to each cloud provider's documentation below)

Optional: Update the Yugabyte Platform database with the new volume size

• If the Platform database is going to be manually updated afterward, then a backup of the Platform database should be taken before making any changes. Be sure to use the --exclude_prometheus flag to keep the backup size small, since Prometheus data is not needed for this task. Do not attempt this procedure without a valid backup of the Platform database!
  Backup procedure: https://docs.yugabyte.com/preview/yugabyte-platform/administer-yugabyte-platform/back-up-restore-yp/
  
  
• Modify the Platform database
  
  We are specifically going to be modifying a column called "universe_details_json". As the name implies, this is a JSON blob with all of the universe details in it. We'll be replacing the "volumeSize" key with the new volume size.
  
  • From the Platform node, enter the database:
    

    $ sudo docker exec -it postgres psql -U postgres -d yugaware

• • The current value of "volumeSize" can be viewed with the following "select" statement:
    

    yugaware=# select universe_details_json from universe;

• • Run the following command to update "volumeSize". In the command add a new value where it say <value> . Be sure to replace "universe_name" with the name of the universe that's being updated.
    

    yugaware=#Begin;
    begin
    yugaware=# UPDATE universe 
    SET universe_details_json = jsonb_set(universe_details_json::jsonb, 
    '{clusters,0,userIntent,deviceInfo,volumeSize}', '"<value>"') 
    where name='universe-name';
    UPDATE 1
    yugaware=# Commit;
    Commit

• • Re-run the "select" statement to make sure that the new value has been applied:
    

    yugaware=# select universe_details_json from universe;

Additional Information

Cloud provider documentation for expanding disks.

AWS

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/requesting-ebs-volume-modifications.html
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html

Azure

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/expand-disks

GCP

https://cloud.google.com/compute/docs/disks/resize-persistent-disk