Skip to main content

Data Not Being Rebalanced Across Nodes

Duke
  • Updated

Environment/Cluster Versions Affected

All

Issue

On a newly added or on an existing tablet server (tserver), it may happen that the data is not being replicated/written into a tablet server.

Resolution

The resolution involves investigation of tserver status and the load balancer status.

Check tserver Status

1. Check whether the tserver process on the node is running. Log into the node via ssh and run the following command:

 /home/yugabyte/bin/yb-server-ctl.sh tserver status

If the tserver is running, you will get an output like following:

yb-tserver running with pid 10062

If the tserver is not running, you will get an output like following:

yb-tserver is not running

 

Start tserver (if not running)

1. Via the Platform UI, on the Nodes tab, in Actions dropdown, click Start Processes to start the processes on the node

actions_dropdown.png

2.  If you are NOT able start the tserver via the Actions dropdown,  you can start it with the following command on the node itself:

/home/yugabyte/bin/yb-server-ctl.sh tserver start
Starting yb-tserver on <hostname>

To verify whether the tserver has started or not, again the run the status command. You may also tail the yb-tserver.info logs in the /home/yugabyte/tserver/logs dir and look for any error that might be preventing the tserver from starting.

 

3. Check whether the load balancer is enabled using the following command:

yb-admin -master_addresses <MASTER_ADDRESS>:7100 get_load_balancer_state

Where <MASTER_ADDRESS> is the IP address of the master leader node.

The output of the following command would be similar to the output below:

Master UUID                       RPC Host/Port         State     Role      Load Balancer State
df1b1c81a1f34607b61bb80bb0c5f43b  172.151.42.98:7100    ALIVE     FOLLOWER  DISABLED
f3df905ba25e4576b39ef4cc3c9b617e  172.151.75.71:7100    ALIVE     LEADER    DISABLED
 

Check tserver Status

As we can see, the load balancer is disabled for this cluster. If the load balancer is disabled, the system will not migrate tablets from node to node. Therefore, we need to enable the load balancer to rebalance the tablets.

 

Enable Load Balancer

1. Enable the load balancer using the following command:
yb-admin -master_addresses <MASTER_ADDRESS>:7100 set_load_balancer_enabled 1 
Where <MASTER_ADDRESS> is the IP address of the master leader node.
 
This will enable the load balancer for all the machines. This can be verified using the get_load_balancer_state option of the yb-admin, as seen below:
yb-admin -master_addresses 172.151.75.71:7100,172.151.42.98:7100 get_load_balancer_state

Master UUID                       RPC Host/Port         State     Role      Load Balancer State
df1b1c81a1f34607b61bb80bb0c5f43b  172.151.42.98:7100    ALIVE     FOLLOWER  ENABLED
f3df905ba25e4576b39ef4cc3c9b617e  172.151.75.71:7100    ALIVE     LEADER    ENABLED
As we see from the output, the load balancer has been enabled.
 
We can see evidence of the load balancing via the Tablet Server UI at http://<platform_ip>/universes/<universe_uuid>/proxy/<node_ip>:7000/tablet-servers
 
load_balancing.png
 
While load balancing is in-progress, you will see movement in the User Tablet-Peers / Leaders tablets column.
 

Root Cause

When a new cluster is deployed, the load balancer is always enabled, and tablets are created and distributed across tservers.
 
However, in a case of a database upgrade, the load balancer is automatically disabled while the upgrade is in-progress. When the upgrade completes, the load balancer is automatically re-enabled.
 
If an upgrade issue causes the upgrade to fail, the load balancer remains disabled until manually re-enabled via the steps above.

Related articles

Comments

0 comments

Please sign in to leave a comment.