Environment
- YugabyteDB Anywhere - 2.18.5
Issue
The YugabyteDB universe rolling restart fails with below error messages in YBA.
java.lang.RuntimeException: WaitForServer(045beb8a-b356-416e-850d-e3882ba50493, yb-test-cluster-n2, type=TSERVER) did not respond in the set time.
Cause
The YBDB (YugabyteDB) takes time to load n number of tablets and when it goes beyond 10 minutes the YBA (YugabyteDB Anywhere) marks the process as timed out which is hard-coded to 10 minutes.
Symptoms in YBA logs:
---------The tablet Loading Starts here:--------------
YW 2024-05-15T21:35:08.839Z [INFO] a34c3577-0716-4163-b207-7c8d8e53dbfa from WaitForServerReady in TaskPool-GFlagsUpgrade(546971f2-a64f-4cf2-97a3-059649b2e5d4)-0 - TSERVER on node yb-dev-cluster-n2 not ready after iters=0, error '4095 tablets not running out of 4123.'.
Check the no of tablet remains after 10 minutes.
---------Failed here after 10 minutes: -------------
java.lang.RuntimeException: WaitForServer(045beb8a-b356-416e-850d-e3882ba50493, yb-test-cluster-n2, type=TSERVER) did not respond in the set time.
This example shows that there were 174 tablets remaining to be loaded after 10 minutes, meanwhile, YBA marked the operation as timed out.
Resolution
Overview
To resolve the issue manually increase the YBA timeout using the parameter wait_for_server_timeout
to some larger value and restart the YBA.
Steps
For Replicated Based YBA Installations:
1. SSH into YBA node.
2. Access the yugaware container using following command:
docker exec -ti yugaware bash
3. Modify the application.docker.conf
file and insert the wait_for_server_timeout
to 900000 ms
as below:
yb {
.
.
health.check_interval_ms = 300000
health.status_interval_ms = 43200000
wait_for_server_timeout = 900000 ms <------ Here we added the timeout.
.
.
releases.path = "/opt/yugabyte/node-agent/releases"
}
4. Restart the yugware container.
sudo docker restart yugaware
5. Retry the Rolling Restart of Universe again.
For YBA-Installer Based Installations:
1. SSH into YBA node.
2. Open the file `/opt/yba-ctl/yba-ctl.yml` using Vi or any text editor.
3. Add the below property near the platform.additional which can be set as.
platform:
... # Other values here
.
.
# This section is for and additional data entries that you can
# add to platform.conf if you desire. Please enter the text
# exactly in the correct format starting from the lines below the
# piped delimiter, as it will be appended to the end of yb-platform.conf
# in the exact format typed.
# Example:
# yb {
# taskGC.task_retention_duration=60
# }
platform:
additional: |
yb {
wait_for_server_timeout=900000ms <--- Here we have added the timeout value.
}
# Config parameters for Postgres
# ==============================
...
4. Save the file run below command to reconfigure the YBA.
sudo yba-ctl reconfigure
Comments
0 comments
Please sign in to leave a comment.