Environment
- YugabyteDB Anywhere - All Versions
Issue
When xCluster replication is configured between two universes, running Stop Node operations on both universes simultaneously (or within a short time interval) causes both tasks to fail. Both universes become frozen with nodes stuck in "Stopping" state, and further operations are blocked.
Error Messages:
On one universe:
Task XClusterConfigUpdateMasterAddresses cannot be run because a previous task StopNodeInUniverse failed on the universe. Please retry the previous task first to fix the universe.On the other universe:
Universe <universe-uuid> is already being updated.Cause
When stopping a node on a universe with xCluster replication, YBA automatically attempts to update the xCluster configuration on the partner universe. If Stop Node tasks are initiated on both universes at the same time:
- Task on Universe A locks Universe A and tries to update Universe B
- Task on Universe B locks Universe B and tries to update Universe A
- Both tasks fail because each partner universe is already locked by its own running task
This creates a deadlock where both universes are frozen and neither task can complete. The "Retry Task" option in YBA UI will not resolve this issue due to the mutual blocking.
Resolution
This issue requires manual intervention to clear the frozen state of both universes.
Please contact Yugabyte Support to resolve this issue. The support team will manually update the universe and node states to restore normal operation.
Recommendation
When performing Stop Node or Start Node operations on universes with xCluster replication:
- Do not run stop/start node tasks on both universes simultaneously.
- Execute the operation on one universe at a time.
- Wait for the task to complete successfully before initiating operations on the partner universe.
Reference number: SUPPORT-883
Comments
0 comments
Please sign in to leave a comment.