Environment

• YugabyteDB Anywhere - All

Issue

• YugabyteDB Anywhere (YBA) reported the node in a Starting or Stopping state but the master UI showed all node's as ALIVE
  

Resolution

Overview

In the universe_details_json, the affected node's state is set to Stopping or Starting. We will need to update this to a Live state to match the Master UI.

YBA reported the node in dead state but master UI showed all node's as ALIVE

Steps

1. To connect to the YBA Postgres:

$sudo docker exec -it postgres psql -U postgres yugaware 

2. Use the below command to get the universe name 

$select name, universe_details_json::jsonb->>'updateInProgress' from universe;

3. Check the state status for all the nodes

$select name, state, "isMaster", "nodeName", (ordinality - 1) as idx from universe, rows from (jsonb_to_recordset(universe_details_json::jsonb->'nodeDetailsSet') as (state text, "isMaster" bool, "nodeName" text)) with ordinality jsn_with_ordinality where name = '<Universe-Name>';

Note: The screenshot above shows nodes in a Live state, but your node may show in a Starting or Stopping state

4. Update the node(s) from a Starting or Stopping state to a Live state, consistent with the master UI

NOTE: In {nodeDetailsSet,n,state}  "n" = Idx value from the above command, This is for the whichever node the status being updated

$update universe set universe_details_json = jsonb_set(universe_details_json::jsonb, '{nodeDetailsSet,<n>,state}', '"Live"') where name = '<Universe-Name>';

OR  use the below command to update the status of all the Nodes in the Universe

$update universe set universe_details_json = REPLACE( universe_details_json, '"state":"<node-state>"', '"state":"Live"' ) where name = '<Universe-Name>';

Refresh your YBA UI and the node state will be updated.