How to resolve master_leader_stepdown fails with error "Suggested peer is not caught up yet" when moving the YB-Master leader

Problem

Running yb-admin master_leader_stepdown to move YB-Master (the process that stores cluster metadata and coordinates the cluster) leadership fails.

Error / alert observed:

LEADER_NOT_READY_TO_STEP_DOWN
message: "Suggested peer is not caught up yet"

Or:

Leader not ready to step down as there are N active config peers in transition

Impact: Rolling upgrades, node decommission, or planned maintenance that requires moving master leadership stalls until leadership can be transferred safely.

Cause

Raft (the consensus protocol YB-Master uses to keep the three metadata replicas consistent) refuses to hand leadership to a follower whose log is behind the leader's committed index. Transferring to a lagging peer could lose committed metadata, so Raft blocks it.

list_all_masters can show a peer as ALIVE / FOLLOWER while it is still far behind — there is no lag column in the current output (GHI-28675), so a healthy-looking follower may still be catching up.

Common triggers:

• A master data directory was wiped or its Kubernetes PVC was lost, so the peer is doing a full remote bootstrap.
• Network stall or slow disk on a follower, leaving it behind on replication.
• WAL (write-ahead log) on the follower was garbage-collected, forcing a slower catch-up.
• A peer is in PRE_VOTER state during an in-flight config change.

Diagnosis

Confirm the cluster state before taking any corrective action:

• All masters report ALIVE in list_all_masters
• No peer is stuck in PRE_VOTER (check the Master UI at :7000)
• No in-flight change_master_config operation
• Port 7100 (master RPC) is reachable between all masters
• If the cluster uses TLS, --certs_dir_name is supplied on every yb-admin command

yb-admin --master_addresses <m1:7100,<m2:7100,<m3:7100 list_all_masters

Output when the issue is present: The stepdown command returns Suggested peer is not caught up yet or peers in transition, even though list_all_masters shows all peers ALIVE.

% ./bin/yb-admin --master_addresses 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 list_all_masters
Master UUID                      	RPC Host/Port        	State    	Role 	Broadcast Host/Port 
af08844be93d4cdf9e0b94858fe33675 	127.0.0.1:7100       	ALIVE    	LEADER 	N/A                 
8bff6598e2624fbdbd20000c5dde8f0f 	127.0.0.2:7100       	ALIVE    	FOLLOWER 	N/A                 
6e9269eaa24740eaa5bc7bccda343917 	127.0.0.3:7100       	ALIVE    	FOLLOWER 	N/A 

The node3 looks like a healthy FOLLOWER but if you try to promote it to LEADER:

% ./bin/yb-admin --master_addresses 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 master_leader_stepdown 6e9269eaa24740eaa5bc7bccda343917
E0923 21:02:23.128075 47841792 yb-admin_client.cc:729] LeaderStepDown for af08844be93d4cdf9e0b94858fe33675received error code: LEADER_NOT_READY_TO_STEP_DOWN status { code: ILLEGAL_STATE message: "Suggested peer is not caught up yet" source_file: "../../src/yb/consensus/raft_consensus.cc" source_line: 851 errors: "\000" }
Error running master_leader_stepdown: Illegal state (yb/consensus/raft_consensus.cc:851): Suggested peer is not caught up yet

Resolution

1. Let Raft auto-elect (preferred). Omit the target UUID so Raft transfers leadership to the most caught-up peer instead of a follower you name manually.

   yb-admin --master_addresses <masters \
     --certs_dir_name /home/yugabyte/yugabyte-tls-config \
     master_leader_stepdown

   Reason: Naming a specific follower that is still replicating is the most common cause of this error. Auto-election picks a peer that satisfies the Raft catch-up requirement.

2. Wait and retry. If a master recently restarted, give replication time to catch up, then retry. Inspect the lagging follower's master log for bootstrap/replication progress.
3. Wiped or WAL-GC'd follower. Restart the lagging master to trigger a remote bootstrap. If it is still stuck after catching up, follow this doc to Replace a failed YB-Master. 
4. Peers in transition. Wait for the in-flight config change to finish, or remove/re-add a peer stuck in PRE_VOTER per the change master config procedure.

Verification

yb-admin --master_addresses <masters list_all_masters

Expected output (healthy state): exactly one LEADER, the others FOLLOWER, all ALIVE. The tablet servers are visible on the new leader's Master UI, and the stepdown/maintenance operation now proceeds.

Note (Long-term Fix): 

This behavior is tracked under GHI-28675 . YugabyteDB is adding a lag column to the list_all_masters output to provide visibility into replication delay, allowing administrators to identify unhealthy followers before attempting maintenance operations.