Issue

Yugabyte errors with the following error while changing the configuration of a tablet using the yb-admin utility.     

 ./yb-admin -master_addresses <master_addresses> change_config 86563975de424b3db74c3861c0da5d43 REMOVE_SERVER da266157b52a4a3e8d32d0882bd324a4
Error: Configuration error (yb/tools/yb-admin_client.cc:689): Unable to change config: Old tablet server leader same as new even after re-election!

Environment

• Yugabyte/YugabyteDB 2.0 or greater
  

Resolution

Disable the load balancer

./yb-admin -master_addresses <master-addresses> set_load_balancer_enabled 0

Remove the tablet leader

 ./yb-admin -master_addresses <master_addresses> change_config 86563975de424b3db74c3861c0da5d43 REMOVE_SERVER da266157b52a4a3e8d32d0882bd324a4

Enable the load balancer

./yb-admin -master_addresses <master-addresses> set_load_balancer_enabled 1

Root Cause

When removing the tablet peer leader from the tablet server(can cause unavailability until a new leader is elected for the particular tablet), the admin utility reaches out to the master service to get the tablet locations. The Master service sends the response back showing the role of the tablet as Leader of the group. The raft consensus algorithm will trigger the leader step-down process for the tablet to re-elect another leader.

It eventually changes the current tablet leader role to the follower(in a 3 node cluster with RF= 3) in order to maintain the replicas but reports the error. This is due to a incorrect check which will be fixed in future releases of Yugabyte.

./src/yb/tools/yb-admin_client.cc

674 // If removing the leader ts, then first make it step down and that
675 // starts an election and gets a new leader ts.
676 if (cc_type == consensus::REMOVE_SERVER &&
677         leader_uuid == peer_uuid) {
678     string old_leader_uuid = leader_uuid;
679     RETURN_NOT_OK(LeaderStepDown(
680         leader_uuid, tablet_id, /* new_leader_uuid */ std::string(), &consensus_proxy));
681     sleep(5); // TODO - election completion timing is not known accurately
682     RETURN_NOT_OK(SetTabletPeerInfo(tablet_id, LEADER, &leader_uuid, &leader_addr));
683     if (leader_uuid != old_leader_uuid) {  # < errant check here
684         return STATUS(ConfigurationError,
685                 "Old tablet server leader same as new even after re-election!"); 
686     } 
687     consensus_proxy.reset(new ConsensusServiceProxy(proxy_cache_.get(), leader_addr)); 
688 }

Note: We don't suggest manually changing the configuration of a tablet since this is managed by the YB-Master load balancer. Please contact Yugabyte support before making such change.