Environment

• On-Prem Workflow
• Yugabyte Platform - 2.x or higher

Issue

When attempting to recreate a Universe, the following error messages are being reported in the `application.log` files on the Platform host:

Already present (yb/tablet/tablet_metadata.cc:264): Unable to initialize catalog manager: Failed to initialize sys tables async: Encountered errors during system catalog initialization:
Error on Load: Not found (yb/util/env_posix.cc:1514): Unable to load consensus metadata for tablet 00000000000000000000000000000000: /yb-data/yb-data/master/consensus-meta/00000000000000000000000000000000: No such file or directory (system error 2)
Error on CreateNew: : Raft group already exists: 00000000000000000000000000000000
@ 0x7f313daccd3c yb::LogFatalHandlerSink::send()
@ 0x7f313ccb0cae google::LogMessage::SendToLog()
@ 0x7f313ccade3a google::LogMessage::Flush()
@ 0x7f313ccb1529 google::LogMessageFatal::~LogMessageFatal()
@ 0x4094b6 yb::master::MasterMain()
@ 0x7f3138902825 __libc_start_main
@ 0x408869 _start
@ (nil) (unknown)

Resolution

Overview

This issue will occur when nodes, typically those in environments provisioned via the on-prem workflow, are not properly cleaned up. Nodes can arrive at this state if any Yugabyte processes were previously running. This can occur when reusing a node after provision failure, or if it was previously a member of a Universe.

Steps

As the Yugabyte user:

1. Delete the Universe from the Yugaware Platform. This may require the need to select "Ignore Errors and Force Delete"

2. For all nodes in the Universe, examine all files and folders under inside the data directory specified for the onprem provider (e.g. `$DATA_DIRECTORY/yb-data”).

• For the nodes running Master services

$ ll yb-data/
total 4
drwxr-xr-x. 2 yugabyte yugabyte 46 Apr 28 19:10 consensus-meta
drwxr-xr-x. 3 yugabyte yugabyte 21 Apr 28 19:10 data
-rw-------. 1 yugabyte yugabyte 635 Apr 28 19:10 instance
drwxr-xr-x. 2 yugabyte yugabyte 241 Apr 28 19:10 logs
drwxr-xr-x. 2 yugabyte yugabyte 46 Apr 28 19:11 tablet-meta
drwxr-xr-x. 3 yugabyte yugabyte 36 Apr 28 19:10 wals

$ ll pg_data/
total 76
drwx------. 3 yugabyte yugabyte 15 Apr 28 19:10 base
-rw-------. 1 yugabyte yugabyte 69 Apr 28 19:10 current_logfiles
drwx------. 2 yugabyte yugabyte 47 Apr 28 19:10 global
drwx------. 2 yugabyte yugabyte 6 Apr 28 19:10 pg_commit_ts
drwx------. 2 yugabyte yugabyte 6 Apr 28 19:10 pg_dynshmem
-rw-------. 1 yugabyte yugabyte 1636 Apr 28 19:10 pg_ident.conf
drwx------. 4 yugabyte yugabyte 68 Apr 28 19:15 pg_logical
drwx------. 4 yugabyte yugabyte 36 Apr 28 19:10 pg_multixact
drwx------. 2 yugabyte yugabyte 18 Apr 28 19:10 pg_notify
drwx------. 2 yugabyte yugabyte 6 Apr 28 19:10 pg_replslot
drwx------. 2 yugabyte yugabyte 6 Apr 28 19:10 pg_serial
drwx------. 2 yugabyte yugabyte 6 Apr 28 19:10 pg_snapshots
drwx------. 2 yugabyte yugabyte 6 Apr 28 19:10 pg_stat
drwx------. 2 yugabyte yugabyte 35 Apr 28 19:10 pg_stat_tmp
drwx------. 2 yugabyte yugabyte 18 Apr 28 19:10 pg_subtrans
drwx------. 2 yugabyte yugabyte 6 Apr 28 19:10 pg_tblspc
drwx------. 2 yugabyte yugabyte 6 Apr 28 19:10 pg_twophase
-rw-------. 1 yugabyte yugabyte 3 Apr 28 19:10 PG_VERSION
drwx------. 3 yugabyte yugabyte 60 Apr 28 19:10 pg_wal
drwx------. 2 yugabyte yugabyte 18 Apr 28 19:10 pg_xact
-rw-------. 1 yugabyte yugabyte 88 Apr 28 19:10 postgresql.auto.conf
-rw-------. 1 yugabyte yugabyte 23852 Apr 28 19:10 postgresql.conf
-rw-------. 1 yugabyte yugabyte 541 Apr 28 19:10 postmaster.opts
-rw-------. 1 yugabyte yugabyte 107 Apr 28 19:10 postmaster.pid
-rw-rw-r--. 1 yugabyte yugabyte 171 Apr 28 19:10 ysql_hba.conf
-rw-rw-r--. 1 yugabyte yugabyte 23907 Apr 28 19:10 ysql_pg.conf

• For the nodes running T-Server services: 

$ ll yb-data/
total 8
drwxr-xr-x. 2 yugabyte yugabyte 166 Apr 28 19:11 consensus-meta
drwxr-xr-x. 3 yugabyte yugabyte 21 Apr 28 19:10 data
-rw-------. 1 yugabyte yugabyte 635 Apr 28 19:10 instance
drwxr-xr-x. 2 yugabyte yugabyte 4096 Apr 28 19:10 logs
drwxr-xr-x. 2 yugabyte yugabyte 166 Apr 28 19:11 tablet-meta
drwxr-xr-x. 3 yugabyte yugabyte 52 Apr 28 19:11 wals

3. Delete all files under the yb-data directory.

• For the nodes running Master services
  

$ rm -rf pg_data/ yb-data/

• For the nodes running T-Server services: 

$ rm -rf yb_data/

4. You can now recreate the Universe.

Next Steps

If the above actions do not result in a successful Universe creation, please open a ticket with Yugabyte Support