Environment

• YugabyteDB Anywhere - Upgrade from 2.16.0 to 2.16.2

Issue

• When the YugabyteDB Anywhere UI is upgraded from 2.16.0 to the next version, the upgrade fails, crashing yb-controller and generates a core.

Resolution

Overview

This is a bug in yb-controller where /tmp is getting cleaned up during an upgrade and this crashes the yb-controller process, generating the cores.

In the yb-controller fatal logs, we see

Log file created at: 2023/03/14 04:59:58
Running on machine: yb-prod-yugabyte-2-n1
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0314 04:59:58.748980  7534 upgrade_task.cc:97] Check failed: ret == 0 (-1 vs. 0) 
Log file created at: 2023/03/14 04:59:59
Running on machine: yb-prod-prod-yugabyte-2-n3
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0314 04:59:59.216954 31090 upgrade_task.cc:97] Check failed: ret == 0 (-1 vs. 0) 

We observe this issue due to missing custom directory /tmp/yb-controller-tmp

To resolve this issue, please retry the upgrade. Retrying the upgrade creates this missing directory and the upgrade goes through successfully.

Steps

1. Please retry the upgrade. Retrying the upgrade creates this missing directory and the upgrade goes through successfully.