YugabyteDB Anywhere Task Fails During ‘Reload daemon’ ansible task – Yugabyte

Environment

Yugabyte Anywhere: All versions
YugabyteDB Version: - All versions

Issue

When performing a task in YugabyteDB Anywhere (YBA), such as a g-flag update, the task fails during an Ansible playbook execution. The YBA UI shows a generic error message indicating a failure in the 'Reload daemon' task, but the root cause is not immediately apparent.

Resolution

Overview

The root cause of this issue is the XDG_RUNTIME_DIR environment variable not being set for the yugabyte user on the database nodes. This variable is required by systemd to manage user sessions and services. When this variable is not set, the systemctl --user daemon-reload command, which is executed by the Ansible playbook, fails. This, in turn, causes the entire YBA task (e.g., g-flag update) to fail. The fix involves setting the XDG_RUNTIME_DIR environment variable in the .bashrc file of the yugabyte user on each database node.

Steps

1. Verify the Issue

When a YBA task fails, navigate to the task details page. You will see an error message similar to the following, indicating a failure in the Reload daemon task:

Failed to execute task ... hit error:
Error occurred. Code: 1. Output: ybops.common.exceptions.YBOpsRuntimeError: Runtime error: Playbook run of yb-server-ctl.yml against <node_hostname>, failed with return code 2
TASK [Reload daemon] ***********************************************************
fatal: [<node_hostname>]: FAILED! => changed=false
  module_stderr: ''
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error
  rc: 1.

This error message confirms that the Ansible playbook failed, but it doesn't provide the root cause. To get more details, you will need to enable verbose logging.

2. Enable Verbose Logging

To diagnose the issue, you need to enable more detailed logging in both YBA and the node agent.

a. Enable Ansible Debug Logging in YBA:

In the YBA UI, go to Admin -> Advanced -> Global configuration.
Set yb.ansible.debug to true.
Set yb.ansible.verbosity to 4.

b. Enable Debug Logging in the Node Agent:

SSH into each of the database nodes as the yugabyte user.
Edit the node agent configuration file located at /home/yugabyte/node-agent/config/config.yml (might vary based on node agent installation path)
Add the following line to the file:

log_level: "0"

Restart the node agent for the changes to take effect.

3. Retry the Task and Check the Logs

After enabling verbose logging, retry the failed task in the YBA UI. Once the task fails again, check the node-agent.log file on the affected database node(s). The log file is located at /home/yugabyte/node-agent/logs/node-agent.log (might vary based on node agent installation path).

You should see an error message similar to this:

Failed to connect to bus: No medium found

This error indicates that the systemd user session is not properly configured.

4. Implement the Fix

To resolve the issue, you need to set the XDG_RUNTIME_DIR environment variable for the yugabyte user on each database node.

a. Get the UID of the yugabyte user:

id -u yugabyte

This command will return the User ID (UID) of the yugabyte user. For example, 6607.

b. Edit the .bashrc file:

SSH into each database node as the yugabyte user.
Open the .bashrc file in a text editor:

vi ~/.bashrc

Add the following line to the end of the file, replacing <UID> with the UID you obtained in the previous step:

export XDG_RUNTIME_DIR=/run/user/<UID>

For example:

export XDG_RUNTIME_DIR=/run/user/6607

Save and close the file.

NOTE: You will need to either source the .bashrc file or log out and log back in for the changes to take effect. A restart of the node-agent may also be required.

5. Retry the YBA Task

After applying the fix to all database nodes, retry the failed task in the YBA UI. The task should now complete successfully.

6. Revert Logging Changes

Once you have confirmed that the issue is resolved, it is recommended to revert the logging changes you made in step 2 to avoid generating excessive logs.

a. Disable Ansible Debug Logging in YBA:

In the YBA UI, go to Admin -> Advanced -> Global configuration.
Set yb.ansible.debug to false.
Set yb.ansible.verbosity to its default value (usually 0 or 1).

b. Disable Debug Logging in the Node Agent:

Remove the log_level: "0" line from the /home/yugabyte/node-agent/config/config.yml file on each database node.
Restart the node agent.