Environment

• YugabyteDB Anywhere - 2.20
• This has been fixed by PLAT-8713 which is fixed in 2.19.10 and later.

Issue

Although there are runtime flags to set "Remote temp directory", "temp directory path", "Ansible local temp directory"  etc .. these may take effect, but the temp files are not cleaned-up automatically for non-default temp directories (i.e. if they are not at /tmp) .

 Customers may need to modify the default temp directory location for space or security reasons. In some cases, the O.S. may be configured to disallow programs to execute if located in /tmp. Customers may choose to have YB temp files created at a different location.

Symptoms

• Disk filling up with "shell_process_*" files in the non-default temp directory
• The "Starting rotating logs" messages always shows Zero files :

  $ grep -A1 rotating application.log
  
     ...- Starting rotating logs, max dir size 10240 retention hours 1
     ...- Found 0 log files 0 bytes total size
     ...- Starting rotating logs, max dir size 10240 retention hours 1
     ...- Found 0 log files 0 bytes total size

Resolution

Overview

The YBA Admin -> Advanced [Global configuration] Tab (runtime flags) can be used to specify alternative temp file locations for "Remote temp directory", "temp directory path", "Ansible local temp directory"  etc.

These take effect immediately.

However, shell_process_*  files accumulate there and may result in disk full.

In YBA Java code, the cleanup routines are a singleton, and initialized only after modifying yb-platform.conf , then restarting YBA.

YBA attempts to clean up temp files every hour by default. It logs this as "Starting rotating logs" in the application.log. When working correctly, this should be removing accumulated files from the temp directory.

Steps

1. Edit /opt/yugabyte/software/active/yb-platform/conf/yb-platform.conf
    and add the line

logs.shell.output_dir = "/var/tmp" #  -->  Your custom temp dir

This should go inside the 

   yb{
     ...
   }

section

2. As root:

yba-ctl restart

3. Verify the application log .. check for temp file removal starting:

After the "Starting rotating logs" , you should see a message like:

 from ShellLogsManager in application-pekko.actor.default-dispatcher-12 - Found 1310 log files 14 MB total size