Environment

• All Yugabyte versions

Issue

Yugabyte tserver process can crash with messages similar to the following:

F0906 13:22:21.111539 2518302 log_util.cc:191] Check failed: _s.ok() Bad status: IO error (yb/util/env_posix.cc:1562): ... Too many open files (system error 24)
F0906 13:19:53.125422 2498599 log_util.cc:191] Check failed: _s.ok() Bad status: IO error (yb/util/env_posix.cc:1562): ... Too many open files (system error 24)
F0906 13:19:53.125572 2498594 log_util.cc:191] Check failed: _s.ok() Bad status: IO error (yb/util/env_posix.cc:1562): ... Too many open files (system error 24)

Where the key observation is the message: Too many open files (system error 24)

Resolution

Overview

In some cases the underlying Linux operating system setting for maximum number of open files is set at a value lower than the expected default, this can then prevent the Yugabyte processes, and Linux Operating System from opening more files to perform critical functions.

Steps

1.  To identify the max open files for the, run the following command as the yugabyte user on the tserver which exhibits the described symptoms.

cat /proc/$(pgrep yb-tserver)/limits

Verify if the value is lower than 1048576

2.  If the value is less than 1048576, modify the following files as the root user:

/etc/systemd/system.conf
/etc/systemd/user.conf

In each file add the following entry:

DefaultLimitNOFILE=1048576

3. Reboot the tserver to ensure that the value specified in the previous is properly applied, and persisted across a system restart. The most common method of reboot is to run the following as user root:

reboot

4.  Once the tserver is reachable by SSH, or terminal access, run the following command as the yugabyte user:

cat /proc/$(pgrep yb-tserver)/limits

and validate that the value is consistent with the earlier file modification step.