Collecting Core Dump of PostgreSQL Backend Process Using `gcore`

Issue

In some scenarios, a query may appear to be hung, or the PostgreSQL backend process may exhibit unexpected behavior. To diagnose such issues, capturing a core dump can provide valuable insights into the state of the process at the time of the problem.

This KB outlines the steps to collect a core dump of a PostgreSQL backend process using the gcore utility.

Environment

• YugabyteDB (YSQL)

Cause

Core dumps are useful in scenarios where:

• A query appears to be hung indefinitely.
• The backend process consumes high CPU or memory.
• The process is unresponsive, but it has not crashed.

By capturing a core dump, we can analyze the process's internal state, check for deadlocks, and investigate low-level issues.

Resolution

Step 1: Identify the PostgreSQL Backend Process

Each query in YugabyteDB (YSQL) runs within a separate PostgreSQL backend process. To find the process ID (PID) of the backend handling a specific query, follow these steps:

1. Run the following query on each node until you find the hung query and its corresponding PID:

   SELECT pid, datname, usename, application_name, query, state 
   FROM pg_stat_activity 
   WHERE state = 'active';
   

2. Identify the node where the hung query is running and note the PID.

Step 2: Verify gcore Availability

Ensure that the gcore utility is installed on the node where the query is running. It is included in the gdb package, which can be installed using:

# On RHEL/CentOS
sudo yum install gdb -y  

# On Ubuntu/Debian
sudo apt install gdb -y  

Step 3: Capture the Core Dump

Run the following command on the same node where the hung query was found to generate a core dump of the identified PostgreSQL backend process:

• Please ensure, you have sufficient disk space to store the core dump file. If not, specify a different location with enough space.

sudo gcore -o /tmp/core_dump <PID>

This will create a core dump file named /tmp/core_dump.<PID>.

Step 4: Verify the Core Dump File

Confirm that the file was created successfully:

ls -lh /tmp/core_dump.*

If the file is missing or empty, ensure that core dumps are enabled on the system (ulimit settings may restrict this).

Step 5: Analyze the Core Dump (Optional)

To get a quick stack trace from the core dump, use:

gdb -q /path/to/postgres /tmp/core_dump.<PID> -ex "thread apply all bt" -ex "quit"

This will print the backtrace of all threads in the process, which can help debug the issue.