Generate a backtrace from a core file with GDB

Environment:

Version: all

Product: Yugabyte DB 

Infrastructure: Any

Description:

When a database process fails and generates a core file, the platform will generate an alert. The alert contains the path to the core file, and the node on which it was generated.

Resolution:

From the Universes -> Nodes -> Actions menu, click Connect

Click Copy 

Connect via SSH to your yugaware node and paste in the ssh connection string. Modify it to log in as the user centos

# sudo ssh -i /opt/yugaware_data/keys/f9e01287-c334-4890-b181-e5b804e581fc/yb-demo-gcp-key.pem -ostricthostkeychecking=no -p 22 centos@10.142.15.230
Last login: Thu Jun 17 05:04:48 2021 from 10.0.10.9
# 

Install gdb

# sudo yum install gdb -y

Log out of the server, log back in as user yugabyte

# sudo ssh -i /opt/yugaware_data/keys/f9e01287-c334-4890-b181-e5b804e581fc/yb-demo-gcp-key.pem -ostricthostkeychecking=no -p 22 yugabyte@10.142.15.230
Last login: Thu Jun 17 05:05:00 2021 from 10.0.10.9
# 

The core files will be located in /home/yugabyte/cores. They will have been generated by one of the following binaries:

• /home/yugabyte/tserver/bin/yb-tserver
• /home/yugabyte/master/bin/yb-master
• /home/yugabyte/tserver/postgres/bin/postgres

List the cores directory to see a list of cores

# ls /home/yugabyte/cores/
core_postgres.23005 core_yb-master.22618 core_yb-tserver.22879

Run gdb to generate the backtrace which you can provide to the support team.

postgres

# gdb /home/yugabyte/tserver/postgres/bin/postgres -c /home/yugabyte/cores/<postgres corefile> --batch -ex 'thread apply all bt full' > backtrace_full_postgres.out

yb-tserver

# gdb /home/yugabyte/tserver/bin/yb-tserver -c /home/yugabyte/cores/<tserver corefile> --batch -ex 'thread apply all bt full' > backtrace_full_tserver.out

yb-master

# gdb /home/yugabyte/master/bin/yb-master -c /home/yugabyte/cores/<master corefile> --batch -ex 'thread apply all bt full' > backtrace_full_tserver.out