Environment

• YugabyteDB Anywhere - All Versions
  

Issue

In the case where YCQL latency cannot be explained through the YugabyteDB Anywhere metrics, a promdump, or log analysis, YCQL tracing may be necessary to generate more granular query logs to identify potential areas of latency.

Resolution

Overview

Run the script below to enable YCQL tracing for a brief period of time.

This will add more detailed information to the logs so that the YB Engineering team can know exactly where specific queries spend the most time in the stack

The YB Support Team will provide feedback on where the particular query performance suffers.

Steps

1. Add this script to the Master leader node, and fill in the appropriate values for the environment variables

set -eux

# Please fill out the hosts that need to enable tracing. Space separated.

ALL="<node_hostname_1> <node_hostname_2> <node_hostname_3>" # All nodes you want to trace

HOSTS=$ALL
BIN_PATH="/home/yugabyte/tserver/bin"

# if ssl is enabled
CERTS_DIR="/home/yugabyte/yugabyte-tls-config/"
OPTIONS="-certs_dir_name $CERTS_DIR"

# if ssl not enabled
# OPTIONS=""

date
for IP in $HOSTS; do

    HOST=${IP}:9100

    ${BIN_PATH}/yb-ts-cli --server_address=${HOST} $OPTIONS set_flag -force enable_tracing 1
    ${BIN_PATH}/yb-ts-cli --server_address=${HOST} $OPTIONS set_flag -force collect_end_to_end_traces 1
    ${BIN_PATH}/yb-ts-cli --server_address=${HOST} $OPTIONS set_flag -force rpc_slow_query_threshold_ms 3000

    # Only For versions 2.6 and above
    ${BIN_PATH}/yb-ts-cli --server_address=${HOST} $OPTIONS set_flag -force tracing_level 1

done




sleep 30s


for IP in $HOSTS; do

    HOST=${IP}:9100    

    ${BIN_PATH}/yb-ts-cli --server_address=${HOST} $OPTIONS set_flag -force enable_tracing 0
    ${BIN_PATH}/yb-ts-cli --server_address=${HOST} $OPTIONS set_flag -force collect_end_to_end_traces 0
    ${BIN_PATH}/yb-ts-cli --server_address=${HOST} $OPTIONS set_flag -force rpc_slow_query_threshold_ms 10000

    # Only For versions 2.6 and above
    ${BIN_PATH}/yb-ts-cli --server_address=${HOST} $OPTIONS set_flag -force tracing_level 0

done
date

exit

What is YCQL Tracing?

$ALL

$CERTS_DIR

Use this variable if SSL is enabled. By default the path to the certs is /home/yugabyte/yugabyte-tls-config/

If SSL is not enabled in your cluster, comment out the CERTS_DIR and OPTIONS lines and uncomment the blank option line under # if ssl not enabled

Flags used in the above script - all are tserver gflags

Flag: enable_tracing

Tracing provides a breakdown in the logs of what section of code the query is calling, and how long it has been there.

Flag: collect_end_to_end_traces

This will collect traces between nodes in the cluster

Flag: rpc_slow_query_threshold_ms

Queries running longer than this threshold will appear in the logs

Other flags to consider depending on the type of Latency and workload under consideration

Flag: print_trace_every

This controls how often to print queries.  If you set -force print_trace_every 1000, it will print 1 out of every 1000 queries.  This is helpful if you have general latency and want to see what queries are running.