Environment

• YugabyteDB - 2.6.x
• YugabyteDB - 2.8.3 and older

Issue

When creating database backups through YugabyteDB Anywhere (YBA), database nodes experience high resource utilization at the beginning and end of the backup process.

Resolution

Overview

When creating database backups through YBA, the master leader sends an RPC request to the leader of each tablet instructing the tserver to create a snapshot of the tablet. These tablet snapshots will later be copied to backup storage. At the end of the backup, the master leader sends RPC requests instructing the tserver to delete the snapshot for each tablet.

In older YugabyteDB releases, the master did not throttle snapshot creation or deletion requests sent to the tservers, meaning that systems with large numbers of tablets could see a significant increase in CPU or I/O load during snapshot creation or deletion.

Beginning in YugabyteDB version 2.8.4, concurrent snapshot RPCs are throttled to one per tserver by default. For other versions, refer to the steps below.

Steps

1a. For 2.6 releases starting with 2.6.12 and for 2.8 releases from 2.8.1 through 2.8.3 (inclusive), set the master GFlags max_concurrent_snapshot_rpcs and max_concurrent_snapshot_rpcs_per_tserver as shown below:

For more information on how to set database GFlags, see the Edit configuration flags section of the documentation.

1b. For all other versions, it is recommended to upgrade to the most recent stable release.