Environment

YugabyteDB 2024.1.3 and above

YSQL and YCQL

Issue

Insert, Update, or Delete to tables fail with the below error:

IO error (yb/tserver/tablet_service.cc:2288): Write to tablet 749e4d78244c43d2bba9cdb8505b732f rejected. Node 9a24d9c8493b44caa97ba4ae06e5829d has insufficient disk space

NOTE: Read only queries are not affected by this error.

TIP: Consider configuring low disk space alerts that will notify you before nodes run out of space so that mitigation actions can be taken ahead of time. This will help to prevent any disruption to your applications.

Resolution

Overview

When the disk hosting the tablet WALs are low on space the database will reject all data modification operations. Even delete of rows are rejected since YugabyteDB MVCC versioning requires a delete to write more data and then lazily garbage collect the dropped rows.

The flags --max_disk_throughput_mbps and --reject_writes_min_disk_space_mb determine the amount of disk space that is considered too low. The default value is 3GB.

The yb-tserver logs will contain the following message when the system nears the threshold (18GB):

W0829 15:35:51.325239 1986375680 log.cc:2186] Low disk space on yb-data/tserver/wals/table-7457ebcd745e4ea89375a30406f2c188/tablet-749e4d78244c43d2bba9cdb8505b732f/wal-000000001. Free space: 3518698209 bytes

and the following message when there is no more space left:

E0829 15:35:51.325239 1986375680 log.cc:2181] Not enough disk space available on yb-data/tserver/wals/table-7457ebcd745e4ea89375a30406f2c188/tablet-749e4d78244c43d2bba9cdb8505b732f/wal-000000001. Free space: 3518698209 bytes

Option 1: Increase capacity

YugabyteDB allows you to expand the cluster in an online manner. There are two ways to scale the cluster.

Scale up

You can scale up by switching to bigger machines, or by adding more storage disks.
If you are using YugabyteDB Anywhere, follow the instructions in Expanding universe node disk capacity with YugabyteDB Anywhere (platform)

Scale out

You can scale out by adding more nodes to the cluster. This will cause the YugabyteDB load balancer to rebalance the tablets and the data across more nodes.

Option 2: Remove unnecessary files from the nodes

It is sometime possible for unnecessary files to accumulate on the nodes. This can include large log files, or older core dumps.

WARNING: Do not modify or remove the YugabyteDB data, bin or config directories and files. Damage or loss of these files can result in unavailability and data loss.

Option 3: Drop unnecessary tables and databases/namespaces

Dropping large tables, YSQL databases, or YCQL namespaces that are no longer necessary will help recover disk space.

When dropping tables on a database/namespace with point in time recovery (PITR) is enabled, the table will be set in a HIDDEN state. It will not be cleaned up until all its snapshot schedules have expired.
Dropping YSQL tables require the yb-master to have sufficient disk space, since it involves updating the YSQL catalog tables.

Option 4: Disable database features that require more disk space

The following features  result in more disk space usage. You can disable or turn these features off to reclaim space on the nodes.

• Time to live (TTL)
• Point in time recovery (PITR)
• xClsuter
• CDC

Option 5: Run manual compaction

YugabyteDB performs automatic compaction of the data to keep the database running efficiently, and reduce disk space. Certain operations like bulk load of data, or deleting a large number of rows can cause a temporary spike in disk usage. If waiting for the automatic compaction task is not an option, then you can manually run the compaction task:

• Compact individual tables
• Compact individual nodes