How to solve Prometheus Storage Filling Up Due to WAL Growth

Environment

YugabyteDB Anywhere: All versions

Issue

Prometheus storage under /opt/yugabyte/data/prometheus/storage/ fills up quickly, leading to potential disk space exhaustion.

Cause

The Write-Ahead Log (WAL) files under the wal/ directory may get stuck and are not purged or replayed correctly by Prometheus. This causes continuous growth of the storage used by Prometheus.

Solution

Step 1: Check Disk Usage

Run the following command to identify which directory is consuming the most space:

du -sh /opt/yugabyte/data/prometheus/storage/* 2>/dev/null | sort -hr | head -10

Example output:

134G   /opt/yugabyte/data/prometheus/storage/wal

If the wal directory consumes excessive space, WAL files are likely stuck.

Step 2: Restart Prometheus

Restart Prometheus to force WAL replay and cleanup.

For systemd-managed Prometheus:

sudo systemctl restart prometheus

Step 3: Verify Disk Usage

After restarting, check the directory again to confirm that disk usage has reduced:

du -sh /opt/yugabyte/data/prometheus/storage/* 2>/dev/null | sort -hr | head -10

Verification

• Confirm that the wal/ directory size has decreased.

• Verify Prometheus service is running properly:

  sudo systemctl status prometheus

If Prometheus fails to start or WAL keeps growing:

a) Stop Prometheus:

sudo systemctl stop prometheus

b) Move the WAL directory:

mv /path/to/prometheus/data/wal /path/to/prometheus/data/wal_old

c) Start Prometheus:

sudo systemctl start prometheus

d) After confirming Prometheus is healthy, delete the old WAL:

rm -rf /path/to/prometheus/data/wal_old