Environment

Yugaware Platform : All versions

Issue

Yugaware platform node installed via replicated is low on disk space.

Resolution

Solution1:

Change the metrics retention time configuration via replicated console to from the default 15 days or decrease the scrape interval.

Reducing metrics retention time will delete old metrics data under /opt/yugabyte/prometheusv2 directory.

Reducing the scrape interval will cause the same data time to be retained, but at a lower granularity. 

Note: The application has to be restarted to apply these settings during which platform won't be accessible due to restart of nginx, yugaware, postgres, prometheus containers.

 http://<Yugaware IP>:8800/settings#metrics 

Solution 2:

Check the  /opt/yugabyte/prometheusv2 directory to see if it contains any snapshots left over from previous Platform backups. If it does, please remove/move them to a different directory or upgrade yugaware application to 2.4.2 or higher. Please refer to Github issue: https://github.com/yugabyte/yugabyte-db/issues/7204 for more details.

Solution 3:

1. Check space utilized by /opt/yugabyte/yugaware
2. Go to /opt/yugabyte/yugaware and remove the prometheus_old directory.
3. If yugaware platform is newer than v 2.0.5 - removing old 'prometheus' directory (not 'prometheusv2') is safe. See the final directory structure below.

# du -sh * /opt/yugabyte
4.0K postgresql
4.0K prometheus_configs
1.4G prometheusv2
481M releases
188K yugaware

# ls -l /opt/yugabyte/
total 20
drwxr-xr-x 3 999 root 4096 May 16 2018 postgresql
drwxr-xr-x 262 root root 4096 May 8 2020 prometheus  # << may be safely removed
drwxrwxrwx 27 root root 4096 Jun 16 01:00 prometheusv2
drwxr-xr-x 12 root root 4096 Nov 11 2020 releases
drwxr-xr-x 4 root root 4096 May 16 2018 yugaware

Setting up Log rotation:

The “local” logging driver is recommended as it performs log-rotation by default and uses a more efficient file format.

$ docker info --format '{{.LoggingDriver}}'
json-file

Add the below configurations to the configuration file in /etc/docker/daemon.json on Linux hosts. Configure the daemon so that containers remain running if the daemon becomes unavailable by enabling live restore. Restart Docker for changes to take effect.

{
  "live-restore":true,
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m"
  }
}

NOTE: There will be a brief period of container downtime when the Docker daemon is restarted.

If you use systemd, then use the command to avoid any downtime for containers.

systemct reload docker

$ docker info --format '{{.LoggingDriver}}'
local