Environment

• YugabyteDB Anywhere

Issue

For YB Anywhere nodes with a smaller amount of memory or that are managing Universes with a large amount of tablets (see note below), the Prometheus service may consume a significant portion of the memory. This will increase as more nodes are added to Universes (more nodes means more metrics to be scraped by Prometheus).

The memory consumption of Prometheus can increase to the point where there is no memory left on the YBA node. In this state, metrics may not appear in YBA, and the Prometheus service may restart intermittently. 

Note: If a Universe has a large number of tablets, Prometheus can consume a significant amount of memory when scraping tablet-level metrics. The same resolution applies.

Resolution

Overview

The ideal solution is to add more memory to the YBA node to accommodate the Prometheus process.

However, if immediate resolution is necessary, and memory cannot be added easily, you can set YBA runtime flag yb.metrics.collection_level=MINIMAL to reduce the amount of metrics collected by YBA.

In YBA >= v2.20, setting this runtime flag is accessible via the YBA UI.

Changing this flag means that Prometheus will scrape a subset of its normal metrics. In this state, some graphs in the YBA metrics will remain blank, but critical metrics will be visible.

The steps below demonstrate how to set yb.metrics.collection_level=MINIMAL 

Note: This is only a temporary workaround. The proper solution is to add more memory to the YBA node.

Steps

1. Navigate to Admin (Gear Icon) > Advanced > Global Configuration

Reference URL: <YBA-hostname>/admin/advanced/global-config

2. In the Search Bar, Search For yb.metrics.collection_level

3. In the Actions Drop-down, Click Edit Configuration

4. Change the Value to MINIMAL and Save the Configuration

Now Prometheus should start scraping a subset of metrics, saving some memory on the YBA node.