Environment
- YugabyteDB Anywhere
Issue
For YB Anywhere nodes with a smaller amount of memory or that are managing Universes with a large amount of tablets (see note below), the Prometheus service may consume a significant portion of the memory. This will increase as more nodes are added to Universes (more nodes means more metrics to be scraped by Prometheus).
The memory consumption of Prometheus can increase to the point where there is no memory left on the YBA node. In this state, metrics may not appear in YBA, and the Prometheus service may restart intermittently.
Note: If a Universe has a large number of tables or tablets, Prometheus can consume a significant amount of memory when scraping table-level metrics. On YBA 2024.2.6.0, 2025.1.2.0, 2.27.0.0 and later, the preferred fix for this case is TABLE_OFF rather than MINIMAL. See Choosing a collection level below.
Resolution
Overview
Overview
The ideal solution is to add more memory to the YBA node to accommodate the Prometheus process.
However, if immediate resolution is necessary, and memory cannot be added easily, you can reduce the amount of metrics collected by YBA using the runtime flag yb.metrics.collection_level.
In YBA >= v2.20, setting this runtime flag is accessible via the YBA UI.
There are two reduced collection levels available. TABLE_OFF drops per-table and per-tablet metrics only. MINIMAL restricts collection to a small allowlist and cuts server-level metrics as well. In most cases TABLE_OFF is the better choice, and the section below explains why.
Note: Both levels are workarounds. The proper solution is to add more memory to the YBA node, or to size Prometheus for the deployment. See How to Estimate Prometheus Resource Requirements for YugabyteDB Anywhere.
Choosing a collection level
TABLE_OFF (preferred)
TABLE_OFF drops per-table and per-tablet metrics and leaves everything else untouched. Server-level metrics are retained in full, including latency, operations per second, connection counts, and all host resource metrics collected by the node exporter. In the YBA UI, Outlier Tables and the per-table breakdowns stop reporting, while the standard graphs continue to work.
This is usually the right first move. The number of metrics scraped per Universe is approximately (60 x number_of_tables + 9000) x number_of_nodes, so once a Universe has more than a few hundred tables, the table term dominates everything else. Removing it is the largest reduction available without losing visibility into the database itself.
You can estimate the effect for your own deployment using the Prometheus sizing calculator.
TABLE_OFF requires YBA 2024.2.6.0, 2025.1.2.0, 2.27.0.0 or later.
MINIMAL
MINIMAL restricts Prometheus to an allowlist of 42 metric families, cutting both server-level and table-level collection. A significant number of graphs in the YBA metrics will be blank in this state, though critical metrics remain visible.
Use MINIMAL only where TABLE_OFF is unavailable on your version, or where dropping table-level metrics alone does not reduce consumption enough.
Scope
yb.metrics.collection_level is Universe-scoped, so it does not have to be applied across the whole deployment. Where only one or two Universes are driving the problem, set the level on those Universes and leave the rest at NORMAL.
Steps
1. Navigate to Admin (Gear Icon) > Advanced > Global Configuration
Reference URL: <YBA-hostname>/admin/advanced/global-config
2. In the Search Bar, Search For yb.metrics.collection_level
3. In the Actions Drop-down, Click Edit Configuration
4. Change the Value to TABLE_OFF or MINIMAL and Save the Configuration
Prometheus will now scrape a reduced set of metrics. The change applies on its own: YBA marks the affected Universes' scrape configuration as stale and rewrites the Prometheus target files in the background, usually within a minute. No restart of YBA or Prometheus is required.
Important: this does not reclaim memory already in use
Changing the collection level reduces what Prometheus scrapes from this point forward. It does not remove series that are already resident in the Prometheus head block, and those remain in memory until they age out of the head.
This matters when Prometheus is already in trouble. If the service is restarting repeatedly, or failing to complete head compaction, lowering the collection level will not resolve it, because the memory is held by series that have already been ingested. A Prometheus that cannot complete a head compaction will also never truncate its write-ahead log, so the WAL and chunks_head directories grow without bound and each restart becomes slower than the last.
Treat a collection level change as a measure to prevent recurrence. Address an in-progress memory problem separately, and contact Yugabyte Support if Prometheus is in a restart loop.
Comments
0 comments
Please sign in to leave a comment.