Environment
- YugabyteDB Anywhere (YBA): 2.20.x and higher. The configuration files and paths are specific to this YBA version but the principles apply to other modern versions.
-
YugabyteDB: All versions. The
node_exportercomponent is independent of the database version, and securing it follows a standard procedure.
Issue Description
By default, the Prometheus instance managed by YugabyteDB Anywhere (YBA) and the node_exporter services on the database nodes are configured without encryption or authentication. The Prometheus web UI (port 9090) and the node exporter metrics endpoints (port 9300) are served over plain HTTP and allow unauthenticated access. This can expose potentially sensitive metrics and configuration data, which is not ideal for production or security-conscious environments.
Overview
The default configuration of Prometheus and its node_exporter services prioritizes ease of setup over security. To harden these components, a comprehensive, multi-part approach is required to enable both TLS encryption (HTTPS) and basic authentication.
- Secure the main Prometheus service on the YBA host: This involves enabling HTTPS and basic authentication for the Prometheus web UI (port 9090).
-
Secure the
node_exporterservice on all YugabyteDB data nodes: This involves enabling HTTPS and basic authentication on each individual database node's metrics endpoint (port 9300). -
Reconfigure the YBA Prometheus service to scrape the newly secured data nodes: After securing the
node_exporterendpoints, the central Prometheus instance must be updated to use the new HTTPS scheme and credentials to successfully scrape metrics.
Failure to complete all parts will result in either an incomplete security setup or a loss of metrics in the YBA UI, where nodes will appear as "Unreachable".
Prerequisites
This article is Part 2 of the series on securing Prometheus. It assumes that you have already secured the main Prometheus service on the YBA host.
-
Part 1: How to enable basic user authentication and secure Prometheus with TLS
If you have not yet enabled TLS and authentication on your main YBA Prometheus instance, please complete Part 1 before continuing.
A Note on Password Security: Plain-Text vs. Hashed
You will notice that we configure the node_exporter (in Step 1) with a secure bcrypt password hash. This is excellent security practice for the server side.
However, for the Prometheus server to connect as a client, the configuration in prometheus.yml (in Step 2) requires the actual password to be stored in plain text. This plain-text password in the main Prometheus configuration file is the specific security risk this note addresses.
Is there a way to avoid this plain-text password?
Yes. For production environments, the recommended approach is to store the password in a separate, protected file and reference that file from the configuration. This prevents the plain-text password from being directly visible in prometheus.yml.
Recommended for Production: Using a password_file
Here is the recommended method for securely providing the password on the Prometheus server.
1. Create the Password File On the YBA host (where the main Prometheus server is running), create a new file that contains only the password.
# Create a directory if it doesn't exist sudo mkdir -p /etc/prometheus/ # Create the password file echo 'YourNodeExporterPassword' | sudo tee /etc/prometheus/node_exporter.pass
2. Secure the Password File It is critical to restrict permissions on this file so that only the user running the Prometheus service can read it.
Important: Verify the Service User The Prometheus service may not run as the
prometheususer. Before proceeding, check the correct user by runningsystemctl status prometheuson the YBA host and looking for theUser:field. Use that username in thechowncommand below.
# Set read/write permissions for the owner only sudo chmod 600 /etc/prometheus/node_exporter.pass # Set the owner to the correct service user (e.g., prometheus:prometheus or yugabyte:yugabyte) sudo chown prometheus:prometheus /etc/prometheus/node_exporter.pass
The configuration in Step 2 of this guide has been updated to use this more secure method.
This Guide's Process
This guide focuses on securing the node_exporter endpoints on your data nodes and reconfiguring the main Prometheus instance to scrape them securely. We will enable TLS (HTTPS) and basic authentication to protect your metrics data.
Important Note on Upgrades
The manual configurations applied in this guide to the
node_exporterservice file and theprometheus.ymlfile may be overwritten during a YugabyteDB universe or YugabyteDB Anywhere (YBA) upgrade. After performing any upgrade, you must verify that these security configurations are still in place and re-apply them if necessary.
Step 1: Secure Node Exporter on each YugabyteDB Data Node
This phase involves securing the metrics endpoint on every individual database node. These steps must be repeated for each data node in your universe.
1: Create Password Hash
SSH into a data node. node_exporter requires a bcrypt password hash. Use htpasswd to generate one. The command will output a string like node_exp_user:$2y$05$... - copy this entire string for the next step.
# On RHEL/CentOS, you may need to run: sudo yum install httpd-tools # On Debian/Ubuntu: sudo apt-get install apache2-utils htpasswd -cbB /tmp/node-exporter-creds.txt node_exp_user YourNodeExporterPassword
Note: You can also use another Python based method to generate a bcrypt password hash.
2: Create Web Configuration File
Copy Certificates and Set Permissions
To avoid "permission denied" errors, we must copy the node's certificate and key to a standardized location and ensure the node_exporter service can read them.
-
Create the configuration directory:
sudo mkdir -p /etc/prometheus-node-exporter
-
Copy the certificate and key (replace the source path with your actual certificate location, e.g.,
/home/yugabyte/yugabyte-tls-config/node.<IP>.crt):sudo cp /home/yugabyte/yugabyte-tls-config/node* /etc/prometheus-node-exporter/*
- Set correct ownership and permissions:
Important: Verify the Service User The
node_exporterservice may run as a different user (e.g.,yugabyte). Before proceeding, check the correct user by runningsystemctl status node_exporteron the data node and looking for theUser:field. Use that username in thechowncommand below.
The node_exporter service often runs as root, so we give root ownership and restrict permissions for security.
sudo chown prometheus:prometheus /etc/prometheus-node-exporter/* sudo chmod 400 /etc/prometheus-node-exporter/*
Create and Populate Web Configuration File
Create a directory and a new configuration file for node_exporter. Paste the content below, replacing the placeholder hash with the one you just generated and providing the correct paths to your node's TLS certificate and key.
sudo mkdir -p /etc/prometheus-node-exporter sudo vi /etc/prometheus-node-exporter/web-config.yml
Paste the following into the web-config.yml file:
basic_auth_users: node_exp_user: '$2y$05$zFPg83a9V08y2jdAD5v1ROvjpnFtjYe23SHafG9xFPnC0i9Fk2vS2' tls_server_config: cert_file: /etc/prometheus-node-exporter/<node>.crt key_file: /etc/prometheus-node-exporter/<node>.key
3: Update and Restart node_exporter Service
Edit the node_exporter systemd service file to point to our new web config. Find the ExecStart line and append the --web.config.file flag. Then, reload and restart the service.
-
Edit the service file:
NOTE: The file location could be different if installation is customized. It's always recommended to check the location of service file by running
systemctl status node_exportercommand.sudo vi /usr/lib/systemd/system/node_exporter.service
-
Find the
ExecStartline and append the following flag to the end of it:Before:
[Service] ... ExecStart=/usr/local/bin/node_exporter ...
After:
[Service] ... ExecStart=/usr/local/bin/node_exporter --web.config.file="/etc/prometheus-node-exporter/web-config.yml" ...
-
Reload and restart:
sudo systemctl daemon-reload sudo systemctl restart node_exporter sudo systemctl status node_exporter
Verify the service is active and running. Accessing https://<DATA-NODE-IP>:9300/metrics should now fail without authentication.
Important: Repeat all sub-steps in Step 1 for every data node in your universe.
Step 2: Reconfigure YBA Prometheus to Scrape Secured Nodes
Finally, we return to the YBA host to tell the central Prometheus instance how to connect to the newly secured node_exporter endpoints.
1: Edit Prometheus Scrape Configuration
Edit the main prometheus.yml file on the YBA host. The default file location is:
/opt/yugabyte/software/active/yba_installer/templates/yba-installer-prometheus.yml
Find the job named "node" and add the scheme, tls_config, and basic_auth sections to tell Prometheus how to connect via HTTPS with the correct credentials.
Before:
- job_name: "node"
file_sd_configs:
- files:
- .../node.*.json
After:
- job_name: "node"
scheme: https
tls_config:
insecure_skip_verify: true
basic_auth:
username: node_exp_user
password_file: /etc/prometheus/node_exporter.pass # <-- Path to the file containing the password
file_sd_configs:
- files:
- .../node.*.json
NOTE:
insecure_skip_verify: trueis often necessary in environments where the node certificates are self-signed or not signed by a CA that the Prometheus host trusts. If you have a proper PKI infrastructure, you can configuretls_configmore securely.
2: Restart and Verify
Restart the Prometheus service for the changes to take effect.
systemctl restart prometheus
After a few minutes, check the YBA UI. The data nodes should now appear as reachable, and metrics should begin populating again.
Congratulations! Your Prometheus node_exporter endpoints are now secured.
Troubleshooting
If you encounter issues after following the guide, here are a few common problems and their solutions.
Problem: node_exporter service fails to start on a data node.
-
Symptom: The command
systemctl status node_exportershows an(auto-restart)status and the Main PID has a(code=exited, status=1/FAILURE)message. -
Hint: Check the Service Logs To see the exact error causing the failure, check the
node_exporterservice logs on the data node:sudo journalctl -u node_exporter -n 50 --no-pager
-
Solution 1: Check Certificate Permissions. This is the most common cause. The service log will contain an error like
permission denied.- Verify ownership and permissions with:
ls -l /etc/prometheus-node-exporter/. -
The owner must match the user the service runs as (e.g.,
prometheusoryugabyte). The permissions should be-r--------(400). If not, re-run thechownandchmodcommands from Step 1 with the correct user.
- Verify ownership and permissions with:
-
Solution 2: Check for Typos.
- Carefully check the file path in the
node_exporter.servicefile to ensure it exactly matches the path to yourweb-config.yml. - Verify that the
cert_fileandkey_filepaths insideweb-config.ymlare correct.
- Carefully check the file path in the
Problem: Nodes are "Unreachable" in the YBA UI.
- Symptom: The nodes in the YBA UI are red and show a status of "Unreachable" after completing all steps.
-
Hint: Check the Prometheus Logs To see if Prometheus is having trouble scraping the nodes, check its logs on the YBA host:
sudo journalctl -u prometheus -n 50 --no-pager
Look for errors related to the data node IPs, such as
tls handshake errororauthentication failed. -
Solution 1: Check for Password Mismatch.
- The plain-text password inside
/etc/prometheus/node_exporter.passon the YBA host must be the exact same password used to generate the bcrypt hash withhtpasswdon the data nodes.
- The plain-text password inside
-
Solution 2: Test Connectivity and Authentication.
-
From the YBA host, run the following
curlcommand. It should successfully return a large page of metrics.curl -kv https://<DATA-NODE-IP>:9300/metrics --user "node_exp_user:YourNodeExporterPassword"
- If
curlfails, it could indicate a firewall is blocking port 9300 between the YBA host and the data node. If it prompts for a password again or gives an authentication error, the passwords do not match.
-
-
Solution 3: Verify
prometheus.ymlConfiguration.- On the YBA host, double-check that the
scheme,tls_config, andbasic_authsections were added correctly under thejob_name: "node"in yourprometheus.ymlfile. - Ensure you restarted the main
prometheusservice after making changes.
- On the YBA host, double-check that the
-
Solution 4: Verify if the Prometheus targets are healthy by opening Prometheus UI > Targets. If UI access is not available then:
-
Run the following command and then search in the output file for
lastErrorcurl -k -u <username:passwd> https://YBA-ip:9090/api/v1/targets > /tmp/targets.json
-
Reference: SUPPORT-579
Comments
0 comments
Please sign in to leave a comment.