Skip to main content

How to enable Basic Authentication and secure Node Exporter with TLS in a YugabyteDB Universe

Ashok Gangwar
Updated

Environment

  • YugabyteDB Anywhere (YBA): 2.20.x and higher. The configuration files and paths are specific to this YBA version but the principles apply to other modern versions.
  • YugabyteDB: All versions. The node_exporter component is independent of the database version, and securing it follows a standard procedure.

Issue Description

By default, the Prometheus instance managed by YugabyteDB Anywhere (YBA) and the node_exporter services on the database nodes are configured without encryption or authentication. The Prometheus web UI (port 9090) and the node exporter metrics endpoints (port 9300) are served over plain HTTP and allow unauthenticated access. This can expose potentially sensitive metrics and configuration data, which is not ideal for production or security-conscious environments.

Overview

The default configuration of Prometheus and its node_exporter services prioritizes ease of setup over security. To harden these components, a comprehensive, multi-part approach is required to enable both TLS encryption (HTTPS) and basic authentication.

  1. Secure the main Prometheus service on the YBA host: This involves enabling HTTPS and basic authentication for the Prometheus web UI (port 9090).
  2. Secure the node_exporter service on all YugabyteDB data nodes: This involves enabling HTTPS and basic authentication on each individual database node's metrics endpoint (port 9300).
  3. Reconfigure the YBA Prometheus service to scrape the newly secured data nodes: After securing the node_exporter endpoints, the central Prometheus instance must be updated to use the new HTTPS scheme and credentials to successfully scrape metrics.

Failure to complete all parts will result in either an incomplete security setup or a loss of metrics in the YBA UI, where nodes will appear as "Unreachable".

Prerequisites

This article is Part 2 of the series on securing Prometheus. It assumes that you have already secured the main Prometheus service on the YBA host.

A Note on Password Security: Plain-Text vs. Hashed

You will notice that we configure the node_exporter (in Step 1) with a secure bcrypt password hash. This is excellent security practice for the server side.

However, for the Prometheus server to connect as a client, the configuration in prometheus.yml (in Step 2) requires the actual password to be stored in plain text. This plain-text password in the main Prometheus configuration file is the specific security risk this note addresses.

Is there a way to avoid this plain-text password?

Yes. For production environments, the recommended approach is to store the password in a separate, protected file and reference that file from the configuration. This prevents the plain-text password from being directly visible in prometheus.yml.

Recommended for Production: Using a password_file

Here is the recommended method for securely providing the password on the Prometheus server.

1. Create the Password File On the YBA host (where the main Prometheus server is running), create a new file that contains only the password.

# Create a directory if it doesn't exist
sudo mkdir -p /etc/prometheus/

# Create the password file
echo 'YourNodeExporterPassword' | sudo tee /etc/prometheus/node_exporter.pass

2. Secure the Password File It is critical to restrict permissions on this file so that only the user running the Prometheus service can read it.

Important: Verify the Service User The Prometheus service may not run as the prometheus user. Before proceeding, check the correct user by running systemctl status prometheus on the YBA host and looking for the User: field. Use that username in the chown command below.

# Set read/write permissions for the owner only
sudo chmod 600 /etc/prometheus/node_exporter.pass

# Set the owner to the correct service user (e.g., prometheus:prometheus or yugabyte:yugabyte)
sudo chown prometheus:prometheus /etc/prometheus/node_exporter.pass

The configuration in Step 2 of this guide has been updated to use this more secure method.

This Guide's Process

This guide focuses on securing the node_exporter endpoints on your data nodes and reconfiguring the main Prometheus instance to scrape them securely. We will enable TLS (HTTPS) and basic authentication to protect your metrics data.

Important Note on Upgrades

The manual configurations applied in this guide to the node_exporter service file and the prometheus.yml file may be overwritten during a YugabyteDB universe or YugabyteDB Anywhere (YBA) upgrade. After performing any upgrade, you must verify that these security configurations are still in place and re-apply them if necessary.

 

Step 1: Secure Node Exporter on each YugabyteDB Data Node

This phase involves securing the metrics endpoint on every individual database node. These steps must be repeated for each data node in your universe.

1: Create Password Hash

SSH into a data node. node_exporter requires a bcrypt password hash. Use htpasswd to generate one. The command will output a string like node_exp_user:$2y$05$... - copy this entire string for the next step.

# On RHEL/CentOS, you may need to run: sudo yum install httpd-tools
# On Debian/Ubuntu: sudo apt-get install apache2-utils

htpasswd -cbB /tmp/node-exporter-creds.txt node_exp_user YourNodeExporterPassword

Note: You can also use another Python based method to generate a bcrypt password hash. 

2: Create Web Configuration File

Copy Certificates and Set Permissions

To avoid "permission denied" errors, we must copy the node's certificate and key to a standardized location and ensure the node_exporter service can read them. 

  1. Create the configuration directory:

    sudo mkdir -p /etc/prometheus-node-exporter

  2. Copy the certificate and key (replace the source path with your actual certificate location, e.g., /home/yugabyte/yugabyte-tls-config/node.<IP>.crt):

    sudo cp /home/yugabyte/yugabyte-tls-config/node* /etc/prometheus-node-exporter/*
  3. Set correct ownership and permissions: 

Important: Verify the Service User The node_exporter service may run as a different user (e.g., yugabyte). Before proceeding, check the correct user by running systemctl status node_exporter on the data node and looking for the User: field. Use that username in the chown command below.

The node_exporter service often runs as root, so we give root ownership and restrict permissions for security.

sudo chown prometheus:prometheus /etc/prometheus-node-exporter/*
sudo chmod 400 /etc/prometheus-node-exporter/*

Create and Populate Web Configuration File

Create a directory and a new configuration file for node_exporter. Paste the content below, replacing the placeholder hash with the one you just generated and providing the correct paths to your node's TLS certificate and key.

sudo mkdir -p /etc/prometheus-node-exporter
sudo vi /etc/prometheus-node-exporter/web-config.yml

Paste the following into the web-config.yml file:

basic_auth_users:
  node_exp_user: '$2y$05$zFPg83a9V08y2jdAD5v1ROvjpnFtjYe23SHafG9xFPnC0i9Fk2vS2'

tls_server_config:
  cert_file: /etc/prometheus-node-exporter/<node>.crt
  key_file: /etc/prometheus-node-exporter/<node>.key

3: Update and Restart node_exporter Service

Edit the node_exporter systemd service file to point to our new web config. Find the ExecStart line and append the --web.config.file flag. Then, reload and restart the service.

  1. Edit the service file:

    NOTE: The file location could be different if installation is customized. It's always recommended to check the location of service file by running  systemctl status node_exporter command. 

    sudo vi /usr/lib/systemd/system/node_exporter.service

  2. Find the ExecStart line and append the following flag to the end of it:

    Before:

    [Service]
...
ExecStart=/usr/local/bin/node_exporter
...

    After:

    [Service]
...
ExecStart=/usr/local/bin/node_exporter --web.config.file="/etc/prometheus-node-exporter/web-config.yml"
...

  3. Reload and restart:

    sudo systemctl daemon-reload
sudo systemctl restart node_exporter
sudo systemctl status node_exporter

Verify the service is active and running. Accessing https://<DATA-NODE-IP>:9300/metrics should now fail without authentication.

Important: Repeat all sub-steps in Step 1 for every data node in your universe.

 

Step 2: Reconfigure YBA Prometheus to Scrape Secured Nodes

Finally, we return to the YBA host to tell the central Prometheus instance how to connect to the newly secured node_exporter endpoints.

1: Edit Prometheus Scrape Configuration

Edit the main prometheus.yml file on the YBA host. The default file location is:

/opt/yugabyte/software/active/yba_installer/templates/yba-installer-prometheus.yml

Find the job named "node" and add the scheme, tls_config, and basic_auth sections to tell Prometheus how to connect via HTTPS with the correct credentials.

Before:

- job_name: "node"
  file_sd_configs:
  - files:
    - .../node.*.json

After:

- job_name: "node"
  scheme: https
  tls_config:
    insecure_skip_verify: true
  basic_auth:
    username: node_exp_user
    password_file: /etc/prometheus/node_exporter.pass # <-- Path to the file containing the password
  file_sd_configs:
  - files:
    - .../node.*.json

NOTE: insecure_skip_verify: true is often necessary in environments where the node certificates are self-signed or not signed by a CA that the Prometheus host trusts. If you have a proper PKI infrastructure, you can configure tls_config more securely.

2: Restart and Verify

Restart the Prometheus service for the changes to take effect. 

systemctl restart prometheus

After a few minutes, check the YBA UI. The data nodes should now appear as reachable, and metrics should begin populating again.

Congratulations! Your Prometheus node_exporter endpoints are now secured.

 

Troubleshooting

If you encounter issues after following the guide, here are a few common problems and their solutions.

Problem: node_exporter service fails to start on a data node.

  • Symptom: The command systemctl status node_exporter shows an (auto-restart) status and the Main PID has a (code=exited, status=1/FAILURE) message.

  • Hint: Check the Service Logs To see the exact error causing the failure, check the node_exporter service logs on the data node:

    sudo journalctl -u node_exporter -n 50 --no-pager
  • Solution 1: Check Certificate Permissions. This is the most common cause. The service log will contain an error like permission denied.
    • Verify ownership and permissions with: ls -l /etc/prometheus-node-exporter/.
    • The owner must match the user the service runs as (e.g., prometheus or yugabyte). The permissions should be -r-------- (400). If not, re-run the chown and chmod commands from Step 1 with the correct user.
  • Solution 2: Check for Typos.
    • Carefully check the file path in the node_exporter.service file to ensure it exactly matches the path to your web-config.yml.
    • Verify that the cert_file and key_file paths inside web-config.yml are correct.

Problem: Nodes are "Unreachable" in the YBA UI.

  • Symptom: The nodes in the YBA UI are red and show a status of "Unreachable" after completing all steps.

  • Hint: Check the Prometheus Logs To see if Prometheus is having trouble scraping the nodes, check its logs on the YBA host:

    sudo journalctl -u prometheus -n 50 --no-pager

    Look for errors related to the data node IPs, such as tls handshake error or authentication failed.

  • Solution 1: Check for Password Mismatch.
    • The plain-text password inside /etc/prometheus/node_exporter.pass on the YBA host must be the exact same password used to generate the bcrypt hash with htpasswd on the data nodes.
  • Solution 2: Test Connectivity and Authentication.

    • From the YBA host, run the following curl command. It should successfully return a large page of metrics.

      curl -kv https://<DATA-NODE-IP>:9300/metrics --user "node_exp_user:YourNodeExporterPassword"
    • If curl fails, it could indicate a firewall is blocking port 9300 between the YBA host and the data node. If it prompts for a password again or gives an authentication error, the passwords do not match.
  • Solution 3: Verify prometheus.yml Configuration.
    • On the YBA host, double-check that the scheme, tls_config, and basic_auth sections were added correctly under the job_name: "node" in your prometheus.yml file.
    • Ensure you restarted the main prometheus service after making changes.
  • Solution 4: Verify if the Prometheus targets are healthy by opening Prometheus UI > Targets. If UI access is not available then:

    • Run the following command and then search in the output file for lastError 

      curl -k -u <username:passwd> https://YBA-ip:9090/api/v1/targets > /tmp/targets.json

 

 

Reference: SUPPORT-579

Related articles

Comments

0 comments

Please sign in to leave a comment.

Powered by Zendesk