Configure a secure metrics scrape

Use this how-to to configure an authenticated OpenBao metrics scrape for a Kubernetes HA deployment. The profile scrapes the active OpenBao node over TLS with a scoped OpenBao token.

Before you begin

  • Run OpenBao outside development mode.
  • Use TLS on the OpenBao listener that serves metrics.
  • Install Prometheus Operator if you use the ServiceMonitor example.
  • Enable Kubernetes service registration or provide an equivalent active Service that only selects the active OpenBao pod.
  • Get an OpenBao token that can create policies and child tokens.
  • Store the OpenBao serving CA certificate in a file named ca.crt.

Choose the scrape profile

ProfileUse whenAuthenticationTargetCaveat
Authenticated active scrapeYou need the secure default for production.Required OpenBao token.Active Service.Standby node state is not visible.
Private all-node scrapeYou need per-node HA and Raft visibility.Token or isolated unauthenticated metrics listener.Each OpenBao pod.Requires strict network isolation.
Local demo scrapeYou run the Docker Compose stack in this repo.None.All Compose nodes.Not for production or shared environments.

Use authenticated active scrape as the baseline. Add a private all-node scrape only when you have a metrics-only listener and network controls that prevent general clients from reaching it.

Configure OpenBao telemetry

  1. Enable Prometheus retention in the OpenBao server configuration.

    telemetry {
      prometheus_retention_time = "30s"
      disable_hostname          = true
    }
    
  2. Disable metrics on the primary client listener when you use a separate metrics listener.

    listener "tcp" {
      address         = "0.0.0.0:8200"
      cluster_address = "0.0.0.0:8201"
      tls_cert_file   = "/openbao/tls/tls.crt"
      tls_key_file    = "/openbao/tls/tls.key"
    
      telemetry {
        disallow_metrics = true
      }
    }
    

    If you scrape the primary listener instead, leave disallow_metrics unset and keep unauthenticated_metrics_access = false.

  3. Use a metrics-only listener when you want to separate client traffic from scrape traffic.

    listener "tcp" {
      address       = "0.0.0.0:8202"
      tls_cert_file = "/openbao/tls/tls.crt"
      tls_key_file  = "/openbao/tls/tls.key"
    
      telemetry {
        metrics_only                   = true
        unauthenticated_metrics_access = false
      }
    }
    
  4. Enable Kubernetes service registration when you want an active Service based on OpenBao pod labels.

    service_registration "kubernetes" {}
    

    Set BAO_K8S_NAMESPACE and BAO_K8S_POD_NAME through the Kubernetes Downward API when you do not set namespace and pod_name in the service_registration stanza.

Create the metrics policy

  1. Write a metrics-only policy.

    cat > openbao-metrics-policy.hcl <<'EOF'
    path "sys/metrics" {
      capabilities = ["read", "list"]
    }
    EOF
    
  2. Apply the policy.

    bao policy write openbao-metrics openbao-metrics-policy.hcl
    
  3. Create a token with only the metrics policy.

    OPENBAO_METRICS_TOKEN="$(bao token create \
      -policy=openbao-metrics \
      -field=token)"
    

    Use your normal secret-management process to set token TTL, renewal, and rotation. Prometheus reads the token; it does not manage the OpenBao token lifecycle for you.

  4. Store the token in Kubernetes.

    kubectl -n openbao create secret generic openbao-metrics-token \
      --from-literal=token="${OPENBAO_METRICS_TOKEN}"
    
  5. Store the OpenBao serving CA certificate.

    kubectl -n openbao create configmap openbao-metrics-ca \
      --from-file=ca.crt=./ca.crt
    

Configure the active ServiceMonitor

  1. Review the example manifest.

    less examples/kubernetes/secure-metrics-scrape.yaml
    

    The manifest creates the active metrics Service and ServiceMonitor. It references the openbao-metrics-token Secret and openbao-metrics-ca ConfigMap that you created earlier.

  2. Update the selectors to match your OpenBao deployment labels.

    The example Service selects pods with openbao-active: "true", which OpenBao sets when Kubernetes service registration is enabled.

  3. Update the ServiceMonitor labels so your Prometheus resource selects it.

    Many kube-prometheus-stack installations require a release label such as release: kube-prometheus-stack.

  4. Apply the manifest.

    kubectl apply -f examples/kubernetes/secure-metrics-scrape.yaml
    

Verify the result

  1. Check that the active metrics Service has one endpoint.

    kubectl -n openbao get endpoints openbao-active-metrics
    
  2. Check that Prometheus Operator picked up the ServiceMonitor.

    kubectl -n openbao get servicemonitor openbao-active-metrics
    
  3. Query Prometheus.

    up{job="openbao"}
    

    Expected result: one active target with value 1.

  4. Check the active node metric.

    vault_core_active
    

    Expected result: one series with value 1.

  5. Test the OpenBao endpoint directly from a pod that can reach the Service.

    curl --cacert ca.crt \
      --header "Authorization: Bearer ${OPENBAO_METRICS_TOKEN}" \
      "https://openbao-active-metrics.openbao.svc:8202/v1/sys/metrics?format=prometheus"
    

    Expected result: Prometheus text that includes vault_core_active.

Troubleshooting

Prometheus does not discover the ServiceMonitor

Check the labels on the ServiceMonitor and the selector on your Prometheus resource. Prometheus Operator only includes ServiceMonitor objects that match the Prometheus selector and namespace selector.

The target returns 403

Check the OpenBao token policy. The token must include access to sys/metrics. Also check that the Kubernetes Secret key name matches the authorization credentials in the ServiceMonitor.

The target has a TLS error

Check the CA certificate and serverName in tlsConfig. Do not use insecureSkipVerify: true for the production profile.

The active Service has no endpoints

Check that Kubernetes service registration is enabled and that OpenBao can patch its pod labels. The OpenBao service account needs permissions to get, update, and patch its pod.

What’s next

Source: OpenBao documents the /sys/metrics endpoint in the OpenBao metrics API documentation . OpenBao documents Prometheus telemetry options in the OpenBao telemetry documentation . OpenBao documents listener-level metrics controls in the OpenBao TCP listener documentation . OpenBao documents Kubernetes service registration labels in the OpenBao Kubernetes service registration documentation . Prometheus Operator documents ServiceMonitor discovery and endpoint configuration in the Prometheus Operator design documentation and Prometheus Operator API reference .