Configure a secure metrics scrape
Use this how-to to configure an authenticated OpenBao metrics scrape for a Kubernetes HA deployment. The profile scrapes the active OpenBao node over TLS with a scoped OpenBao token.
Before you begin
- Run OpenBao outside development mode.
- Use TLS on the OpenBao listener that serves metrics.
- Install Prometheus Operator if you use the
ServiceMonitorexample. - Enable Kubernetes service registration or provide an equivalent active Service that only selects the active OpenBao pod.
- Get an OpenBao token that can create policies and child tokens.
- Store the OpenBao serving CA certificate in a file named
ca.crt.
Choose the scrape profile
| Profile | Use when | Authentication | Target | Caveat |
|---|---|---|---|---|
| Authenticated active scrape | You need the secure default for production. | Required OpenBao token. | Active Service. | Standby node state is not visible. |
| Private all-node scrape | You need per-node HA and Raft visibility. | Token or isolated unauthenticated metrics listener. | Each OpenBao pod. | Requires strict network isolation. |
| Local demo scrape | You run the Docker Compose stack in this repo. | None. | All Compose nodes. | Not for production or shared environments. |
Use authenticated active scrape as the baseline. Add a private all-node scrape only when you have a metrics-only listener and network controls that prevent general clients from reaching it.
Configure OpenBao telemetry
Enable Prometheus retention in the OpenBao server configuration.
telemetry { prometheus_retention_time = "30s" disable_hostname = true }Disable metrics on the primary client listener when you use a separate metrics listener.
listener "tcp" { address = "0.0.0.0:8200" cluster_address = "0.0.0.0:8201" tls_cert_file = "/openbao/tls/tls.crt" tls_key_file = "/openbao/tls/tls.key" telemetry { disallow_metrics = true } }If you scrape the primary listener instead, leave
disallow_metricsunset and keepunauthenticated_metrics_access = false.Use a metrics-only listener when you want to separate client traffic from scrape traffic.
listener "tcp" { address = "0.0.0.0:8202" tls_cert_file = "/openbao/tls/tls.crt" tls_key_file = "/openbao/tls/tls.key" telemetry { metrics_only = true unauthenticated_metrics_access = false } }Enable Kubernetes service registration when you want an active Service based on OpenBao pod labels.
service_registration "kubernetes" {}Set
BAO_K8S_NAMESPACEandBAO_K8S_POD_NAMEthrough the Kubernetes Downward API when you do not setnamespaceandpod_namein theservice_registrationstanza.
Create the metrics policy
Write a metrics-only policy.
cat > openbao-metrics-policy.hcl <<'EOF' path "sys/metrics" { capabilities = ["read", "list"] } EOFApply the policy.
bao policy write openbao-metrics openbao-metrics-policy.hclCreate a token with only the metrics policy.
OPENBAO_METRICS_TOKEN="$(bao token create \ -policy=openbao-metrics \ -field=token)"Use your normal secret-management process to set token TTL, renewal, and rotation. Prometheus reads the token; it does not manage the OpenBao token lifecycle for you.
Store the token in Kubernetes.
kubectl -n openbao create secret generic openbao-metrics-token \ --from-literal=token="${OPENBAO_METRICS_TOKEN}"Store the OpenBao serving CA certificate.
kubectl -n openbao create configmap openbao-metrics-ca \ --from-file=ca.crt=./ca.crt
Configure the active ServiceMonitor
Review the example manifest.
less examples/kubernetes/secure-metrics-scrape.yamlThe manifest creates the active metrics Service and
ServiceMonitor. It references theopenbao-metrics-tokenSecret andopenbao-metrics-caConfigMap that you created earlier.Update the selectors to match your OpenBao deployment labels.
The example Service selects pods with
openbao-active: "true", which OpenBao sets when Kubernetes service registration is enabled.Update the
ServiceMonitorlabels so your Prometheus resource selects it.Many kube-prometheus-stack installations require a release label such as
release: kube-prometheus-stack.Apply the manifest.
kubectl apply -f examples/kubernetes/secure-metrics-scrape.yaml
Verify the result
Check that the active metrics Service has one endpoint.
kubectl -n openbao get endpoints openbao-active-metricsCheck that Prometheus Operator picked up the
ServiceMonitor.kubectl -n openbao get servicemonitor openbao-active-metricsQuery Prometheus.
up{job="openbao"}Expected result: one active target with value
1.Check the active node metric.
vault_core_activeExpected result: one series with value
1.Test the OpenBao endpoint directly from a pod that can reach the Service.
curl --cacert ca.crt \ --header "Authorization: Bearer ${OPENBAO_METRICS_TOKEN}" \ "https://openbao-active-metrics.openbao.svc:8202/v1/sys/metrics?format=prometheus"Expected result: Prometheus text that includes
vault_core_active.
Troubleshooting
Prometheus does not discover the ServiceMonitor
Check the labels on the ServiceMonitor and the selector on your Prometheus
resource. Prometheus Operator only includes ServiceMonitor objects that match
the Prometheus selector and namespace selector.
The target returns 403
Check the OpenBao token policy. The token must include access to sys/metrics.
Also check that the Kubernetes Secret key name matches the authorization
credentials in the ServiceMonitor.
The target has a TLS error
Check the CA certificate and serverName in tlsConfig. Do not use
insecureSkipVerify: true for the production profile.
The active Service has no endpoints
Check that Kubernetes service registration is enabled and that OpenBao can patch its pod labels. The OpenBao service account needs permissions to get, update, and patch its pod.
What’s next
- Use OpenBao observability model to understand how metrics fit with logs, audit logs, and platform signals.
- Use Active-node and all-node observability to understand what this scrape profile shows and hides.
- Use Understanding OpenBao metrics to understand source metrics, labels, and recording rules.
- Use Configure an all-node metrics scrape when you need standby, sealed-node, or per-pod Raft visibility.
- Use Run the Docker Compose stack when you need a local all-node HA scrape for dashboard validation.
- Use OpenBao metrics scrape failing
when the
OpenBaoUnreachablealert fires.
Source: OpenBao documents the /sys/metrics endpoint in the
OpenBao metrics API documentation
. OpenBao documents
Prometheus telemetry options in the
OpenBao telemetry documentation
. OpenBao documents
listener-level metrics controls in the
OpenBao TCP listener documentation
. OpenBao documents
Kubernetes service registration labels in the
OpenBao Kubernetes service registration documentation
.
Prometheus Operator documents ServiceMonitor discovery and endpoint
configuration in the
Prometheus Operator design documentation
and
Prometheus Operator API reference
.