Observability

KMS v2 Status is the health signal kube-apiserver uses. The HTTP endpoints, metrics, and logs on this page serve node-local operations and monitoring. None of them carry secrets, and key_id values appear only as hashes. For scrape setup and the dashboard, see Configure: Monitor the provider .

Endpoints

/ready can fail while API server reads still succeed from its cache, which makes it the earliest warning.

The response is a cached view; requesting /ready does not contact OpenBao. An unhealthy response includes reasons, a list of bounded condition codes. metadata_error_class and deep_error_class, when present, identify the last failed or skipped probe. Successful probes clear their own failure details. A healthy response has an empty reasons list. For example:

{"status":"unavailable","healthz":"unhealthy","cache_age_ms":120,"stale":false,"rotation_state":"active","reasons":["metadata_read_failed"],"metadata_error_class":"tls_failed"}

The optional persistence_degraded: true field reports a deferred pending observation save. Readiness can remain HTTP 200 only while the published keys remain validated and both persistence files are confirmed unchanged. The field does not override other health failures and clears after successful state publication. See Persistence failures .

Metrics

OAuth token acquisition failures use bounded status values on openbao_kms_auth_login_total: oauth2_credential for local credential failures, oauth2_request for transport failures, oauth2_unavailable for HTTP 429 or 5xx, oauth2_rejected for other non-200 responses, and oauth2_response for invalid responses. JWT claim failures retain the existing JWT status classes. Remote response bodies and credentials are never included in these labels or error messages.

Labels hold only bounded values. key_id values are exported as base64url-sha256 hashes. Raw OpenBao paths, key names, Kubernetes object names, request UIDs, and error strings never appear as labels.

The operation label takes jwt_login, cert_login, token_renew_self, transit_metadata_read, transit_disable_upsert_read, transit_encrypt, transit_decrypt, transit_batch_decrypt, or capabilities_self. Logs use the same names with spaces instead of underscores.

Logs

serve writes structured JSON (logging.format: json). Successful high-frequency KMS and OpenBao requests log at debug level; failures log as warnings. These fields are stable across preview patch releases:

Logs never contain plaintext, JWTs, OpenBao tokens, full ciphertext, key material, full annotation maps, or, by default, raw OpenBao paths and key names.

At the default info level, serve.start marks the start of runtime setup. serve.ready follows successful bootstrap probes and listener startup. It is a startup event; use /ready or KMS Status for ongoing health. serve.shutdown follows startup failure or runtime exit. Failures use startup_failed or runtime_failed without raw error details. Invalid configuration or logging settings can fail before structured logging starts.

key.promoted logs once when a previously active key changes, after state is saved and published. Bootstrap, pending observations, and failed state saves do not emit promotion events. The new key still needs its deep probe before Encrypt becomes available. Probe failures log at warning level with reason and error_class; successful periodic probes remain at debug level.

A deferred observation save logs reason: state_save_failed and increments the failed metadata-probe counter even when cached KMS Status remains healthy.

Error classes

Failed KMS calls carry a bounded error_class. Use these classes as alert routing keys and dashboard groups.

Missing or empty decrypt annotations use annotation_invalid. aad_missing is retained for an unsupported AAD mode in an internal snapshot; validated configuration and state reject that mode before serving requests.

Token errors keep their cause: local credential failures and rejected logins are auth_failed, while transport, unavailable, sealed, rate-limited, canceled, and timed-out auth calls keep their cause classes. A Transit 403 that persists after the single recovery attempt, or while recovery is throttled, is transit_policy_denied, because OpenBao does not always tell a revoked token from a policy denial. OpenBao request metrics count both the rejected attempt and the retry.

Typed certificate verification and TLS record failures use openbao_tls_failed; DNS failures use openbao_dns_failed; socket connection failures, including connection refusal and reset, use openbao_connection_failed. These KMS errors still return gRPC Unavailable. Unrecognized transport failures remain openbao_unavailable. No destination, certificate, or raw transport error is logged. OpenBao request logs and their metric status label use tls_failed, dns_failed, and connection_failed without the openbao_ prefix.

Status probe logs and readiness probe error fields use the OpenBao classes without a prefix: invalid_request, unauthenticated, permission_denied, not_found, decrypt_failed, rate_limited, unavailable, sealed, tls_failed, dns_failed, connection_failed, and unknown. Rejected logins and local credential failures use auth_failed; cancellation and deadlines use canceled and timeout. Local state or validation failures use their readiness reason code as the error class. Raw OpenBao responses and local filesystem errors are never copied into probe log fields or readiness responses.

Alerts

Alert on: KMS Status unhealthy, stale Status cache, OpenBao error rate, login or renewal failures, low token TTL, different key_id hashes across nodes, a rotation stuck pending, AAD or unknown key_id errors, encrypt or decrypt latency, rising concurrency rejections, restart loops, and stale socket reclaims.

Alert when openbao_kms_rotation_persistence_degraded == 1 persists across probes. Repair state storage before the next key transition; healthy readiness during a deferred observation save does not mean rotation can continue.

Starting rules ship in deploy/prometheus/rules/openbao-kms.rules.yaml. Tune their thresholds to your OpenBao latency, probe cadence, token TTLs, and scrape topology before paging on them.

Debug correlation

Debug correlation temporarily adds request_uid_hash, openbao_request_id, and the incident fields to debug logs so you can match provider logs with kube-apiserver and OpenBao audit records. It is off by default and turns on only when all of these hold:

• logging.level: debug and logging.logOpenBaoRequestIDs: true,
• logging.debugCorrelation.incidentId is set,
• logging.debugCorrelation.ttl is positive and at most one hour.

It switches itself off when the TTL expires, without a restart, and never relaxes the logging rules above. OpenBao request IDs are never stored in KMS annotations.

logging:
  level: debug
  logOpenBaoRequestIDs: true
  debugCorrelation:
    enabled: true
    ttl: 15m
    incidentId: INC-12345

Clock corrections and cached health

Status cache age is the greatest age observed from elapsed time or wall time. A backward clock correction cannot make the cache younger. Once cached status is stale, only a fresh successful metadata probe can restore metadata health; the existing deep-probe requirements still apply. A forward correction or host suspension can expire the cache early. Status continues to use cached evidence and does not contact OpenBao.

Retry, discovery, and circuit-breaker cooldowns use elapsed time. UTC deadline fields are projections onto the current wall clock. Token TTL and cache-age metrics report the conservative duration used by the corresponding validity check.

clock.regressed warns that a new rotation bookkeeping timestamp was floored to preserve event order after a wall-clock regression. Its bounded reason is timestamp_order_preserved. The log timestamp shows current wall time. The persisted pending stable timestamp is historical evidence, not the current activation deadline; a restart starts the full delay again.