A provider failure can stop the API server from starting or make cluster data unreadable, so tests favor negative paths, rotation, and recovery over a single happy-path round trip. For the runnable lanes, see E2E framework ; for CI stages and release evidence, see CI and supply chain .
What tests must prove
| Priority | Proof |
|---|---|
| KMS v2 correctness | Kubernetes accepts Status, Encrypt, and Decrypt, and Status.key_id == EncryptResponse.key_id always holds. |
key_id stability | key_id values are deterministic, never reused, and never roll back. |
| Decrypt compatibility | Data stays readable across restarts, upgrades, rollbacks, and rotations. |
| Fail-closed behavior | OpenBao, auth, socket, policy, and key failures never expose plaintext or fall back. |
| Startup and deployment | The provider is ready for API server restarts, with the distinct boot and socket behavior of systemd and static pods. |
| Recovery | Restores keep data readable only with a matching backup pair. |
| Redaction | Logs, metrics, reports, and artifacts never contain plaintext, JWTs, tokens, full ciphertext, or key material. |
Every negative test asserts both the error behavior and the absence of
sensitive values in logs, metrics, and artifacts. Negative cases cover OpenBao
outages, failover, and old restores, expired, rotated, or mismatched
credentials, missing, recreated, or restricted keys, unknown or rolled-back
key_id values, tampered or foreign AAD, unsafe sockets, and host failures
during restarts.
Test layers
| Layer | Covers | Where |
|---|---|---|
| Unit and golden | Registry decisions, AAD, configuration, sockets, redaction, wire-format fixtures | go test ./..., testdata/ |
| KMS v2 conformance | The real socket server with fake OpenBao and KMS v2 protobuf requests | test/kmsconformance, every pull request |
| OpenBao integration | Transit, auth, TLS, policy diagnostics, error handling | Hermetic integration tests and the OpenBao lane |
| CLI | Diagnostics and hardening failures against real OpenBao | Provider CLI lane |
| API server | Real encryption, raw etcd envelopes, restarts, multi-node convergence | Kind lanes and kubeadm VM validation |
| Rotation and compatibility | Promotion, old readback, retirement, missing state, rollback rejection | Rotation and upgrade lanes |
| Failure injection | Outages, sealing, failover, revoked tokens, bad policy or auth, missing keys, stale sockets | Failure and HA lanes |
| Performance | Status, Encrypt, and Decrypt latency, soak, startup decrypt, resource growth | Load lanes; targets in KMS v2 contract |
| Security and supply chain | Redaction, fuzzing, static analysis, vulnerability and license scans, SBOM, vendor verification | make ci-core, security CI, release workflow |
| Recovery | Raft restore, state rehydration, etcd pairing, readback after replacement | Kind DR, restore lane, VM validation |
The Kind smoke lane uses the generated static-pod manifest. It retires the JWT signing key, revokes provider tokens, checks authentication failure, and then atomically replaces the host JWT. Recovery must decrypt the existing KMS sample and encrypt new data without restarting the provider. The lane also restarts the API server and reads the existing encrypted Secret.
Install regression check
make systemd-install-check builds a systemd bundle and runs the shell blocks
from Run with systemd
in a disposable Linux
container, using the pinned Go builder image with Debian systemd tools. It
checks service-user file access, socket group separation, unit syntax, and that
re-running the install keeps configuration and state. Building the test image
needs network access; the check itself does not.
The Deployment Samples CI job runs it when installation docs, packaging, the bundle builder, or deployment tests change. It does not start systemd, log in to OpenBao, or test Kubernetes boot; the VM and E2E lanes do.
make static-pod-install-check runs the host preparation commands from the
static-pod kit’s README, then generates files for both JWT sources with the
bundled binary. It checks UID/GID 65532 access, socket-group isolation,
matching identity on a second node, and preservation on reinstall. It does
not start kubelet or claim cluster activation. The Deployment Samples job
runs this check too.
Set IMAGE_PLATFORM=linux/amd64 or linux/arm64 to choose the Linux test
architecture. Docker needs native support or emulation for that platform.
To test an existing systemd or static-pod archive, set BUNDLE_ARCHIVE to its
path relative to the repository root. The container runs without network
access. Systemd archives containing the minimal values files also exercise
both JWT sources and identity reuse through the packaged init command.
The release workflow runs both checks on amd64 and arm64 using the exact
archives selected for publication. It also installs the selected .deb and
.rpm artifacts through dpkg and rpm in separate Debian and AlmaLinux
containers. Run that check with:
make native-package-install-check IMAGE_PLATFORM=linux/amd64 \
PACKAGE_VERSION=0.1.0-preview.3 \
PACKAGE_FILE=dist/primary/bao-kms-provider_0.1.0-preview.3_linux_amd64.debThe package check verifies the executable version and ELF architecture,
maintainer-script account and directory setup, access as the service user,
socket-group isolation, configuration retention across reinstall and removal,
and the absence of automatic activation. RPM checks use the digest-pinned
AlmaLinux fixture in .ci/versions.yaml. Installation runs offline after the
test image is prepared. Archive checks also inspect ELF architecture so QEMU
cannot conceal a package containing the wrong binary.
Publication depends on these jobs passing. This gate covers host installation and file generation; API-server activation, credential recovery, and VM boot remain separate qualification steps.
Fuzzing
make ci-core runs short fuzz smoke campaigns (FUZZTIME=10s). To run longer
campaigns locally:
FUZZTIME=1m make fuzz
go test ./internal/keyregistry -run '^$' -fuzz '^FuzzStateFileDecode$' -fuzztime=5m
go test ./internal/aad -run '^$' -fuzz '^FuzzPrepareDecrypt$' -fuzztime=5m