Testing Strategy

Testing strategy by change scope

Validation depth by change scope across unit, integration, E2E, fuzzing, and exploratory coverage.

Diagram

Testing layers

Move to the next layer when the lower-cost layer no longer proves the behavior you changed.

Decision matrix

Choose test depth by change scope

Choose test depth by change scope.

If your change affects	Run this first	What it proves
Pure Go logic, parsers, renderers, helpers, or small decision functions	make test or targeted go test ./...	Deterministic in-process behavior with no Kubernetes API dependency.
Object builders, manifests, patches, or fake-client contract behavior	make test plus targeted package tests	Resource shape and emitted contracts when API-server semantics do not matter.
Reconciliation, finalizers, status writes, admission behavior, or server-side validation/defaulting	make test-integration	Real API-server semantics through EnvTest.
Lifecycle flows, networking, storage, upgrades, backup and restore, or workload startup	make test-e2e or a label-filtered make test-e2e-ci ... run	Real cluster behavior with real images and controller wiring.
Disaster recovery, performance thresholds, or compatibility against an existing platform cluster	Focused manual or scheduled workflow validation	Evidence from the environment that production-like assumptions actually hold.

Fake client boundary

Use the controller-runtime fake client as a fast contract tool, not as a substitute for the API server. If the test depends on validation, defaulting, Generation or ResourceVersion behavior, subresources, watches, cache wiring, or controller-manager setup, move to EnvTest.

Verify

PR-equivalent local gate

bash

make bootstrap
make doctor
make ci-core

Standard local validation baseline before review or handoff.

Inspect

Focused fuzz and existing-cluster checks

bash

make fuzz
FUZZTIME=30s make fuzz
FUZZ_TARGET_FILTER='FuzzRenderHCL|internal/adapter/auth' make fuzz

export KUBECONFIG=/path/to/your/kubeconfig
export E2E_OPERATOR_IMAGE=quay.io/your-org/openbao-operator:dev
export E2E_API_SERVER_CIDR=0.0.0.0/0
make test-e2e-existing E2E_LABEL_FILTER='openshift'

These commands cover parser, auth, renderer, and platform-specific validation that does not fit the default local gate.

For E2E debugging, prefer structured reports over log-only inspection. Set E2E_JSON_REPORT for the native Ginkgo report, E2E_JUNIT_REPORT for CI-compatible test output, and E2E_FAIL_ON_EMPTY=true when using a label filter so an accidental empty selection fails immediately.

E2E suites are declared in test/e2e/suites.yaml. When adding or changing an E2E file, update the manifest with the suite owner, risk tier, isolation class, labels, coverage tags, CI lane, and nightly policy, then run make verify-e2e-manifest.

Run the manual HSM/KMIP E2E lane locally

bash

make docker-build-e2e-openbao-softhsm OPENBAO_SOFTHSM_IMG=openbao-softhsm:dev
make docker-build-e2e-pykmip-server PYKMIP_SERVER_IMG=pykmip-server:dev

E2E_ENABLE_SOFTHSM_SUITE=true \
E2E_ENABLE_KMIP_SUITE=true \
E2E_OPENBAO_IMAGE=openbao-softhsm:dev \
E2E_KMIP_SERVER_IMAGE=pykmip-server:dev \
E2E_LABEL_FILTER='hsm && !openshift' \
make test-e2e

The HSM lane is manual because it needs test-only images and provider fixtures. The KMIP suite uses a PyKMIP server fixture with mTLS and a pre-seeded active AES key; the PKCS#11 suite uses a SoftHSM-enabled OpenBao image.

Decision matrix

Special validation lanes

Special validation lanes.

Lane	When to use it	Primary entry point
Fuzzing	Parser, renderer, auth, or normalization changes that benefit from mutated input coverage.	make fuzz locally, make fuzz-long in longer sweeps.
Performance	Controller or lifecycle changes that may affect reconcile cost, startup time, or upgrade behavior.	make verify-perf and the Performance Baseline Capture workflow.
Existing-cluster compatibility	OpenShift or platform-specific validation that a local Kind cluster cannot faithfully represent.	make test-e2e-existing ... with a preconfigured cluster context.

Performance scenarios are declared in hack/perf/scenarios.yaml. Each scenario owns its Ginkgo label filter and the metric policies that are meaningful for that workflow. Keep those policies narrow: add a metric only when the scenario is expected to exercise that behavior, and use warning severity for broad diagnostic counters such as aggregate workqueue retries.

After test selection

Continuous integrationContinuous integration maps these validation layers to workflow gates and routing.Coding standardsCoding standards cover implementation cleanup when more test depth would not help yet.Release managementRelease management covers release and publish flow once the branch is stable.