E2E Framework

The E2E framework is the command reference for integration and end-to-end validation. It uses Ginkgo and Gomega specs, label-based routing, a suite manifest, pinned versions from .ci/versions.yaml, and machine-readable JUnit and Ginkgo JSON reports.

Unit tests and hermetic integration tests should stay free of external services. E2E lanes own real OpenBao, provider container, Kind, and Kubernetes API server behavior.

Command Matrix

Local kubeadm VM validation is a release-candidate gate, not public CI. It exercises host boot ordering, systemd, static-pod behavior, node reboot, paired restore, and multi-control-plane recovery in a VM substrate. Infrastructure-specific maintainer notes live next to the local harness implementation.

Labels

Labels are the routing API. Prefer small composable labels:

Label("openbao", "transit", "ci")
Label("openbao", "certauth", "ci")
Label("openbao", "kmsv2", "failure", "ci")
Label("openbao", "kmsv2", "ha", "ci")
Label("openbao", "kmsv2", "restore", "ci")
Label("openbao", "kmsv2", "rotation", "ci")
Label("openbao", "kmsv2", "soak", "ci")
Label("openbao", "kmsv2", "upgrade", "rollback", "ci")
Label("kind", "kmsv2", "smoke")
Label("kind", "kmsv2", "convergence")
Label("kind", "kmsv2", "restore", "dr")
Label("kind", "kmsv2", "upgrade", "rollback")

Use stable case:<id> labels when a spec becomes release evidence or a known regression target.

Run a label-filtered suite when Ginkgo is installed:

make test-e2e E2E_LABEL_FILTER='openbao && transit && ci'

Suite Manifest

test/e2e/suites.yaml describes E2E lanes, run selectors, timeouts, required environment, reports, and the preview release gate groups. It does not own concrete OpenBao or Kubernetes versions. Those remain in .ci/versions.yaml, and lanes reference the relevant fields through versionRefs.

The release workflow calls only the aggregate preview gate targets:

make test-e2e-release-preview-openbao
make test-e2e-release-preview-kind

Those targets prepare the required local images once, then the release-gate runner reads releaseGate.preview.groups from the suite manifest and executes the listed lanes directly. The per-lane makeTarget values remain supported operator/developer entrypoints, but release evidence must come from the manifest-defined groups, run selectors, timeouts, and environment. Ginkgo label lanes constrain Go’s test entrypoint to TestE2E; plain testing lanes must be selected explicitly with manifest runRegex.

The Kind aggregate also reads validation.kubernetes.previewMatrix from .ci/versions.yaml. By default it runs the Kind lane group once for every matrix entry with releaseGate: true. Set E2E_KUBERNETES_LINE=1.34 or E2E_KUBERNETES_LINE=1.35 to run one validated line locally. Kubernetes 1.36 is tracked as the intended next validation line until a digest-pinned Kind node image is available.

For focused release-gate diagnosis after the required images are available, run a single lane through the same runner:

E2E_PROVIDER_IMAGE=ghcr.io/dc-tec/bao-kms-provider:e2e-local \
  go run ./hack/tools/e2e_release_gate -group openbao -lane openbao-ha-ci

Soak lanes are release evidence for the pinned CI environment only. They are not an SLO, capacity, or production performance claim.

The SPIRE certificate-source lane is not part of the CI or preview release gate. It remains an explicit implementation check only. It does not make auth.cert.source: spiffe a supported user configuration until the supported OpenBao version can derive cert-auth identity aliases from URI SANs and release evidence covers that login path.

Current lane IDs:

Manifest validation:

make verify-e2e-manifest

The validation checks schema shape, required fields, lane IDs, status values, release-gate references, release workflow wiring, Make target availability, and the absence of floating latest references.

Layout

test/e2e/
  e2e_suite_test.go
  kind_dr_test.go
  kind_smoke_test.go
  openbao_transit_test.go
  openbao_cert_auth_test.go
  provider_cli_test.go
  provider_container_test.go
  provider_failure_test.go
  provider_ha_test.go
  provider_load_test.go
  provider_restore_test.go
  provider_rotation_test.go
  provider_upgrade_test.go
  suites.yaml
  suites_manifest_test.go
  kmsclient/
    main.go
  framework/
    artifacts.go
    env.go
    labels.go
    openbao_environment.go
    openbao_ha_environment.go

The root test/e2e package is the primary suite package. New Ginkgo E2E specs are added to this package unless there is a concrete reason to split binaries. Container-only helpers, such as the KMS v2 socket client, live under test/e2e.

Shared helpers live under test/e2e/framework. Helpers stay small and explicit; broad harness abstractions wait until multiple specs need them.

Environment

Individual provider lanes build their required image by default. The preview release aggregate targets prebuild the required images once, then run the manifest-defined lanes with E2E_PROVIDER_BUILD=false so the same image tags are reused across the release gate. Use E2E_PROVIDER_BUILD=false directly only when the referenced images are already built and available to the selected runtime.

CI builds the default provider image and PKCS#11 validation image once in the e2e-provider-images job, uploads Docker archives, and has the image scan, OpenBao E2E, and Kind E2E jobs load those exact images with E2E_PROVIDER_BUILD=false. The release workflow pulls the digest produced by the release build job, tags it locally as E2E_PROVIDER_IMAGE, and runs the preview gate against that immutable build output. OpenBao release validation still builds local PKCS#11 and upgrade/rollback images because those are validation-only variants, not public release images.

Reports And Artifacts

Preview release gates write per-lane artifacts under artifacts/e2e/<group>/<lane-id>/ for OpenBao lanes and artifacts/e2e/kind/<kubernetes-line>/<lane-id>/ for Kind lanes. Every lane writes console.log. Ginkgo-spec lanes also write junit.xml and ginkgo.json; plain testing lanes are selected through manifest runRegex and keep their evidence in console.log. The shared E2E_JUNIT_REPORT and E2E_JSON_REPORT variables apply to the generic make test-e2e entrypoint.

E2E logs are redacted. The framework never writes OpenBao tokens, JWTs, plaintext, or full ciphertext to artifacts.

Parallelism

Default parallelism is one node. Increase E2E_PARALLEL_NODES only for lanes whose environments are isolated per process. Shared socket paths and single Kind clusters remain single-node until the environment code proves isolation.