Use override restore only when the cluster is locked after a failed upgrade.

This runbook exists for the narrow case where the normal restore path is blocked by an existing cluster operation lock, usually after a failed rollback or another crashed automation loop. It is a break-glass restore path, not the default way to restore a cluster.

Use this runbook only when

• the cluster is stuck behind an operation lock after a failed upgrade or rollback
• a normal OpenBaoRestore request is blocked by that lock
• you accept that the current cluster state will be overwritten by the snapshot
• you already identified the last known good snapshot you are willing to restore

This bypasses the existing operation lock

This path ignores the current lock owner, overwrites cluster state with the selected snapshot, and is meant for disaster recovery under operator supervision. Do not use it when the normal restore workflow is still available.

Decision matrix

Know what this override changes

Know what this override changes.

Field	What it does	Why it matters
force: true	Allows restore to proceed on an unhealthy cluster.	You are explicitly acknowledging that normal safety checks are being relaxed for recovery.
overrideOperationLock: true	Clears the existing cluster operation lock so restore can proceed.	This is what makes the workflow break-glass instead of routine restore.
spec.breakGlassAck on OpenBaoCluster	May still be required later if the cluster remains in break-glass mode after restore.	The restore override is separate from the cluster break-glass acknowledgment flow.

Create the break-glass restore request

Configure

Force restore past an existing operation lock

yaml

apiVersion: openbao.org/v1alpha1
kind: OpenBaoRestore
metadata:
name: emergency-restore-001
namespace: security
spec:
cluster: prod-cluster
source:
  target:
    provider: s3
    endpoint: https://s3.amazonaws.com
    bucket: openbao-backups
    region: us-east-1
    credentialsSecretRef:
      name: s3-credentials
  key: clusters/prod/last-good-snapshot.snap
jwtAuthRole: openbao-operator-restore
force: true
overrideOperationLock: true

force is required when restore targets an unhealthy cluster. overrideOperationLock is what bypasses the stuck upgrade or backup lock. Keep them together only for this break-glass path.

Apply

Apply the override restore

bash

kubectl apply -f emergency-restore.yaml

Verify the restore and plan the follow-up

Verify

Inspect restore and cluster state

bash

kubectl get openbaorestore <name> -n <namespace> -o yaml
kubectl describe openbaocluster <cluster> -n <namespace>
kubectl get jobs -n <namespace>

A completed restore only means the restore workflow finished. The target cluster may still require unseal, Raft repair, or break-glass acknowledgment before it is truly operational again.

Finish the recovery

Recover a sealed clusterUse this when the restored workload starts but cannot unseal cleanly.Recover from no leaderUse this when the restored snapshot leaves the cluster needing Raft peer or quorum repair.Enter safe modeGo here when the operator still requires explicit break-glass acknowledgment after the restore completes.