Operate / Recovery & Restore

Separate incident recovery from backup-driven restore.

This section exists for the moments when normal troubleshooting is no longer enough. Some failures need a focused recovery path such as safe mode or no-leader repair. Others are better handled by restoring known-good state from backup through an explicit OpenBaoRestore request.

Run a restoreEnter safe mode

Use this section when you need to

• decide whether the cluster still has a normal repair path
• move from incident diagnosis into a narrow recovery runbook
• restore state from backup without guessing at the destructive impact
• return the cluster to normal operations only after the status surface is clean again

Decision matrix

Choose the right path

Choose the right path.

Situation	Start here	Why
The cluster lost leadership, sealed unexpectedly, or entered break-glass mode after a failed change.	Incident recovery	You still want to repair the live cluster state rather than replace it.
The safest route is to reintroduce known-good state from object storage.	Restore from backup	Restore is explicit, destructive, and auditable instead of being hidden inside normal reconciliation.
You are still diagnosing the failure and do not know whether repair or restore is necessary.	Troubleshoot the cluster first	Do not jump into a destructive path before you know whether a narrower fix is still available.

Incident recovery

1. 01

   Enter safe mode

   Stabilize the cluster and stop risky automation when the operator needs explicit human acknowledgment.

   Open
2. 02

   Recover from no leader

   Repair quorum and leadership when Raft can no longer elect or keep a leader.

   Open
3. 03

   Recover a sealed cluster

   Diagnose trust, seal, network, or identity failures that leave Pods running but not serving.

   Open
4. 04

   Recover from failed rollback

   Use the dedicated upgrade-recovery path when blue-green rollback automation stops in break glass mode.

   Open

Restore from backup

1. 05

   Run a restore

   Apply an OpenBaoRestore request when you intentionally need to overwrite cluster state from a snapshot.

   Open
2. 06

   Restore overview

   Understand why restore is modeled as an explicit CRD-backed operation and what safety boundaries it enforces.

   Open
3. 07

   Recover after upgrade restore

   Use the override-lock path only when a failed upgrade or rollback blocks the normal restore workflow.

   Open

Next actions

Troubleshoot the clusterGo back to the symptom-driven incident guide when you still need to narrow the failure before choosing recovery.Check backup readinessGood recovery depends on the snapshot path already being healthy before the incident starts.Open status and eventsUse the reference material when you need exact condition and event semantics during triage.