# Disaster recovery

> Recover from losing the Falak control plane or a server: falak-ctl dr setup, restore on a fresh host, drills, the emergency kit and readiness.

Source: https://falak.sh/docs/operations/disaster-recovery/

Two things can be lost: the **control plane host** (the panel, its database, the Fleet CA, the keys) and an **application server** (the sites, databases and volumes on it). This page covers both. Until disaster recovery is set up, owners and admins see a banner in the panel, a checklist item on the Projects page, and a weekly `dr.not_configured` alert; `falak-ctl doctor` warns.

## 1. Set it up

On the control plane host, as root:

```bash
sudo falak-ctl dr setup
```

It asks for (or takes as flags, see `falak-ctl dr setup --help`):

- an **S3-compatible bucket**: endpoint (`https://s3.eu-central-1.amazonaws.com`, `https://<account>.r2.cloudflarestorage.com`, …), bucket, region, key prefix (default `falak`), access key and secret key. Secrets come from files (`--secret-key-file`, `--passphrase-file`) or the terminal, never the command line;
- a **DR passphrase** (12+ characters). Every uploaded backup is encrypted and authenticated with it;
- the schedule: every 6 hours by default (`--every 1|2|3|4|6|8|12|24`), and a monthly restore drill (`--drill monthly|off`). `--include-registry` also backs up the built-in image registry (it can be large).

It tests the bucket, stores the settings in `/opt/falak/dr/dr.env` (root only, never mounted into a container, never in a backup), and installs two systemd timers:

| Timer | Runs |
|---|---|
| `falak-backup.timer` | `falak-ctl backup --upload --scheduled` every N hours |
| `falak-drill.timer` | `falak-ctl dr drill --scheduled` on the 1st of each month |

Then take a first backup and prove it:

```bash
sudo falak-ctl backup --upload
sudo falak-ctl dr drill
sudo falak-ctl dr status
```

**Settings → Disaster recovery** shows the last backup, its age and size, the last drill and the commands; the configuration itself stays on the host.

### The bucket

Use a bucket for the control plane alone, ideally with another provider or in another region than the control plane:

- **versioning** on, or better **object lock** (a retention of at least a month), so a stolen key or a compromised host can neither delete nor overwrite the backups you need;
- a **lifecycle rule** expiring old objects (and old versions) after your retention;
- keys that can `PutObject`, `GetObject` and `ListBucket` on the prefix, nothing else.

### Several organizations

The **operator organization** sees the control plane's DR. `install.sh` records it as `FALAK_DR_ORGANIZATION` in `.env` (the first admin's organization). An install with one organization needs nothing; on an install with several and no setting, nobody sees it: set `FALAK_DR_ORGANIZATION=<organization id>` in `.env` and run `falak-ctl up`.

## 2. What a backup holds

`falak-ctl backup` writes `/opt/falak/backups/falak-backup-[-label].fdr` when the DR passphrase is set (or `.tar.gz` without one):

| Part | Why |
|---|---|
| `db.dump` | The database: servers, sites, users, the Fleet CA key, every secret (sealed under the KEK) |
| `falak-ca` volume | The Fleet CA certificate: agents trust only this CA |
| `edge-pki` volume | The edge's client certificate for the agent API |
| `app-storage`, `caddy-data` volumes | Build artifacts and app files; ACME account and certificates |
| `registry-data` volume | Only with `--include-registry` |
| `env`, `custom.env` | `APP_KEY`, passwords, every setting |
| `secrets/kek` | The key-encryption key, **only in an encrypted backup**, wrapped by the DR passphrase |
| `manifest` | Version, image digests, volumes, KEK id, row counts |

`.fdr` files are encrypted (AES-256) and then authenticated (HMAC-SHA256) with keys derived from the passphrase. A restore checks the MAC before anything is decrypted, and refuses a renamed or replayed file. An unencrypted backup leaves the KEK out, never leaves the host (`--upload` refuses it), and says so.

## 3. The emergency kit

Keep, **off the host and apart from the backups**:

1. the DR passphrase;
2. the bucket's endpoint, name, region, prefix and a key that can read it;
3. the KEK on its own: `falak-ctl kek export <file>` (needed only for unencrypted backups; export it again after every `falak-ctl kek rotate`).

With 1 and 2, a lost control plane comes back from the latest upload. Nothing in the bucket can be decrypted without 1.

## 4. The control plane host is lost

1. **Get a new VPS** (Ubuntu 22.04, 24.04 or 26.04, or Debian 12; 2+ GB RAM).
2. **Point DNS at it**: the panel domain, `agents.<domain>`, `registry.<domain>` (and `grafana.<domain>`) — the same names as before. Agents keep calling `agents.<domain>`. With `--skip-dns-check` you can install first and move DNS after.
3. **Install with `--restore-from`**, same domain and e-mail:

   ```bash
   export FALAK_BACKUP_S3_ENDPOINT=https://s3.eu-central-1.amazonaws.com FALAK_BACKUP_S3_BUCKET=acme-falak-dr \
          FALAK_BACKUP_S3_REGION=eu-central-1 FALAK_BACKUP_S3_ACCESS_KEY=AKIA…
   curl -fsSL https://falak.sh/install.sh | sudo -E bash -s -- \
     --domain falak.example.com --email you@example.com --restore-from s3://latest
   ```

   The secret key and the DR passphrase are asked on the terminal without echo. `s3://falak-backup-<time>-scheduled` restores a named backup instead. Install the same release as before or a newer one (`--version`): migrations bring an older dump forward, never back.

4. **Check**: `falak-ctl doctor`, `falak-ctl dr status`, then the panel. Servers come back online on their own within a minute or two: their certificates are signed by the restored Fleet CA, so nothing is re-enrolled.
5. Writes after the backup are gone: deployments, settings and new secrets since then. Redeploy what changed.

The installer runs `falak-ctl restore s3://… --yes` (download, check, decrypt; the backup's `.env`, KEK, database, Fleet CA, edge PKI and volumes replace the fresh install's), and only then `falak-ctl dr setup` with the same bucket and passphrase, so the schedule resumes.

On an existing install (a standby host): `falak-ctl restore s3://latest --yes`. With a backup file: `falak-ctl restore <file> --yes`. An unencrypted backup has no KEK: import it first with `falak-ctl kek import <kit> --force`.

## 5. An application server is lost

Servers → the lost server → **This server is gone…** (owners and admins).

1. **Pick a replacement**: another active server of the organization. For a new machine, create it first (through a provider or with the install command), wait until it is active, then come back.
2. **Read the dry run**: every database container with each database's latest restorable backup and the **data loss** (time since that backup; "all data" when there is none), every volume the same way, the services that ran there, and which domains Falak moves (Cloudflare zones it manages) and which need a DNS change by hand. Nothing changes yet.
3. **Confirm** by typing the lost server's name. The steps run in order, and the page follows them live:

   | Step | What happens |
   |---|---|
   | Replacement server | Waits until it is active |
   | Move services | Each site, Compose stack and function gets the replacement instead of the lost server; edge routes and Falak-managed DNS records follow |
   | Restore databases | Each container is recreated on the replacement with the same name, DNS name, password, settings, limits, users and schedules, then restored from its latest backup |
   | Restore volumes | Each volume's latest backup goes into a volume of the same name; its attachments move to it |
   | Redeploy services | Every deployed service is redeployed onto the restored data |
   | Domains | Falak-managed records are checked; the others are listed with the replacement's address |

4. A failed item stops the recovery at its step. Fix the cause and **Retry** the step: only what failed runs again.
5. Afterwards, delete the lost server in Falak, and the machine at its provider if it still exists.

- A database container with [point-in-time recovery](/docs/databases/point-in-time-recovery/) on (and keys held by Falak) comes back at the **latest point of its shipped log**; its data loss in the dry run is the PITR lag.
- Databases and volumes whose backups use **your own age key** aren't restored automatically: the wizard marks them "Your turn"; restore them from their page.
- **The lost server must really be gone.** The wizard refuses while its agent reported within the last 10 minutes (`FALAK_RECOVERY_LOST_AFTER_MINUTES`). When the recovery starts, the lost server's agent is **revoked**, so a machine that comes back can't reconnect.

## 6. Readiness

**Disaster recovery readiness** (<kbd>⌘K</kbd> → "Disaster recovery readiness") scores each project: every database backed up on a schedule and proven by a restore drill, point-in-time recovery on for production SQL databases, every volume backed up on a schedule. Each gap links to where it is fixed.

## 7. Drills

- **Control plane:** `falak-ctl dr drill` (monthly from the timer) downloads the latest upload (`--local` for the newest local backup), restores it into a throwaway Compose project (`falak-drill`, ports 18080/18443, its own volumes and network; no edge, workers or scheduler, so nothing talks to your servers), checks that migrations run, that the KEK unwraps every data key, the row counts and the Fleet CA, then removes it. It needs about 1.5 GB of free memory.
- **Databases and volumes:** [restore drills](/docs/databases/backups/#restore-drills) per backup schedule.
- **Once a year, for real:** restore the control plane onto a scratch VPS with `--restore-from` (firewalled off, so real agents keep talking to the real control plane), and run "This server is gone" for a staging server.

## Alerts

| Type | When |
|---|---|
| `dr.not_configured` | Weekly until `dr setup` ran |
| `dr.backup_failed` | A scheduled backup failed |
| `dr.backup_missing` | No backup for twice the schedule; cleared by the next backup |
| `dr.drill_failed` | A control plane restore drill failed |

## Permissions

| Action | Permission | Roles |
|---|---|---|
| See the control plane's DR status and prompt (operator organization) | `recovery.control_plane` | owner, admin |
| Recover a lost server | `recovery.servers` | owner, admin |
| View readiness | `recovery.view` | owner, admin, developer, viewer |

## Next steps
