Skip to content

Disaster recovery

Two things can be lost: the control plane host (the panel, its database, the Fleet CA, the keys) and an application server (the sites, databases and volumes on it). This page covers both. Until disaster recovery is set up, owners and admins see a banner in the panel, a checklist item on the Projects page, and a weekly dr.not_configured alert; falak-ctl doctor warns.

On the control plane host, as root:

Terminal window
sudo falak-ctl dr setup

It asks for (or takes as flags, see falak-ctl dr setup --help):

  • an S3-compatible bucket: endpoint (https://s3.eu-central-1.amazonaws.com, https://<account>.r2.cloudflarestorage.com, …), bucket, region, key prefix (default falak), access key and secret key. Secrets come from files (--secret-key-file, --passphrase-file) or the terminal, never the command line;
  • a DR passphrase (12+ characters). Every uploaded backup is encrypted and authenticated with it;
  • the schedule: every 6 hours by default (--every 1|2|3|4|6|8|12|24), and a monthly restore drill (--drill monthly|off). --include-registry also backs up the built-in image registry (it can be large).

It tests the bucket, stores the settings in /opt/falak/dr/dr.env (root only, never mounted into a container, never in a backup), and installs two systemd timers:

Timer Runs
falak-backup.timer falak-ctl backup --upload --scheduled every N hours
falak-drill.timer falak-ctl dr drill --scheduled on the 1st of each month

Then take a first backup and prove it:

Terminal window
sudo falak-ctl backup --upload
sudo falak-ctl dr drill
sudo falak-ctl dr status

Settings → Disaster recovery shows the last backup, its age and size, the last drill and the commands; the configuration itself stays on the host.

Use a bucket for the control plane alone, ideally with another provider or in another region than the control plane:

  • versioning on, or better object lock (a retention of at least a month), so a stolen key or a compromised host can neither delete nor overwrite the backups you need;
  • a lifecycle rule expiring old objects (and old versions) after your retention;
  • keys that can PutObject, GetObject and ListBucket on the prefix, nothing else.

The operator organization sees the control plane’s DR. install.sh records it as FALAK_DR_ORGANIZATION in .env (the first admin’s organization). An install with one organization needs nothing; on an install with several and no setting, nobody sees it: set FALAK_DR_ORGANIZATION=<organization id> in .env and run falak-ctl up.

falak-ctl backup writes /opt/falak/backups/falak-backup-<UTC time>[-label].fdr when the DR passphrase is set (or .tar.gz without one):

Part Why
db.dump The database: servers, sites, users, the Fleet CA key, every secret (sealed under the KEK)
falak-ca volume The Fleet CA certificate: agents trust only this CA
edge-pki volume The edge’s client certificate for the agent API
app-storage, caddy-data volumes Build artifacts and app files; ACME account and certificates
registry-data volume Only with --include-registry
env, custom.env APP_KEY, passwords, every setting
secrets/kek The key-encryption key, only in an encrypted backup, wrapped by the DR passphrase
manifest Version, image digests, volumes, KEK id, row counts

.fdr files are encrypted (AES-256) and then authenticated (HMAC-SHA256) with keys derived from the passphrase. A restore checks the MAC before anything is decrypted, and refuses a renamed or replayed file. An unencrypted backup leaves the KEK out, never leaves the host (--upload refuses it), and says so.

Keep, off the host and apart from the backups:

  1. the DR passphrase;
  2. the bucket’s endpoint, name, region, prefix and a key that can read it;
  3. the KEK on its own: falak-ctl kek export <file> (needed only for unencrypted backups; export it again after every falak-ctl kek rotate).

With 1 and 2, a lost control plane comes back from the latest upload. Nothing in the bucket can be decrypted without 1.

  1. Get a new VPS (Ubuntu 22.04, 24.04 or 26.04, or Debian 12; 2+ GB RAM).

  2. Point DNS at it: the panel domain, agents.<domain>, registry.<domain> (and grafana.<domain>) — the same names as before. Agents keep calling agents.<domain>. With --skip-dns-check you can install first and move DNS after.

  3. Install with --restore-from, same domain and e-mail:

    Terminal window
    export FALAK_BACKUP_S3_ENDPOINT=https://s3.eu-central-1.amazonaws.com FALAK_BACKUP_S3_BUCKET=acme-falak-dr \
    FALAK_BACKUP_S3_REGION=eu-central-1 FALAK_BACKUP_S3_ACCESS_KEY=AKIA…
    curl -fsSL https://falak.sh/install.sh | sudo -E bash -s -- \
    --domain falak.example.com --email [email protected] --restore-from s3://latest

    The secret key and the DR passphrase are asked on the terminal without echo. s3://falak-backup-<time>-scheduled restores a named backup instead. Install the same release as before or a newer one (--version): migrations bring an older dump forward, never back.

  4. Check: falak-ctl doctor, falak-ctl dr status, then the panel. Servers come back online on their own within a minute or two: their certificates are signed by the restored Fleet CA, so nothing is re-enrolled.

  5. Writes after the backup are gone: deployments, settings and new secrets since then. Redeploy what changed.

The installer runs falak-ctl restore s3://… --yes (download, check, decrypt; the backup’s .env, KEK, database, Fleet CA, edge PKI and volumes replace the fresh install’s), and only then falak-ctl dr setup with the same bucket and passphrase, so the schedule resumes.

On an existing install (a standby host): falak-ctl restore s3://latest --yes. With a backup file: falak-ctl restore <file> --yes. An unencrypted backup has no KEK: import it first with falak-ctl kek import <kit> --force.

Servers → the lost server → This server is gone… (owners and admins).

  1. Pick a replacement: another active server of the organization. For a new machine, create it first (through a provider or with the install command), wait until it is active, then come back.

  2. Read the dry run: every database container with each database’s latest restorable backup and the data loss (time since that backup; “all data” when there is none), every volume the same way, the services that ran there, and which domains Falak moves (Cloudflare zones it manages) and which need a DNS change by hand. Nothing changes yet.

  3. Confirm by typing the lost server’s name. The steps run in order, and the page follows them live:

    Step What happens
    Replacement server Waits until it is active
    Move services Each site, Compose stack and function gets the replacement instead of the lost server; edge routes and Falak-managed DNS records follow
    Restore databases Each container is recreated on the replacement with the same name, DNS name, password, settings, limits, users and schedules, then restored from its latest backup
    Restore volumes Each volume’s latest backup goes into a volume of the same name; its attachments move to it
    Redeploy services Every deployed service is redeployed onto the restored data
    Domains Falak-managed records are checked; the others are listed with the replacement’s address
  4. A failed item stops the recovery at its step. Fix the cause and Retry the step: only what failed runs again.

  5. Afterwards, delete the lost server in Falak, and the machine at its provider if it still exists.

  • A database container with point-in-time recovery on (and keys held by Falak) comes back at the latest point of its shipped log; its data loss in the dry run is the PITR lag.
  • Databases and volumes whose backups use your own age key aren’t restored automatically: the wizard marks them “Your turn”; restore them from their page.
  • The lost server must really be gone. The wizard refuses while its agent reported within the last 10 minutes (FALAK_RECOVERY_LOST_AFTER_MINUTES). When the recovery starts, the lost server’s agent is revoked, so a machine that comes back can’t reconnect.

Disaster recovery readiness (⌘K → “Disaster recovery readiness”) scores each project: every database backed up on a schedule and proven by a restore drill, point-in-time recovery on for production SQL databases, every volume backed up on a schedule. Each gap links to where it is fixed.

  • Control plane: falak-ctl dr drill (monthly from the timer) downloads the latest upload (--local for the newest local backup), restores it into a throwaway Compose project (falak-drill, ports 18080/18443, its own volumes and network; no edge, workers or scheduler, so nothing talks to your servers), checks that migrations run, that the KEK unwraps every data key, the row counts and the Fleet CA, then removes it. It needs about 1.5 GB of free memory.
  • Databases and volumes: restore drills per backup schedule.
  • Once a year, for real: restore the control plane onto a scratch VPS with --restore-from (firewalled off, so real agents keep talking to the real control plane), and run “This server is gone” for a staging server.
Type When
dr.not_configured Weekly until dr setup ran
dr.backup_failed A scheduled backup failed
dr.backup_missing No backup for twice the schedule; cleared by the next backup
dr.drill_failed A control plane restore drill failed
Action Permission Roles
See the control plane’s DR status and prompt (operator organization) recovery.control_plane owner, admin
Recover a lost server recovery.servers owner, admin
View readiness recovery.view owner, admin, developer, viewer