Skip to content

Backups and restore drills

Falak backs up each database inside its container, then compresses and encrypts the backup on the server before it leaves it. The server uploads straight to object storage through a presigned URL, so servers never hold storage credentials. There is no unencrypted option. Schedules can also restore their latest backup into a throwaway container on a regular basis and check it (restore drills).

Engine What a backup holds
PostgreSQL pg_dump -Fc of one database
MySQL / MariaDB mysqldump / mariadb-dump of one database (single transaction, routines, triggers, events)
Redis / Valkey An RDB snapshot of the instance
  1. Open Settings → Backup storage → Add (permission databases.storage.manage, owners and admins).

  2. Pick the driver and fill in the bucket and credentials:

    Driver Value Region hint
    Amazon S3 s3 e.g. eu-central-1
    Cloudflare R2 r2 auto
    Backblaze B2 b2 e.g. us-west-004
    DigitalOcean Spaces spaces e.g. fra1
    MinIO / S3-compatible minio us-east-1 unless configured otherwise; endpoint required (https)
    Field Rule
    Name Unique in the organization
    Bucket 3–63 characters, lowercase
    Prefix Optional path prefix
    Endpoint https:// URL; required for MinIO
    Path-style For S3-compatible services that need it
  3. Save. Falak verifies access and stores the credentials encrypted.

Settings → Backup storage: storage providers with their driver, bucket and status.

  1. Open the database panel → Backups → New schedule.

  2. Fill in:

    Field Rule
    Name Up to 100 characters
    Storage provider A storage provider
    Databases One or more databases of this container
    Schedule (cron) A cron expression, for example 0 3 * * *
    Keep last N / Max age (days) Retention; the newest successful backup is always kept
    Backup keys Held by Falak (default) or Customer-held (age) — see Keys
    Restore drills Off, Weekly or Monthly; the default is weekly in production environments, off elsewhere — see Restore drills
    Drill server This server, or another server of the organization to run drills on
    Check query Optional SELECT a drill runs on the restored copy
  3. Save. Old backups beyond the retention are pruned from storage.

Database panel → Backups: schedules and the backup history with status, size and actions.

Back up now runs a one-off backup of a single database from its row in Databases & users or Backups. Manual backups always use keys held by Falak.

Every backup is compressed with zstd, then encrypted with AES-256-GCM in 64 KiB segments, so a backup of any size streams through the agent in constant memory, and a file that was changed, reordered or cut short fails to decrypt. Each backup has its own random 32-byte data key. Objects end in .sql.zst.fkb or .rdb.zst.fkb.

The backup records the SHA-256 of the stored file and of the plaintext; both are checked on every restore.

Who holds a backup’s data key is chosen per schedule:

Held by Falak (default) Customer-held (age)
Generated by the control plane the agent, on the server
Stored on the backup, sealed under the organization’s data key (and so under the KEK) in the file’s header, encrypted with age to your public key
Reaches the server inside the command, in memory only, forgotten once the command settled never leaves the server
Restore automatic asks for your age private key, or offline with falak-restore
Restore drills yes no (Falak can’t decrypt them; drills are skipped)

Create a customer key pair with age-keygen -o falak-backups.key, keep that file offline, and paste the # public key: age1… line into the schedule. A customer-held schedule never falls back to Falak’s keys: without a valid public key its backups fail.

  1. Open Backups, pick a successful backup and choose Restore.
  2. Choose the target: an existing database of a running container of the same engine family. Type its name to confirm. For a customer-held backup, paste your age private key (AGE-SECRET-KEY-1…); it is sent once and never stored.
  3. The agent downloads the file, checks its SHA-256, decrypts the whole file to verify it, and only then restores it.
  • PostgreSQL restores into a scratch database that replaces the target only when the restore succeeded.
  • MySQL / MariaDB restore into the target directly (they have no RENAME DATABASE). With the verification pass, only an error of the engine itself can leave a partial restore; restore into a new database to keep the current one untouched.
  • Redis / Valkey replace dump.rdb while the engine is stopped.

Restoring needs databases.restore (owners and admins).

falak-restore is a small static binary published with each release (falak-restore-{linux,darwin}-{amd64,arm64}, listed in SHA256SUMS). It decrypts a backup offline.

  1. Download the object: Download in the backup list, or from your bucket.

  2. Get the key: for a backup held by Falak, Export backup key on the backup (admins; asks for your password, and your code with two-factor authentication; recorded in the audit log). For a customer-held backup, use your age identity.

  3. Decrypt:

    Terminal window
    falak-restore --info backup.sql.zst.fkb # key mode and key id
    falak-restore --key-file falak-backup-<id>.key --out dump backup.sql.zst.fkb # keys held by Falak
    falak-restore --identity falak-backups.key --out dump backup.sql.zst.fkb # customer-held keys
  4. Load the plaintext with the engine’s tools: pg_restore -d <db> dump (PostgreSQL), mysql <db> < dump (MySQL / MariaDB), or copy it as dump.rdb into a stopped Redis / Valkey.

Exit codes: 0 verified, 1 wrong key or a corrupt or truncated file, 2 usage. With --out, nothing is left behind unless the whole file verified.

A drill restores the latest backup into a throwaway container and checks it:

  • the backup downloads, its SHA-256 matches, it decrypts and restores;
  • SQL: at least one table; Redis / Valkey: at least one key;
  • the row counts of the 10 largest tables (Redis / Valkey: the key count) are within ±10 % of the counts recorded just before the backup (FALAK_DRILL_TOLERANCE_PERCENT);
  • the optional check query returns at least one row. It must be a single SELECT without ;, comments or \, and runs read-only as a temporary role that can only read, with a 60 s limit.

The drill container uses the instance’s image digest, at most 512 MiB of memory (FALAK_DRILL_MEMORY_BYTES), one CPU, no network and no published port, and is removed with its data whatever happened. Drills run on their schedule, or now with Drill now.

Each drill records passed, failed or skipped, its checks, duration and an RTO estimate (download plus restore time). A passed drill marks the backup verified; a failed one fires databases.drill_failed. A drill is skipped when there is no backup yet, the keys are customer-held, the agent is offline, or the server lacks memory or disk — pick a drill server for small servers. Choosing a drill server takes the restore permission, because it receives the restored data.

Setting Value
Backup / restore command timeout 3600 s
Drill timeout 2 h
Upload URL lifetime 12 h (FALAK_BACKUP_UPLOAD_URL_TTL)
Download URL lifetime 6 h (FALAK_BACKUP_DOWNLOAD_URL_TTL)
Alert type Severity
databases.backup_failed critical
databases.backup_missed critical (no backup within twice the schedule’s interval)
databases.backup_recovered info
databases.storage_unreachable critical
databases.restore_failed / databases.restore_succeeded critical / info
databases.drill_failed / databases.drill_recovered critical / info

They are part of the default rule pack.

  • Backups taken before encryption (v0.10.0-rc.1 and earlier) keep their rows but can no longer be restored from Falak.
  • There is no local-disk storage driver; use an S3-compatible bucket (MinIO works).
  • Deleting an organization deletes its data keys: its backups held by Falak can no longer be opened. Export the keys you want to keep first.