Skip to content

Watch after deploy

A deployment already rolls back when a step fails or the health check before the switch fails (see Rollbacks). Since v0.10.0 a site can also watch each release for a few minutes after it went live, and roll back to the previous release when it turns bad. It is opt-in per site, and suggested for production services.

  1. Open the site’s Settings → Deploy → Watch after deploy.

  2. Turn it on and set the window and the triggers:

    Setting Default
    Watch for (minutes) 5 (1–60)
    Health check on: fails N times in a row (health_failures, default 3, 1–20) through the edge
    Crashes on: an OOM kill or a restart loop of the site, its Compose services, workers or daemons
    Errors on: the release’s 5xx share above max(3 × baseline, 5 %)
    New Insights issues off: a new exception issue during the window
    When triggered Roll back or Alert only
  3. Save. Changes apply from the next deployment.

After a successful deployment — not the site’s first, not a rollback, not a function — a window opens. The deployment view shows a countdown and each trigger’s status, and the canvas card shows Watching.

Every 30 seconds Falak:

  • runs the health check through the edge on each server (failures must be at least 25 s apart; skipped while the site’s health check is off);
  • compares the release’s 5xx share in the edge access log with max(3 × baseline, 5 %), once it has served at least 20 requests and 5 errors. The baseline is the previous release’s last hour (with at least 20 requests); without one, the 5 % rule applies.

OOM kills and restart loops, and (if on) new exception issues, trip the window at once.

The first trigger queues a rollback deployment to the previous release. The deployment is marked rolled_back with the reason and the numbers, the card shows Rolled back, and deployments.rolled_back fires once the rollback is live. With Alert only, deployments.watch_triggered fires instead.

  • Never back to a release that was itself rolled back automatically.
  • At most one automatic rollback per site per hour.
  • Never while another deployment of the site is queued or running.

A rollback the guard holds back fires deployments.watch_triggered instead. A window closes early (stopped) when another deployment of the site starts or goes live.

GET|PUT /api/v1/sites/{site}/release-watch, and the watch object on each deployment. See Deployments API.