Skip to content

Alerts and channels

Falak raises an alert when something needs attention. Rules decide which alerts go to which channels. Every alert also lands in the in-app notification center (the bell). Since v0.10.0 every organization starts with a default rule pack, routed to its default channel.

  1. Open Settings → Alert channels → Add (alerting.manage, owners and admins).
  2. Pick the type and fill in its settings.
  3. Save, then send a test.
Setting Rule
Recipients 1–20 e-mail addresses

Needs working mail on the control plane (MAIL_*). See Configuration.

Settings → Alert channels: configured channels with their type and status.

The first channel an organization adds is its default channel; Make default on the channels page picks another. New organizations are prompted to add one. Until there is a default channel, the default rule pack delivers in-app only (the rules page says so). Organizations that already had several channels before v0.10.0 get no default: the rules page asks you to choose one.

Every organization gets one editable rule per area whose alerts reach warning — servers, certificates, databases and backups, point-in-time recovery, volumes, deployments, limits, secrets, security, disaster recovery, network, HTTP and uptime — matching the area’s alert types from warning up and routed to the default channel.

  • Edit a pack rule like any rule; rules you edited are never re-routed.
  • Switch one off or delete it: deleted rules and removed patterns stay deleted.
  • The pack is applied when an organization is created, by the upgrade for existing ones, and daily, so an area added later joins every organization.

Settings → Alert rules → New rule:

Field Rule
Name Up to 100 characters
Event types 1–50 patterns: an exact type (deployments.failed), a prefix wildcard (deployments.*) or * for everything
Minimum severity info, warning or critical
Channels Up to 20 channels
Rate limit Optional: at most N deliveries per hour (1–1000)
Quiet hours Optional start/end (HH:MM), timezone, days (1–7), and whether critical alerts still go through

Settings → Alert rules: rules with their event types, minimum severity and channels.

Many alerts carry a suggested fix — a labelled link to where it is fixed, such as Grow volume, Fix in baseline, Review backups, Inspect certificate or Update agent. Alerts that track a condition (disk, memory, PITR lag, …) are raised once and resolved when it clears, and the recovery goes to the channels that got the alert. Third-party channels get the title and a neutral message; details (who, from where, raw errors) stay in Falak’s history.

Area Type Label Severity
Deployments deployments.failed Deployment failed critical
deployments.rolled_back Site rolled back warning
deployments.watch_triggered Release unhealthy after going live (not rolled back) critical
Builds builds.failed Build failed warning
Fleet fleet.agent_offline Server agent offline critical
fleet.agent_online Server agent back online info
fleet.agent_revoked Server agent revoked warning
fleet.agent_upgrade_failed / fleet.agent_upgraded Agent upgrade failed / upgraded after a failure warning / info
Servers servers.provisioned Server provisioned info
servers.disk_usage Disk almost full (80 % / 90 %, held 5 minutes) warning
servers.disk_forecast Disk fills within 48 hours (fitted over 6 hours) warning
servers.memory_high Memory above 90 % for 10 minutes warning
servers.cpu_high CPU above 90 % for 15 minutes warning
servers.load_high Load above twice the CPUs for 15 minutes warning
servers.reboot_required Server needs a reboot warning
servers.agent_outdated Agent outdated for an hour warning
Processes processes.crash_loop / processes.recovered Process keeps crashing / running again critical / info
Limits limits.oom_killed Service killed for running out of memory critical
limits.restart_loop Service keeps restarting critical
Network network.firewall_failed / network.firewall_recovered Firewall apply failed / applied again critical / info
network.conntrack_full Conntrack table at 85 % for 5 minutes warning
network.exposed_port A port became reachable from the internet warning
network.wireguard_stale WireGuard peer without a handshake for 5 minutes warning
network.latency_spike Latency between servers far above its baseline warning
network.block_suspended An IP block was lifted because it now covers a protected address warning
network.dns_failures DNS failures spike (eBPF) warning
network.tcp_retransmits TCP retransmits spike (eBPF) warning
HTTP http.errors_spike 5xx spike on a site warning
http.slow_endpoint A route is slow at the edge warning
http.bandwidth Monthly bandwidth threshold reached warning
http.scanner_surge Scanner surge on a site warning
Uptime uptime.monitor_down / uptime.monitor_recovered Monitor down / recovered critical / info
uptime.tls_problem Certificate problem on a monitored host warning
uptime.slow Monitor slower than its threshold warning
uptime.checks_delayed Uptime checks running late warning
Edge edge.certificate_failed Certificate install failed critical
edge.certificate_expiring Certificate expires in 14, 7 or 1 days warning
edge.certificate_installed Certificate installed info
Databases databases.backup_failed / databases.backup_recovered Database backup failed / succeeds again critical / info
databases.backup_missed No database backup within twice the schedule interval critical
databases.storage_unreachable Backup storage unreachable critical
databases.connections_high Connections above 80 % of the limit for 5 minutes warning
databases.slow_queries_spike 30 or more slow queries a minute for 5 minutes warning
databases.replication_lag Replication lag of 60 s or more for 5 minutes warning
databases.deadlocks Deadlocks in the last minute warning
databases.memory_high Near maxmemory or the memory limit for 5 minutes warning
databases.restore_failed / databases.restore_succeeded Database restore failed / finished critical / info
databases.drill_failed / databases.drill_recovered Restore drill failed / passes again critical / info
PITR pitr.lag Unshipped log older than 5 minutes warning
pitr.gap, pitr.spool_full, pitr.base_failed, pitr.stopped Gap in the timeline, spool above 20 %, base backup failed, shipping stopped critical
pitr.recovered Point-in-time recovery is fine again info
Volumes volumes.almost_full Volume above 85 % of its size warning
volumes.backup_failed / volumes.backup_succeeded Volume backup failed / succeeds again critical / info
volumes.backup_missed No volume backup within twice the schedule interval critical
volumes.drill_failed / volumes.drill_recovered Volume drill failed / passes again critical / info
Secrets secrets.provider_unreachable / secrets.provider_recovered Secret provider unreachable / reachable again warning / info
secrets.linked_changed Linked secret changed upstream info
secrets.rotation_due Secret due for rotation warning
secrets.unusual_reveals More than 20 reveals by one user in 10 minutes warning
Security security.score_dropped Security score dropped by 10 points or more warning
security.critical_finding New high or critical security finding critical
security.unexpected_port Unexpected public port warning
Disaster recovery dr.not_configured Control plane disaster recovery not configured (weekly) warning
dr.backup_failed, dr.backup_missing, dr.drill_failed Control plane backup failed, no recent backup, restore drill failed critical
Insights insights.issue_opened New issue warning
insights.issue_regressed Issue regressed warning
insights.issue_resolved Issue resolved info
insights.threshold_breached Performance threshold breached warning
insights.heartbeat_missed Scheduled task missed critical
  • Observability → Alerts (or Alert history) lists every alert and its deliveries.
  • The bell in the top bar shows in-app notifications for members with alerting.view.
  • History older than 90 days is pruned (FALAK_ALERTING_RETENTION_DAYS).
  • Delivery to real Slack, Discord and Telegram is covered by unit and feature tests, not yet by the end-to-end suite.
  • The webhook SSRF guard checks the host but does not resolve DNS, so DNS rebinding is not blocked.