Alerts and channels
Falak raises an alert when something needs attention. Rules decide which alerts go to which channels. Every alert also lands in the in-app notification center (the bell). Since v0.10.0 every organization starts with a default rule pack, routed to its default channel.
1. Create a channel
Section titled “1. Create a channel”- Open Settings → Alert channels → Add (
alerting.manage, owners and admins). - Pick the type and fill in its settings.
- Save, then send a test.
| Setting | Rule |
|---|---|
| Recipients | 1–20 e-mail addresses |
Needs working mail on the control plane (MAIL_*). See Configuration.
| Setting | Rule |
|---|---|
| Webhook URL | https://hooks.slack.com/services/…, /workflows/… or /triggers/… |
| Setting | Rule |
|---|---|
| Webhook URL | https://discord.com/api/webhooks/… (also discordapp.com, canary., ptb.) |
| Setting | Rule |
|---|---|
| Bot token | 123456:ABC… from @BotFather |
| Chat id | A numeric id (-100… for groups) or @channelname |
| Setting | Rule |
|---|---|
| URL | http(s)://…, public hosts only (see below) |
| Secret | 16–255 characters, used to sign deliveries |
Each delivery carries these headers:
| Header | Value |
|---|---|
X-Falak-Event |
The alert type, e.g. deployments.failed |
X-Falak-Delivery |
Unique delivery id |
X-Falak-Timestamp |
Unix timestamp |
X-Falak-Signature |
sha256= + HMAC-SHA256(secret, <timestamp>.<raw body>) |
$expected = 'sha256='.hash_hmac('sha256', $timestamp.'.'.$rawBody, $secret);abort_unless(hash_equals($expected, $request->header('X-Falak-Signature')), 401);Webhooks to loopback, private and link-local addresses are refused unless the operator sets FALAK_ALERTING_ALLOW_PRIVATE_WEBHOOKS=true.

2. The default channel
Section titled “2. The default channel”The first channel an organization adds is its default channel; Make default on the channels page picks another. New organizations are prompted to add one. Until there is a default channel, the default rule pack delivers in-app only (the rules page says so). Organizations that already had several channels before v0.10.0 get no default: the rules page asks you to choose one.
Default rule pack
Section titled “Default rule pack”Every organization gets one editable rule per area whose alerts reach warning — servers, certificates, databases and backups, point-in-time recovery, volumes, deployments, limits, secrets, security, disaster recovery, network, HTTP and uptime — matching the area’s alert types from warning up and routed to the default channel.
- Edit a pack rule like any rule; rules you edited are never re-routed.
- Switch one off or delete it: deleted rules and removed patterns stay deleted.
- The pack is applied when an organization is created, by the upgrade for existing ones, and daily, so an area added later joins every organization.
3. Create a rule
Section titled “3. Create a rule”Settings → Alert rules → New rule:
| Field | Rule |
|---|---|
| Name | Up to 100 characters |
| Event types | 1–50 patterns: an exact type (deployments.failed), a prefix wildcard (deployments.*) or * for everything |
| Minimum severity | info, warning or critical |
| Channels | Up to 20 channels |
| Rate limit | Optional: at most N deliveries per hour (1–1000) |
| Quiet hours | Optional start/end (HH:MM), timezone, days (1–7), and whether critical alerts still go through |

Alert types
Section titled “Alert types”Many alerts carry a suggested fix — a labelled link to where it is fixed, such as Grow volume, Fix in baseline, Review backups, Inspect certificate or Update agent. Alerts that track a condition (disk, memory, PITR lag, …) are raised once and resolved when it clears, and the recovery goes to the channels that got the alert. Third-party channels get the title and a neutral message; details (who, from where, raw errors) stay in Falak’s history.
| Area | Type | Label | Severity |
|---|---|---|---|
| Deployments | deployments.failed |
Deployment failed | critical |
deployments.rolled_back |
Site rolled back | warning | |
deployments.watch_triggered |
Release unhealthy after going live (not rolled back) | critical | |
| Builds | builds.failed |
Build failed | warning |
| Fleet | fleet.agent_offline |
Server agent offline | critical |
fleet.agent_online |
Server agent back online | info | |
fleet.agent_revoked |
Server agent revoked | warning | |
fleet.agent_upgrade_failed / fleet.agent_upgraded |
Agent upgrade failed / upgraded after a failure | warning / info | |
| Servers | servers.provisioned |
Server provisioned | info |
servers.disk_usage |
Disk almost full (80 % / 90 %, held 5 minutes) | warning | |
servers.disk_forecast |
Disk fills within 48 hours (fitted over 6 hours) | warning | |
servers.memory_high |
Memory above 90 % for 10 minutes | warning | |
servers.cpu_high |
CPU above 90 % for 15 minutes | warning | |
servers.load_high |
Load above twice the CPUs for 15 minutes | warning | |
servers.reboot_required |
Server needs a reboot | warning | |
servers.agent_outdated |
Agent outdated for an hour | warning | |
| Processes | processes.crash_loop / processes.recovered |
Process keeps crashing / running again | critical / info |
| Limits | limits.oom_killed |
Service killed for running out of memory | critical |
limits.restart_loop |
Service keeps restarting | critical | |
| Network | network.firewall_failed / network.firewall_recovered |
Firewall apply failed / applied again | critical / info |
network.conntrack_full |
Conntrack table at 85 % for 5 minutes | warning | |
network.exposed_port |
A port became reachable from the internet | warning | |
network.wireguard_stale |
WireGuard peer without a handshake for 5 minutes | warning | |
network.latency_spike |
Latency between servers far above its baseline | warning | |
network.block_suspended |
An IP block was lifted because it now covers a protected address | warning | |
network.dns_failures |
DNS failures spike (eBPF) | warning | |
network.tcp_retransmits |
TCP retransmits spike (eBPF) | warning | |
| HTTP | http.errors_spike |
5xx spike on a site | warning |
http.slow_endpoint |
A route is slow at the edge | warning | |
http.bandwidth |
Monthly bandwidth threshold reached | warning | |
http.scanner_surge |
Scanner surge on a site | warning | |
| Uptime | uptime.monitor_down / uptime.monitor_recovered |
Monitor down / recovered | critical / info |
uptime.tls_problem |
Certificate problem on a monitored host | warning | |
uptime.slow |
Monitor slower than its threshold | warning | |
uptime.checks_delayed |
Uptime checks running late | warning | |
| Edge | edge.certificate_failed |
Certificate install failed | critical |
edge.certificate_expiring |
Certificate expires in 14, 7 or 1 days | warning | |
edge.certificate_installed |
Certificate installed | info | |
| Databases | databases.backup_failed / databases.backup_recovered |
Database backup failed / succeeds again | critical / info |
databases.backup_missed |
No database backup within twice the schedule interval | critical | |
databases.storage_unreachable |
Backup storage unreachable | critical | |
databases.connections_high |
Connections above 80 % of the limit for 5 minutes | warning | |
databases.slow_queries_spike |
30 or more slow queries a minute for 5 minutes | warning | |
databases.replication_lag |
Replication lag of 60 s or more for 5 minutes | warning | |
databases.deadlocks |
Deadlocks in the last minute | warning | |
databases.memory_high |
Near maxmemory or the memory limit for 5 minutes |
warning | |
databases.restore_failed / databases.restore_succeeded |
Database restore failed / finished | critical / info | |
databases.drill_failed / databases.drill_recovered |
Restore drill failed / passes again | critical / info | |
| PITR | pitr.lag |
Unshipped log older than 5 minutes | warning |
pitr.gap, pitr.spool_full, pitr.base_failed, pitr.stopped |
Gap in the timeline, spool above 20 %, base backup failed, shipping stopped | critical | |
pitr.recovered |
Point-in-time recovery is fine again | info | |
| Volumes | volumes.almost_full |
Volume above 85 % of its size | warning |
volumes.backup_failed / volumes.backup_succeeded |
Volume backup failed / succeeds again | critical / info | |
volumes.backup_missed |
No volume backup within twice the schedule interval | critical | |
volumes.drill_failed / volumes.drill_recovered |
Volume drill failed / passes again | critical / info | |
| Secrets | secrets.provider_unreachable / secrets.provider_recovered |
Secret provider unreachable / reachable again | warning / info |
secrets.linked_changed |
Linked secret changed upstream | info | |
secrets.rotation_due |
Secret due for rotation | warning | |
secrets.unusual_reveals |
More than 20 reveals by one user in 10 minutes | warning | |
| Security | security.score_dropped |
Security score dropped by 10 points or more | warning |
security.critical_finding |
New high or critical security finding | critical | |
security.unexpected_port |
Unexpected public port | warning | |
| Disaster recovery | dr.not_configured |
Control plane disaster recovery not configured (weekly) | warning |
dr.backup_failed, dr.backup_missing, dr.drill_failed |
Control plane backup failed, no recent backup, restore drill failed | critical | |
| Insights | insights.issue_opened |
New issue | warning |
insights.issue_regressed |
Issue regressed | warning | |
insights.issue_resolved |
Issue resolved | info | |
insights.threshold_breached |
Performance threshold breached | warning | |
insights.heartbeat_missed |
Scheduled task missed | critical |
History and notifications
Section titled “History and notifications”- Observability → Alerts (or Alert history) lists every alert and its deliveries.
- The bell in the top bar shows in-app notifications for members with
alerting.view. - History older than 90 days is pruned (
FALAK_ALERTING_RETENTION_DAYS).
Limits
Section titled “Limits”- Delivery to real Slack, Discord and Telegram is covered by unit and feature tests, not yet by the end-to-end suite.
- The webhook SSRF guard checks the host but does not resolve DNS, so DNS rebinding is not blocked.