Skip to content

Metrics

The agent reads host metrics from /proc, per-workload metrics from each workload’s cgroup (or its processes), and forwards metrics your apps send over OTLP. They are stored in VictoriaMetrics (default) or Mimir and shown in Falak and Grafana.

View Shows
Server page → Metrics CPU, memory, disk, load, network and disk I/O for one server
Service panel → Metrics What the site itself uses, against its server and its limits; requests, 5xx and p95 latency (1h / 6h / 24h / 7d)
Database instance → Metrics Engine metrics and the instance’s CPU and memory: see Database observability
Servers list Current load, memory and disk percentages
Grafana Server, Workloads, Laravel site, Node app, Queues and Deployments dashboards

Server metrics: CPU, memory, load and network charts for one server.

Since v0.14.0 a site’s Metrics tab shows the site’s own usage, not the whole server’s.

Chart Content
CPU (cores) The site stacked on the rest of the server, with a line for the enforced CPU limit and one for the server’s cores. Near the limit, the share of time the site was throttled.
Memory Used memory stacked on the rest of the server; page cache (reclaimable) as a separate line; the memory limit and the server’s memory; OOM kills marked
Disk I/O Read and write
Network Receive and transmit (container sites; native sites show the server’s, labelled so)
Processes Process count, against the pids limit when one is set
Per service Compose sites: each service’s CPU and memory against its own limit

How a workload is measured:

Workload Measured from
Docker site, compose service, database container The container’s cgroup (cgroup v2): CPU time and throttling, memory, OOM kills, I/O, processes and limits. Network from the container’s own namespace.
Native site with resource limits Its systemd slice (falak-site_<slug>.slice)
Isolated native site without limits The processes of its own system user
Native site sharing the falak user Its processes, assigned one by one: PHP-FPM workers by pool name; workers, Horizon, schedules and consoles by the site folder they run in (v0.14.1)

SSH sessions, other sites’ deploy hooks and processes inside containers count for no site. Sites on several servers get a server picker (default: all servers, summed).

Requests per second, 5xx and p95 latency come from the app’s traces when it sends them, and from the edge’s access logs otherwise. Falak’s own health checks are left out. The chart says which source it uses: from the app or from the edge. The site’s HTTP tab goes further: routes, clients, status codes and bandwidth.

Metrics follow OpenTelemetry semantic conventions, exported to Prometheus naming:

Metric Meaning
system_cpu_utilization_ratio CPU utilization (0–1)
system_cpu_time_seconds_total CPU time by cpu_mode
system_cpu_load_average_1m, _5m, _15m Load averages
system_memory_usage_bytes Memory by system_memory_state
system_memory_utilization_ratio Memory utilization
system_filesystem_utilization_ratio Disk utilization by system_filesystem_mountpoint
system_network_io_bytes_total Network bytes by network_io_direction
system_disk_io_bytes_total Disk bytes by disk_io_direction

Filter by host_name.

PromQL examples
avg by (host_name) (system_cpu_utilization_ratio)
max by (host_name, system_filesystem_mountpoint) (system_filesystem_utilization_ratio) > 0.85

Each series carries service_name (the site slug, or falak-db with falak_db_instance_id for databases), falak_workload_kind (container, slice, user or processes), falak_workload_name and, for compose, falak_compose_service.

Metric Meaning
falak_workload_cpu_time_seconds_total CPU time; rate() gives cores used
falak_workload_cpu_limit CPU limit in cores (only while a limit is set)
falak_workload_cpu_throttled_time_seconds_total, falak_workload_cpu_throttled_periods_total, falak_workload_cpu_periods_total CPU throttling
falak_workload_memory_usage_bytes Memory by falak_memory_type (used, cache)
falak_workload_memory_limit_bytes Memory limit
falak_workload_memory_oom_kills_total OOM kills
falak_workload_disk_io_bytes_total Disk bytes by disk_io_direction
falak_workload_network_io_bytes_total Network bytes by network_io_direction (containers)
falak_workload_pids, falak_workload_pids_limit Processes and their limit
PromQL: cores used by one site
sum(rate(falak_workload_cpu_time_seconds_total{service_name="shop"}[5m]))

The falak.container.* metrics of earlier versions are gone; the Grafana dashboard is now Falak / Workloads.

Setting Default
Metrics interval 15 s (organization setting, 5–3600 s)
Backend victoriametrics (FALAK_METRICS_BACKEND), or mimir with FALAK_MIMIR_TENANT
Query URL FALAK_METRICS_QUERY_URL (the installer sets http://gateway:9090/prometheus)
Retention 30 days (FALAK_METRICS_RETENTION)

Deployments add a Grafana annotation and a deployment.id resource attribute, so you can see metric changes around each deploy.