Troubleshooting

Start with the two diagnostic commands on the control plane host:

sudo falak-ctl status
sudo falak-ctl doctor

Installation

Symptom	Cause and fix
DNS does not point at this host	Create the printed A/AAAA records and wait for propagation. Behind Cloudflare, use DNS only.
port 80 is in use	Another web server runs: systemctl disable --now nginx apache2 caddy.
Pulling images fails	The release is not published or the GHCR packages are private (docker login ghcr.io).
FALAK_EDGE_SUBNET … overlaps	Choose another private /24 in .env and re-run the installer.
Certificate error in the browser	falak-ctl logs edge. Ports 80/443 must be reachable from the internet; Let’s Encrypt rate-limits repeated reinstalls.
Stack not healthy	falak-ctl logs control-plane (migration errors show there).

Connecting servers

Symptom	Cause and fix
Install command fails to enroll	The server must reach https://<panel> (public CA) and https://agents.<panel> (Fleet CA). The agent API must answer 401 without a client certificate (falak-ctl doctor → Agent API).
Token expired / already used	Regenerate the install command on the server page (valid 24 h, single use).
Stuck in provisioning	Read the output on the server page; transient downloads are retried. Re-provision after fixing.
Locked out of SSH after provisioning	Password logins are disabled. Add an SSH key, or use the web terminal.
Agent offline	systemctl status falak-agent, journalctl -u falak-agent. Check outbound HTTPS to agents.<panel>.
Server stays Waiting for agent after you deleted it and ran a new install command on the same machine	From v0.5.2 the install command replaces the old identity by itself. With an older Falak, move the old identity aside first: see Reconnect a machine.
Install command ends with falak-agent is installed but not connected	It prints the reason from falak-agent check: revoked identity, agents.<panel> unreachable, TLS error or clock skew. Run sudo falak-agent check again; logs: journalctl -u falak-agent.
this machine's clock is …s off the panel's	Turn on time sync: sudo timedatectl set-ntp true, then run the install command again.
Provisioning step apt, caddy or php:<version> fails on apt-get update	The error names the repository and its file under /etc/apt/sources.list.d. Fix or remove that file, then Re-provision. A ppa:ondrej/php source with no release for the server’s Ubuntu (for example 26.04) is disabled automatically (renamed to <file>.disabled-by-falak).
PHP 8.4 is not available on Ubuntu 26.04; installing PHP 8.5 instead	The PHP PPA has no packages for Ubuntu 26.04 yet, so PHP comes from Ubuntu’s archive, which has only PHP 8.5. Use Ubuntu 24.04 if you need another version.

Machine check

A server shows Needs attention when the machine check found software Falak won’t change on its own; nothing was installed. Fix what the server page’s Machine check panel lists, then Re-check and Provision.

Message	Fix
Port 80/443/2019 is in use by nginx (Apache, …)	Another web server holds the edge’s ports: sudo systemctl disable --now nginx, or move it to other ports.
caddy.service is running	Move the sites it serves into Falak, then sudo systemctl disable --now caddy.
A container (…) publishes port 5432/3306/6379/80	docker stop <name> or publish it on another port. To keep the database in Docker, deploy it as a compose service instead of choosing the engine for the server.
MariaDB … is installed, but this server is set up for MySQL (or Redis ↔ Valkey)	Falak won’t run two engines of a kind on one machine. Remove the other one, or set the server up with the engine that is installed (it is then used as is).
… is older than …, the oldest Falak supports	Upgrade it from the same source to at least Docker 20.10, PostgreSQL 14, MySQL 8.0, MariaDB 10.6, Redis 6.0 or Valkey 7.2.
Docker from Docker's repository has no compose (buildx), and that repository is not configured	Add Docker’s apt repository, or install docker-compose-plugin / docker-buildx-plugin. Falak never mixes Ubuntu’s Docker packages with Docker’s: they overwrite each other’s files.
docker.service is masked / Only the Docker CLI is installed	sudo systemctl unmask docker.service docker.socket, or install the engine from the CLI’s source, or remove the CLI so Falak installs Ubuntu’s Docker.
Docker is installed as a snap / Only a rootless Docker is set up / podman-docker provides the docker command	Falak needs the system Docker daemon from apt: sudo snap remove docker (or apt purge podman-docker); Falak then installs Docker or keeps one you install from Docker’s repository.
Password login would be turned off, but no user who may log in over SSH has a key	Add your public key to ~/.ssh/authorized_keys of a sudo user sshd lets in. Root’s keys count only when root may log in; AllowUsers / DenyUsers / AllowGroups / DenyGroups apply.
Re-provision says Re-provisioning stopped. Machine check: …	The server keeps running as it is and nothing was applied. Fix the listed conflicts, then Re-provision again.
Warnings (provisioning goes on)	ufw or firewalld active: allow ports in both (sudo ufw allow 80,443/tcp). An earlier sshd_config.d file wins over Falak’s 50-falak.conf. "iptables": false in daemon.json breaks published ports.
Step adopt:<component> fails: … is no longer installed	Something the check found was removed since. Re-provision: the check runs again first.

Builds

Symptom	Cause and fix
Build stays queued	No eligible builder. falak-ctl logs builder. Docker builds need a builder with Docker (a builder server).
railpack detected …, which native builds do not support	Use the Docker build mode.
Docker build fails at push (lookup registry.falak.local … no such host, 401, x509)	No built-in registry yet, or its DNS is missing: falak-ctl up (adds FALAK_REGISTRY_*), create the registry.<domain> record, then falak-ctl registry status. x509 with --tls internal: Docker doesn’t trust the registry’s certificate.
Builder <name> restarted during the build.	The builder restarted (for example during an update). Redeploy.
Frontend variables undefined	Build-time variables need a public prefix (VITE_, NEXT_PUBLIC_, …) or Expose to deploy script.
Lockfile or install errors	Commit the lockfile, or set FALAK_INSTALL_COMMAND.

Deployments

Symptom	Cause and fix
Waiting for servers	Servers are still being prepared (site user, PHP-FPM pool, Bun/Deno). It starts by itself; it fails after 30 minutes.
Unresolved variable references: …	A ${{ service.KEY }} points at an unknown service or key, or a cycle. Fix the variable.
Health check failed	Open View logs and the site’s Logs. Check the health path under Settings → Deploy (/up for Laravel, / for Node).
The site has no app port for its container.	Set an app port on the Docker site.
Migrations ran on every server	Move them inside if [ "$FALAK_IS_LEADER" = "1" ].
Old code still served after deploy	A custom deploy script without $FALAK_ACTIVATE/$FALAK_RESTART_PROCS in the right place.
The agent restarted before running the command	The agent restarted during a deploy step. Redeploy.
network <stack>_default does not exist: deploy the compose stack it belongs to first	A Compose service split into its own Docker site needs its stack’s networks. Deploy the stack, then redeploy the site. See Deploy order.
<service> now runs as its own Falak site, which hasn't been deployed yet	Deploy the split-out site first, then the stack.

Domains and TLS

Symptom	Cause and fix
DNS check proxied	Set the Cloudflare record to DNS only until the certificate is issued.
DNS check mismatch	Remove extra records; point only at the listed targets.
Certificate stays pending	DNS must point at the server; 80/443 open in the cloud firewall.
Generated domain 422	Generated names are off for the organization, or the server has no public IPv4.

Processes and apps

Symptom	Cause and fix
Workers or cron never start	Programs start only after the site’s first successful deployment on that server.
Process keeps crashing alert	The program is fatal or restarting repeatedly. Read its output in the site’s Logs.
502 on a Node/Bun/Deno site	The app does not listen on PORT, has no start script, or crashed.
Laravel can’t write storage/logs	Redeploy so writable directories get their group ACLs.
Database connection refused	On a db server, add a firewall rule for the app server.
… cannot be used here: … accepts connections from that server only	The database runs on an app or worker server, which only sites and containers on that same server can reach. Move it to a dedicated database server (type db).
… containers on <server> can't reach its databases yet	Update the server’s agent to 0.4.5 or newer; container access turns on once the agent reports it.

Observability

Symptom	Cause and fix
Logs tab empty / API 503	Observability is not enabled or Loki is down.
Network Logs empty after an upgrade from ≤ v0.2.5	falak-ctl reload-configs.
Laravel request logs missing	LOG_CHANNEL=stderr; use daily and redeploy.
No traces	Install falak/apm-laravel or @falak/apm-node; enable observability.

Panel

Symptom	Cause and fix
Pages take seconds	falak-ctl doctor → PHP threads. Raise FALAK_PHP_MAX_THREADS (panel) or FALAK_AGENT_API_THREADS (agents).
Live updates don’t refresh	reverb must be healthy; browsers connect to wss://<panel>/app/….
Something breaks only in worker mode	FALAK_WORKER_MODE=0, falak-ctl up, and report it.
Lost admin password	falak-ctl admin reset-password you@example.com.
Agents offline after a restore	The .env/APP_KEY must come from the same backup as the database.
Low memory	Lower FALAK_HORIZON_MAX_PROCESSES, or move observability to another host.

Tip

Still stuck? Collect falak-ctl doctor output, the failed deployment’s output and the relevant falak-ctl logs <service> and open an issue at github.com/OthmanHaba/falak. Remove secrets first.