# Troubleshooting

> Fixes for Falak problems by symptom — installation, connecting servers, builds, deployments, domains and TLS, processes, observability and the panel.

Source: https://falak.sh/docs/reference/troubleshooting/

Start with the two diagnostic commands on the control plane host:

```bash
sudo falak-ctl status
sudo falak-ctl doctor
```

## Installation

| Symptom | Cause and fix |
|---|---|
| `DNS does not point at this host` | Create the printed A/AAAA records and wait for propagation. Behind Cloudflare, use **DNS only**. |
| `port 80 is in use` | Another web server runs: `systemctl disable --now nginx apache2 caddy`. |
| Pulling images fails | The release is not published or the GHCR packages are private (`docker login ghcr.io`). |
| `FALAK_EDGE_SUBNET … overlaps` | Choose another private /24 in `.env` and re-run the installer. |
| Certificate error in the browser | `falak-ctl logs edge`. Ports 80/443 must be reachable from the internet; Let's Encrypt rate-limits repeated reinstalls. |
| Stack not healthy | `falak-ctl logs control-plane` (migration errors show there). |

## Connecting servers

| Symptom | Cause and fix |
|---|---|
| Install command fails to enroll | The server must reach `https://<panel>` (public CA) and `https://agents.<panel>` (Fleet CA). The agent API must answer `401` without a client certificate (`falak-ctl doctor` → Agent API). |
| Token expired / already used | Regenerate the install command on the server page (valid 24 h, single use). |
| Stuck in `provisioning` | Read the output on the server page; transient downloads are retried. Re-provision after fixing. |
| Locked out of SSH after provisioning | Password logins are disabled. Add an SSH key, or use the web terminal. |
| Agent **offline** | `systemctl status falak-agent`, `journalctl -u falak-agent`. Check outbound HTTPS to `agents.<panel>`. |
| Server stays **Waiting for agent** after you deleted it and ran a new install command on the same machine | From v0.5.2 the install command replaces the old identity by itself. With an older Falak, move the old identity aside first: see [Reconnect a machine](/docs/servers/agent/#reconnect-a-machine). |
| Install command ends with `falak-agent is installed but not connected` | It prints the reason from `falak-agent check`: revoked identity, `agents.<panel>` unreachable, TLS error or clock skew. Run `sudo falak-agent check` again; logs: `journalctl -u falak-agent`. |
| `this machine's clock is …s off the panel's` | Turn on time sync: `sudo timedatectl set-ntp true`, then run the install command again. |
| Provisioning step `apt`, `caddy` or `php:<version>` fails on `apt-get update` | The error names the repository and its file under `/etc/apt/sources.list.d`. Fix or remove that file, then **Re-provision**. A `ppa:ondrej/php` source with no release for the server's Ubuntu (for example 26.04) is disabled automatically (renamed to `<file>.disabled-by-falak`). |
| `PHP 8.4 is not available on Ubuntu 26.04; installing PHP 8.5 instead` | The PHP PPA has no packages for Ubuntu 26.04 yet, so PHP comes from Ubuntu's archive, which has only PHP 8.5. Use Ubuntu 24.04 if you need another version. |

## Machine check

A server shows **Needs attention** when the [machine check](/docs/servers/machine-check/) found software Falak won't change on its own; nothing was installed. Fix what the server page's **Machine check** panel lists, then **Re-check** and **Provision**.

| Message | Fix |
|---|---|
| `Port 80/443/2019 is in use by nginx` (Apache, …) | Another web server holds the edge's ports: `sudo systemctl disable --now nginx`, or move it to other ports. |
| `caddy.service is running` | Move the sites it serves into Falak, then `sudo systemctl disable --now caddy`. |
| `A container (…) publishes port 5432/3306/6379/80` | `docker stop <name>` or publish it on another port. To keep the database in Docker, deploy it as a compose service instead of choosing the engine for the server. |
| `MariaDB … is installed, but this server is set up for MySQL` (or Redis ↔ Valkey) | Falak won't run two engines of a kind on one machine. Remove the other one, or set the server up with the engine that is installed (it is then used as is). |
| `… is older than …, the oldest Falak supports` | Upgrade it from the same source to at least Docker 20.10, PostgreSQL 14, MySQL 8.0, MariaDB 10.6, Redis 6.0 or Valkey 7.2. |
| `Docker from Docker's repository has no compose (buildx), and that repository is not configured` | Add Docker's apt repository, or install `docker-compose-plugin` / `docker-buildx-plugin`. Falak never mixes Ubuntu's Docker packages with Docker's: they overwrite each other's files. |
| `docker.service is masked` / `Only the Docker CLI is installed` | `sudo systemctl unmask docker.service docker.socket`, or install the engine from the CLI's source, or remove the CLI so Falak installs Ubuntu's Docker. |
| `Docker is installed as a snap` / `Only a rootless Docker is set up` / `podman-docker provides the docker command` | Falak needs the system Docker daemon from apt: `sudo snap remove docker` (or `apt purge podman-docker`); Falak then installs Docker or keeps one you install from Docker's repository. |
| `Password login would be turned off, but no user who may log in over SSH has a key` | Add your public key to `~/.ssh/authorized_keys` of a sudo user sshd lets in. Root's keys count only when root may log in; `AllowUsers` / `DenyUsers` / `AllowGroups` / `DenyGroups` apply. |
| Re-provision says `Re-provisioning stopped. Machine check: …` | The server keeps running as it is and nothing was applied. Fix the listed conflicts, then Re-provision again. |
| Warnings (provisioning goes on) | ufw or firewalld active: allow ports in both (`sudo ufw allow 80,443/tcp`). An earlier `sshd_config.d` file wins over Falak's `50-falak.conf`. `"iptables": false` in `daemon.json` breaks published ports. |
| Step `adopt:<component>` fails: `… is no longer installed` | Something the check found was removed since. Re-provision: the check runs again first. |

## Builds

| Symptom | Cause and fix |
|---|---|
| Build stays **queued** | No eligible builder. `falak-ctl logs builder`. Docker builds need a builder with Docker (a `builder` server). |
| `railpack detected …, which native builds do not support` | Use the Docker build mode. |
| Docker build fails at push (`lookup registry.falak.local … no such host`, `401`, `x509`) | No built-in registry yet, or its DNS is missing: `falak-ctl up` (adds `FALAK_REGISTRY_*`), create the `registry.<domain>` record, then `falak-ctl registry status`. `x509` with `--tls internal`: Docker doesn't trust the registry's certificate. |
| `Builder <name> restarted during the build.` | The builder restarted (for example during an update). Redeploy. |
| Frontend variables undefined | Build-time variables need a public prefix (`VITE_`, `NEXT_PUBLIC_`, …) or **Expose to deploy script**. |
| Lockfile or install errors | Commit the lockfile, or set `FALAK_INSTALL_COMMAND`. |

## Deployments

| Symptom | Cause and fix |
|---|---|
| **Waiting for servers** | Servers are still being prepared (site user, PHP-FPM pool, Bun/Deno). It starts by itself; it fails after 30 minutes. |
| `Unresolved variable references: …` | A `${{ service.KEY }}` points at an unknown service or key, or a cycle. Fix the variable. |
| Health check failed | Open **View logs** and the site's **Logs**. Check the health path under **Settings → Deploy** (`/up` for Laravel, `/` for Node). |
| `The site has no app port for its container.` | Set an app port on the Docker site. |
| Migrations ran on every server | Move them inside `if [ "$FALAK_IS_LEADER" = "1" ]`. |
| Old code still served after deploy | A custom deploy script without `$FALAK_ACTIVATE`/`$FALAK_RESTART_PROCS` in the right place. |
| `The agent restarted before running the command` | The agent restarted during a deploy step. Redeploy. |
| `network <stack>_default does not exist: deploy the compose stack it belongs to first` | A Compose service split into its own Docker site needs its stack's networks. Deploy the stack, then redeploy the site. See [Deploy order](/docs/guides/compose-apps/#deploy-order-for-split-out-services). |
| `<service> now runs as its own Falak site, which hasn't been deployed yet` | Deploy the split-out site first, then the stack. |

## Domains and TLS

| Symptom | Cause and fix |
|---|---|
| DNS check `proxied` | Set the Cloudflare record to **DNS only** until the certificate is issued. |
| DNS check `mismatch` | Remove extra records; point only at the listed targets. |
| Certificate stays pending | DNS must point at the server; 80/443 open in the cloud firewall. |
| Generated domain `422` | Generated names are off for the organization, or the server has no public IPv4. |

## Processes and apps

| Symptom | Cause and fix |
|---|---|
| Workers or cron never start | Programs start only after the site's first successful deployment on that server. |
| **Process keeps crashing** alert | The program is fatal or restarting repeatedly. Read its output in the site's Logs. |
| 502 on a Node/Bun/Deno site | The app does not listen on `PORT`, has no `start` script, or crashed. |
| Laravel can't write `storage/logs` | Redeploy so writable directories get their group ACLs. |
| Database connection refused | On a `db` server, add a firewall rule for the app server. |
| `… cannot be used here: … accepts connections from that server only` | The database runs on an app or worker server, which only sites and containers on that same server can reach. Move it to a dedicated database server (type `db`). |
| `… containers on <server> can't reach its databases yet` | Update the server's agent to 0.4.5 or newer; container access turns on once the agent reports it. |

## Observability

| Symptom | Cause and fix |
|---|---|
| Logs tab empty / API `503` | Observability is not enabled or Loki is down. |
| Network Logs empty after an upgrade from ≤ v0.2.5 | `falak-ctl reload-configs`. |
| Laravel request logs missing | `LOG_CHANNEL=stderr`; use `daily` and redeploy. |
| No traces | Install `falak/apm-laravel` or `@falak/apm-node`; enable observability. |

## Panel

| Symptom | Cause and fix |
|---|---|
| Pages take seconds | `falak-ctl doctor` → PHP threads. Raise `FALAK_PHP_MAX_THREADS` (panel) or `FALAK_AGENT_API_THREADS` (agents). |
| Live updates don't refresh | `reverb` must be healthy; browsers connect to `wss://<panel>/app/…`. |
| Something breaks only in worker mode | `FALAK_WORKER_MODE=0`, `falak-ctl up`, and report it. |
| Lost admin password | `falak-ctl admin reset-password you@example.com`. |
| Agents offline after a restore | The `.env`/`APP_KEY` must come from the same backup as the database. |
| Low memory | Lower `FALAK_HORIZON_MAX_PROCESSES`, or move observability to another host. |

Still stuck? Collect `falak-ctl doctor` output, the failed deployment's output and the relevant `falak-ctl logs <service>` and open an issue at [github.com/OthmanHaba/falak](https://github.com/OthmanHaba/falak/issues). Remove secrets first.