Troubleshooting
Last updated
First steps for any issue
jabali repair --diagnose, covers most known states./jabali-admin/server-status, visual at-a-glance.journalctl -u jabali-panel -u jabali-agent -fin a separate terminal while you reproduce the problem.
Install-time issues
CrowdSec fetch fails (DNS)
Symptom: installer hangs at “Installing CrowdSec” or Could not resolve host: packagecloud.io.
Cause: outbound :53/udp is blocked, so the recursor can’t recurse.
Fix:
JABALI_DNS_FORWARDER=<reachable-resolver> bash install.sh
The installer then masks systemd-resolved, writes /etc/resolv.conf with options use-vc timeout:5 attempts:2, strips the resolve [!UNAVAIL=return] NSS shim, and forwards . via TCP.
pdns-recursor crash-loops with source value 18446744073709551615 is larger than 65535
Cause (now fixed): the install template had forward-zones-recurse=.=${DNS_FORWARDER:-1.1.1.1;9.9.9.9} inside a quoted heredoc; the variable wasn’t interpolated, recursor parsed the literal string, and the YAML loader cast a length-of-string overflow into the uint16 port field.
Fix: jabali update to pick up the post-fix install template.
Installer fails at “render-config” for AppSec
Cause: install_crowdsec_appsec ran pre-build, so render-config had no binary. PR#92 gates this; PR#93 re-runs render-config post-build.
Fix: jabali update.
Vite OOM on small VM
Symptom: npm run build exits 137 during panel-ui build on a 1 GB RAM VPS.
Cause: V8 heap default outsizes the VM.
Fix (already in install.sh ≥ PR #69): NODE_OPTIONS=--max-old-space-size=<50%-of-RAM> + 1 GB swap auto-created on exit 137 fallback.
Runtime issues
Dirty database version N
Cause: a migration ran partially, then failed.
Fix:
jabali migrate up # idempotent retry
If it still errors, look at the journal:
journalctl -u jabali-panel -n 200 | grep -A 20 'migration'
Common root cause: a previous merge dropped a migration file but later migrations ALTER tables expecting it. The “audit migration numbering after merge” rule prevents this in CI; if you hit it, file an issue with the migration sequence (ls panel-api/internal/db/migrations).
Webmail won’t log in
Causes, in rough order of likelihood:
- The mailbox’s password changed in the panel but the user is using the old one, reset under Mail → Mailboxes.
- Stalwart is down,
systemctl status stalwart-mail. - The Roundcube SSO bridge clock-skewed (SSO file TTL is 60 s): sync NTP.
- AppArmor in
enforceblocked something,journalctl -k | grep DENIED.
SSL cert doesn’t issue
Check the row at /jabali-admin/ssl:
| Last error | Likely cause |
|---|---|
unauthorized | Domain doesn’t resolve to this server’s IP, fix DNS at registrar. |
rateLimited | LE rate limit hit. Wait. |
connection refused on :80 | Firewall blocks :80 from outside, or nginx isn’t listening on :80 for that vhost (it must, even when SSL is on, for HTTP-01). |
Panel cert: panel hostname not set | Server Settings → Panel Hostname not yet configured. |
Per-user PHP-FPM pool socket missing
Symptom: 502 Bad Gateway on a user’s vhost.
Fix:
jabali repair --diagnose # confirms which user is affected
systemctl restart 'php*-fpm.service'
Underlying cause is usually a stale drop-in after a package change. The reconciler regenerates on next tick.
Backup destination test fails
/jabali-admin/backups → Destinations → Test. Common causes:
- SFTP: wrong host key, wrong username, wrong key path on the panel host.
- S3: clock skew on the panel host > 15 minutes (AWS rejects SigV4).
- Restic REST: server has self-signed cert and
JABALI_RESTIC_INSECURE_TLSisn’t set.
Auto-init: the test action runs restic init if the repository doesn’t exist. Won’t auto-init on a destination that has files but isn’t a restic repo (refuses to overwrite).
Full teardown
There is no jabali --uninstall. Intended path: redeploy the VM.
If you must keep the host:
systemctl disable --now jabali-panel jabali-agent stalwart-mail kratos bulwark pdns pdns-recursor crowdsec
apt purge -y stalwart-mail crowdsec
trash /etc/jabali /var/lib/jabali /var/lib/jabali-* /etc/nginx/sites-*/jabali-* /etc/php/*/fpm/pool.d/jabali-* /etc/systemd/system/jabali-* /etc/letsencrypt
userdel -r jabali
# DB still has panel data; drop it manually.
Then reboot. The host won’t be Jabali-clean (CrowdSec data dirs, restic caches, etc. linger) but the panel won’t run.