Troubleshooting

Last updated

First steps for any issue

  1. jabali repair --diagnose, covers most known states.
  2. /jabali-admin/server-status, visual at-a-glance.
  3. journalctl -u jabali-panel -u jabali-agent -f in a separate terminal while you reproduce the problem.

Install-time issues

CrowdSec fetch fails (DNS)

Symptom: installer hangs at “Installing CrowdSec” or Could not resolve host: packagecloud.io.

Cause: outbound :53/udp is blocked, so the recursor can’t recurse.

Fix:

JABALI_DNS_FORWARDER=<reachable-resolver> bash install.sh

The installer then masks systemd-resolved, writes /etc/resolv.conf with options use-vc timeout:5 attempts:2, strips the resolve [!UNAVAIL=return] NSS shim, and forwards . via TCP.

pdns-recursor crash-loops with source value 18446744073709551615 is larger than 65535

Cause (now fixed): the install template had forward-zones-recurse=.=${DNS_FORWARDER:-1.1.1.1;9.9.9.9} inside a quoted heredoc; the variable wasn’t interpolated, recursor parsed the literal string, and the YAML loader cast a length-of-string overflow into the uint16 port field.

Fix: jabali update to pick up the post-fix install template.

Installer fails at “render-config” for AppSec

Cause: install_crowdsec_appsec ran pre-build, so render-config had no binary. PR#92 gates this; PR#93 re-runs render-config post-build.

Fix: jabali update.

Vite OOM on small VM

Symptom: npm run build exits 137 during panel-ui build on a 1 GB RAM VPS.

Cause: V8 heap default outsizes the VM.

Fix (already in install.sh ≥ PR #69): NODE_OPTIONS=--max-old-space-size=<50%-of-RAM> + 1 GB swap auto-created on exit 137 fallback.

Runtime issues

Dirty database version N

Cause: a migration ran partially, then failed.

Fix:

jabali migrate up        # idempotent retry

If it still errors, look at the journal:

journalctl -u jabali-panel -n 200 | grep -A 20 'migration'

Common root cause: a previous merge dropped a migration file but later migrations ALTER tables expecting it. The “audit migration numbering after merge” rule prevents this in CI; if you hit it, file an issue with the migration sequence (ls panel-api/internal/db/migrations).

Webmail won’t log in

Causes, in rough order of likelihood:

  1. The mailbox’s password changed in the panel but the user is using the old one, reset under Mail → Mailboxes.
  2. Stalwart is down, systemctl status stalwart-mail.
  3. The Roundcube SSO bridge clock-skewed (SSO file TTL is 60 s): sync NTP.
  4. AppArmor in enforce blocked something, journalctl -k | grep DENIED.

SSL cert doesn’t issue

Check the row at /jabali-admin/ssl:

Last errorLikely cause
unauthorizedDomain doesn’t resolve to this server’s IP, fix DNS at registrar.
rateLimitedLE rate limit hit. Wait.
connection refused on :80Firewall blocks :80 from outside, or nginx isn’t listening on :80 for that vhost (it must, even when SSL is on, for HTTP-01).
Panel cert: panel hostname not setServer Settings → Panel Hostname not yet configured.

Per-user PHP-FPM pool socket missing

Symptom: 502 Bad Gateway on a user’s vhost.

Fix:

jabali repair --diagnose       # confirms which user is affected
systemctl restart 'php*-fpm.service'

Underlying cause is usually a stale drop-in after a package change. The reconciler regenerates on next tick.

Backup destination test fails

/jabali-admin/backups → Destinations → Test. Common causes:

  • SFTP: wrong host key, wrong username, wrong key path on the panel host.
  • S3: clock skew on the panel host > 15 minutes (AWS rejects SigV4).
  • Restic REST: server has self-signed cert and JABALI_RESTIC_INSECURE_TLS isn’t set.

Auto-init: the test action runs restic init if the repository doesn’t exist. Won’t auto-init on a destination that has files but isn’t a restic repo (refuses to overwrite).

Full teardown

There is no jabali --uninstall. Intended path: redeploy the VM.

If you must keep the host:

systemctl disable --now jabali-panel jabali-agent stalwart-mail kratos bulwark pdns pdns-recursor crowdsec
apt purge -y stalwart-mail crowdsec
trash /etc/jabali /var/lib/jabali /var/lib/jabali-* /etc/nginx/sites-*/jabali-* /etc/php/*/fpm/pool.d/jabali-* /etc/systemd/system/jabali-* /etc/letsencrypt
userdel -r jabali
# DB still has panel data; drop it manually.

Then reboot. The host won’t be Jabali-clean (CrowdSec data dirs, restic caches, etc. linger) but the panel won’t run.