Troubleshooting

Common failures of jailoc workspaces and how to fix them.

Find the logs

jailoc writes its own structured log to ~/.cache/jailoc/jailoc.log. It rotates at startup above 5 MB to jailoc.log.1; if the rename fails, the file is truncated instead. Without a home directory, the log goes to $TMPDIR/jailoc/jailoc.log. Look for level=ERROR lines.

To find out where a slow start spends its time, list the duration of each step: image builds, the import into the VM image cache, VM start and session start. VM boot steps also carry their details, such as microvm.image.from_cache. A step that failed is logged as vm step failed; an advisory problem that did not stop the boot, such as a failed non-required preflight check, as a level=WARN vm step finished line with an error.

grep -E 'step (finished|failed)' ~/.cache/jailoc/jailoc.log | tail -n 30

Each workspace VM keeps its state in ~/.cache/jailoc-vm/workspaces/<workspace>/:

Print the two logs with:

jailoc logs <workspace>
jailoc logs -f <workspace>   # follow

Inspect what runs inside the VM:

jailoc traffic -f            # connections and DNS lookups, live
jailoc strace -n 50          # the last 50 processes started or failed
jailoc shell                 # a shell as the agent user
jailoc exec -- ps aux        # one command

Config rejects a jailoc v1 setting

Symptom: any command fails with:

mode is a jailoc v1 setting; run `jailoc migrate` to update the config

The same error appears for backend = "docker", password_mode and expose_port.

Fix: run the migration, which rewrites these settings and carries over v1 Docker workspaces:

jailoc migrate --dry-run   # preview
jailoc migrate

See Migrate from jailoc v1.

VM runtime is missing

Symptom: jailoc up fails with:

this jailoc build does not include the VM runtime for <os>/<arch>; use a release build of jailoc or set runner and lib_dir in the [vm] section of the config

Fix: release builds embed the runtime for macOS (Apple Silicon) and Linux (amd64, arm64). On another platform, or with a binary built from source, set [vm] runner and [vm] lib_dir to a runtime you built yourself. See How to install jailoc.

VM can't open /dev/kvm

Symptom: on Linux, the VM fails to start and supervisor.log reports that /dev/kvm is missing or permission is denied.

Fix:

• If /dev/kvm doesn't exist, enable virtualization (VT-x/AMD-V) in the firmware and load the kvm_intel or kvm_amd module. Inside a cloud VM, nested virtualization must be enabled.
• If it exists but isn't accessible, add your user to its group and log in again:

ls -l /dev/kvm
sudo usermod -aG kvm "$USER"

VM process can't be confined

Symptom: on Linux, the VM fails to start with confining the VM process needs Landlock.

Fix: jailoc runs the VM process under Landlock and does not start a VM without it. Use Linux 5.19 or later and check that landlock appears in /sys/kernel/security/lsm; if it doesn't, add it to the lsm= kernel parameter and reboot. See How to install jailoc.

VM doesn't become ready

Symptom: jailoc up fails with vm not ready after ..., vm supervisor exited ..., VM NIC did not connect within ... or broker not ready after ....

Debugging:

jailoc logs <workspace>
tail -n 50 ~/.cache/jailoc-vm/workspaces/<workspace>/supervisor.log

Common causes:

• the hypervisor isn't available (see /dev/kvm on Linux);
• too little memory for the requested memory;
• a custom image without the UID 1000 agent user or the agent binary (see How to use a custom image);
• a stale VM left over from a crash.

Clear a stale VM with jailoc down <workspace> and start again.

Processes die or the VM stops when memory runs out

Symptom: builds, tests or a language server die without an error of their own, or jailoc logs reports vm stopped: workspace vm stopped: nic-disconnected while the VM was busy.

Debugging: the guest copies each kernel OOM kill to the console log:

jailoc logs <workspace> | grep 'out of memory'
jailoc strace -n 200 | grep SIGKILL   # processes that ended with SIGKILL

A line such as msg="out of memory: kernel killed a process" pid=271 process=gopls anon_rss_mib=1536 oom_score_adj=0 names the victim and its memory use. See Memory pressure for the order in which the kernel picks victims.

Fix: give the workspace more memory with memory in its config (see Configuration reference), or run fewer memory-hungry jobs at once, for example go test -p 1 or a lower GOMAXPROCS.

A mount fails at start

Symptom: jailoc up fails with mount "...": backend "vm" shares directories only; use snapshots for a read-only copy of a file.

Fix: move file entries from mounts to snapshots. See Snapshot files into the workspace.

Network rules block a required host

Symptom: the agent can't reach an internal service; connections are refused or time out.

Private and reserved addresses, IPv4 and IPv6 (including ULA fc00::/7), are blocked by default.

Fix: add the host or range, then reload:

[workspaces.myproject]
allowed_hosts = ["internal-registry.example.com"]
allowed_networks = ["10.10.5.0/24"]

jailoc reload myproject

jailoc traffic shows every denied connection with its reason. On macOS, local-network hosts also need the Local Network permission. See How to allow specific hosts or networks.

Image build fails

Symptom: jailoc up reports errors pulling or building the image.

The first matching image source wins, and a failure there is fatal. Common causes:

• a registry that isn't reachable or needs authentication;
• a URL Dockerfile returning an HTTP error;
• a local Dockerfile path that doesn't exist;
• [vm] builder = "docker" without a usable host Docker daemon;
• the shared builder VM pulling from a private registry it can't reach (use isolated_build = true with the host in allowed_hosts).

grep -Ei "image|pull|build|dockerfile" ~/.cache/jailoc/jailoc.log

See the Image resolution reference.

Docker inside the VM fails

Symptom: docker is not found in the guest, jailoc up fails with enable_docker is on, but the workspace image has no dockerd, or docker commands fail in the guest.

Fix: Docker is off by default; set enable_docker = true and run jailoc restart. An image used directly (image, or defaults.image without a workspace dockerfile) must contain Docker; install it there or set enable_docker = false. Images jailoc builds get Docker when enable_docker is on. Docker is also off when protect_git = true. See Use Docker inside the VM.

OpenCode fails to start with a server or database error

Symptom: jailoc exits before the TUI appears with Unexpected server error. Check server logs for details. or opencode exited with status 1.

OpenCode batches writes to ~/.local/share/opencode/log/opencode.log, so an immediate exit at startup may not flush the failure to the log file.

Debugging: run OpenCode with --print-logs inside the workspace VM to print unbuffered logs and the underlying error to standard error:

jailoc -w <workspace> exec -- opencode --print-logs
jailoc -w <workspace> exec -- opencode debug config --print-logs

Fix: if the output reports a SQLite schema error (such as no such column after downgrading agents.<name>.version or sharing ~/.local/share/opencode across incompatible OpenCode versions), upgrade the agent version to match the database schema, or stop any running sessions and move the database together with its WAL and SHM sidecars into a backup directory so OpenCode starts with an empty active database:

jailoc -w <workspace> exec -- sh -c 'bak=~/.local/share/opencode/db-backup-$(date +%Y%m%d-%H%M%S) && mkdir -p "$bak" && for f in opencode.db opencode.db-wal opencode.db-shm; do [ -e ~/.local/share/opencode/"$f" ] && mv ~/.local/share/opencode/"$f" "$bak"/; done'

Clean up

Stop a workspace and delete its persistent home, Docker data and isolated builder:

jailoc down <workspace> --purge

Remove the log file:

rm -f ~/.cache/jailoc/jailoc.log ~/.cache/jailoc/jailoc.log.1