443 dead FUSE mounts took pod boots from 1 s to 20 to 105 s
How dead mounts wedged kubelet on one node, the three cleanup traps, and a 60-second janitor.
On July 23, 2026, 443 dead FUSE mounts on one node slowed every pod boot. This post describes Plori in July 2026, when agents ran in Kubernetes pods with FUSE-mounted disks. Agents now run in Firecracker microVMs, and that storage plane is retired.
The short answer
- One dead mount is harmless, and 443 wedged kubelet. The mounts built up over four weeks with no single triggering event, and cold pod boots went from about a second to 20 to 105 seconds.
- Kubelet cannot clean them up. Each dead mount answers every access with
ENOTCONN, so volume teardown fails and retries about every 2 minutes, forever. - Lazy-detaching them fixed the node within minutes. The k3s-agent process dropped from 0.83 to 0.18 cores in about five minutes.
Three traps while cleaning up
| Trap | Rule |
|---|---|
umount in kubectl debug node returns zero and changes nothing on the host |
Enter the host mount namespace with nsenter -t 1 -m |
Shell globs skip dead mounts, because lstat returns ENOTCONN |
Enumerate from /proc/mounts |
Mounts stack, so one umount reveals a second dead record |
Unmount in a loop until the path leaves /proc/mounts |
When it applies
- No agent data was at risk. The data lives in the storage plane, and the dead mount is a host-local problem. There was no user impact.
- Reap only on
ENOTCONN. A dead endpoint holds no unflushed state, so lazy detach is safe. A mount that hangs but is alive is left alone. - Automate the cleanup. A one-off run lets the pile rebuild, so our CSI node driver now sweeps every 60 seconds and once at startup.
The full post has the cleanup commands, the second bug behind the same alert, a checklist for your own nodes and four common questions.