AI agent files: a local disk, saved to object storage every 60 seconds
What a crash can lose, how results are accepted, and the start and save times we measured.
Plori keeps each AI agent's files on a local disk next to its microVM and saves read-only versions to object storage with restic. This issue covers how saves work, what a crash can lose, and what we measured on staging.
The short answer
- Each run has a local disk. The agent reads and writes an ordinary ext4 filesystem, with no FUSE or network filesystem on the write path.
- Saves happen every 60 seconds during a run and again at the end. A crash can lose the work done since the last finished save.
- Results are accepted by compare-and-swap. If another result moved the Workspace's pointer first, the accept fails.
Start and save times on staging
| Case | Start p50 | Save p50 |
|---|---|---|
| Same server, 100 MB tree, no changes | 229 ms | 2.40 s |
| Other server, 100 MB tree | 541 ms | 2.60 s |
| 56,000 small files, nothing on the server | 1.85 s | 7.66 s |
| 56,000 small files, file list built ahead | 830 ms | 7.73 s |
These are staging measurements on one shared 16-core development host that also ran builds and tests. The rows come from different rounds and builds with small samples (9 to 24 runs per row). They are not a matched comparison, and we have none with JuiceFS.
When it applies
- Large new trees: our target is 10 s from the end of a run to a remote save. In three staging rounds, 14 of 18 first saves of a fresh 56,000-file tree met it. The slowest took 14.05 s.
- Cold start on another server: reading a whole 56,000-file tree took 9 min 58 s in one staging round, because the agent's VM fetched each file on its first read.
- Recovery: it rebuilds a lost copy from its last ready revision. It does not restore running processes or open files.
The full post covers reflink copies, why the server kernel never mounts an agent's disk, and what to test if you build similar storage.