On July 3rd every devflow session — the agent workers that plan and run coding tasks for us — started dying at the same spot. The plan node would fire up its worker container and get nothing back. No output, no error, no event posted anywhere. Just a worker that started and vanished. This went on for about five hours.
Here’s the short version for anyone who isn’t neck-deep in containers: we run each agent job inside a locked-down sandbox — think of it like a disposable clean room the job lives in for a minute and then gets torn down. That day, every clean room was collapsing the instant we opened the door, and because the room self-destructs on exit, there was nothing left to inspect. If you’re not here for the Linux internals, the one thing to take away is this: when a worker dies in about a second with zero logs, the problem is almost never your task — it’s the room, not the work.
I did not believe that at first. I chased a broken account. I poked at the load balancer. I checked for version drift between images. Every one of those is a task-time failure — something that goes wrong after the worker is up and doing things. I was debugging the wrong hour of the timeline.
What tipped me off was the timing. Zero events and exit 1 in ~1 second. A worker that had actually started working would have posted something — a heartbeat, a log line, a partial event — before it fell over. Nothing at all means it died before task logic ever ran. That points at startup: the image, the entrypoint script, the capabilities, the tmpfs mounts.
The culprit was a PR from earlier that day. It added a new PID-1 entrypoint — the very first process the container runs — that did a chown and setpriv as root to isolate a GitHub token so the task couldn’t read it back. Reasonable idea. Except our sandbox launches with --cap-drop=ALL, which strips every Linux capability, including CHOWN. And the script ran under set -euo pipefail, so the first chown hit “Operation not permitted” and the whole thing exited 1 immediately.
The reason I couldn’t see any of this: --rm. The container was configured to delete itself on exit, and it took stderr down with it. The one message that would have solved this in thirty seconds — chown: Operation not permitted — was destroyed by the container before I could read it.
Reproducing an entrypoint that eats its own stderr give me the detail
The fix is to stop guessing and reproduce the startup with the real caps, env, and mounts — minus --rm, plus a shell you control:
# Reproduce the sandbox exactly, but keep the corpse
docker run --rm=false \
--cap-drop=ALL \
--security-opt=no-new-privileges \
--tmpfs /tmp \
-e GITHUB_TOKEN=fake \
our-worker-image:latest \
/entrypoint.sh; echo "exit=$?"
# -> chown: changing ownership of '/run/secrets/gh': Operation not permitted
# -> exit=1The actual fix was to add back only the capabilities the entrypoint needs, and nothing more:
--cap-drop=ALL
--cap-add=CHOWN --cap-add=FOWNER
--cap-add=SETUID --cap-add=SETGID
--security-opt=no-new-privilegesKeep no-new-privileges on — that’s what stops a process from gaining caps later. Re-adding CHOWN/SETUID for the entrypoint doesn’t undo it; the token isolation still holds. And drop set -e’s silent-death mode for anything at PID 1: trap the error and echo it somewhere that survives the container.
Why does dropping all capabilities break this specific pattern? Because chown and setuid-based privilege isolation are capability operations — CHOWN, FOWNER, SETUID, SETGID. When you drop ALL, you haven’t just tightened security; you’ve deleted the exact primitives your isolation script depends on. The security hardening and the security feature were fighting each other, and neither logged the fight.
So the rule I’d hand you: the shape of a failure tells you where to look. Instant death plus silent logs equals a startup failure — reproduce the container, don’t debug the code. And any entrypoint that chowns or setprivs after --cap-drop=ALL needs those four caps handed back explicitly, or it will die in a second and take the evidence with it.