← All posts

Git Worktrees Ate My Edits — Why We Switched to Dedicated Machines for Agent Isolation

I was in the middle of a refactor — removing dead code from a shared SDK module — when my edits vanished. No error. No warning. Just gone.

  • ai
  • agents
  • git
  • infrastructure
  • engineering

I was three files deep into a refactor — stripping dead code from a shared SDK module — when my edits just vanished. No error. No warning. Not even a flicker in the terminal.

Gone.

I’d been running a fleet of Claude Code agents, each in its own git worktree (an isolated working directory that shares the same underlying .git folder — same history, same branches, same everything). While three agents churned through feature branches in their own little sandboxes, I was editing files on the main checkout. One git checkout -- . to clean up stray files an agent left behind, and my staged changes evaporated with them. The worktree isolation model had failed. Silently.

Git worktrees sharing state while dedicated machines stay isolated

The Setup That Seemed Right

Git worktrees are genuinely elegant for parallel agent work. Each agent gets its own working directory, its own checked-out branch — but they all share the same .git directory underneath. Same object store, same refs, same index. Claude Code’s isolation: "worktree" flag creates these automatically when you fan out agents.

/opt/project/                    # Main checkout (orchestrator)
/opt/project/.claude/worktrees/
  ├── agent-a1b2c/              # Agent 1's worktree
  ├── agent-d3e4f/              # Agent 2's worktree
  └── agent-g5h6i/              # Agent 3's worktree

Lightweight, no network overhead, each agent on a different branch without cloning the entire repo. On paper, it’s the right abstraction.

So what went wrong?

What Actually Happened

The orchestrator session — my main Claude Code pane that was coordinating all three agents — was editing files directly on the main checkout. Meanwhile, three agents were running in their worktrees, doing feature work on separate branches.

Here’s the exact chain of events:

  1. I edited agent_sdk.py on the main checkout, staged it with git add
  2. A worktree agent hit an issue — maybe a lock contention, maybe a permissions hiccup — and fell back to operating on the main checkout directly instead of its own worktree
  3. That agent modified files I wasn’t tracking: daemon.py, a handful of test files
  4. I ran git checkout -- . to clean up the agent’s mess, thinking I was only discarding the files that agent touched
  5. That command restored ALL tracked files to their HEAD state — including my staged agent_sdk.py changes

Poof. No git reflog rescue because the changes were only staged, never committed.

I paused, stared at a blank diff where my work should have been, and felt that specific kind of stupid you feel when you trusted a system that wasn’t designed for what you were asking of it.

The fix was simple: cp -r the repo to /tmp/ — a completely independent clone with its own .git directory, its own index, its own lock namespace. Made my changes there, committed, pushed. No interference possible. That’s when it clicked: the workaround was the answer.

Why Worktrees Can’t Be Fully Trusted

The shared .git directory is the problem. Everything else follows from that one design decision.

the mechanism — why shared .git breaks under concurrent agents give me the detail

The root cause is git’s single-writer locking model. Every git command that mutates refs or the index acquires .git/index.lock. Two concurrent writers race for that lock — the loser gets fatal: Unable to create '.git/index.lock': File exists. But the worse failure is silent: git checkout -- . reads HEAD and the current index to restore tracked files, with no awareness that a different process staged changes in the same index moments earlier. The index has one slot per path; the last writer wins.

You can reproduce the race locally in two terminals:

# Terminal 1 — simulates your editing session
git add some_file.py
sleep 5                         # hold the staged state

# Terminal 2 — simulates an agent cleanup
git checkout -- .               # wipes Terminal 1's staged changes silently

No error. No warning. The staged diff is gone.

Why a separate clone (or a dedicated machine) fixes this completely: each clone has its own .git directory, its own index, its own lock namespace. There is no shared mutable state to race on. Agents push to the same remote via git push origin <branch>, coordinating through the remote ref namespace — the one place git is designed for concurrent access (it’s append-only via pack-refs and server-side locking).

For the dispatch layer, ssh -o BatchMode=yes agent-box "cd ~/project && git pull --ff-only && <command>" gives you a clean, synchronous handoff with a non-zero exit code on any failure — a behavioral contract worktrees can’t offer.

When an agent hits a worktree error — permissions, lock contention, a disk hiccup — the natural fallback is to operate on the original checkout. That fallback is silent and destructive. You don’t find out until you go looking for your work and it isn’t there.

Mitigations I Considered (And Why They All Failed)

I ran through every standard hardening approach before accepting that the foundation was the problem:

ApproachWhy It Fails
”Just remember not to edit the main checkout”Fragile under pressure — fails on attempt #100
Lockfile guard scriptVoluntary compliance, easy to bypass
Filesystem permissions (chmod -R a-w)Breaks git fetch/git pull
Orchestrator also uses a worktreeStill shares .git, still has edge cases
Bare repo with all worktreesAdds complexity without eliminating shared state

Every mitigation was trying to bolt discipline onto a fundamentally shared resource. The answer wasn’t more rules. It was removing the shared resource entirely.

Dedicated Machines

We already had a fleet of agent machines on the same LAN — small form-factor PCs running Ubuntu, each with its own disk, its own git clone, its own everything. I started dispatching work to them instead of spawning worktrees, and the entire class of problem disappeared.

What you get:

  • Total filesystem isolation — separate .git, separate object store, no shared state
  • Zero discipline required — there’s nothing to accidentally corrupt because there’s nothing shared to corrupt
  • Clean failure modes — if an agent goes haywire, it trashes its own box, not your working directory
  • No fallback path — an agent physically can’t “fall back” to editing the orchestrator’s files; they’re on different machines

The overhead? About 2-3 seconds per dispatch over SSH on a local network. For tasks that run minutes to hours, that’s noise.

# Before: worktree (shared .git)
claude --worktree /opt/project "fix the auth module"

# After: dedicated machine (fully isolated)
ssh agent-box "cd ~/project && claude 'fix the auth module'"

Stripe Got Here First

This isn’t a unique observation. Stripe’s engineering blog documented the same journey — they moved away from git worktrees for their agent fleet for the same reasons. Shared git state creates subtle, hard-to-debug corruption. The failure mode is always silent data loss. That’s the worst kind of bug: you don’t know it happened until you need what’s gone.

Production Details

  • Fleet: 15 dedicated agent machines, each with full repo clones
  • Dispatch: Custom CLI tool routes work to available machines via SSH
  • Sync: Agents push to the same remote origin — coordination happens through git branches, not shared filesystems
  • Orchestrator: Development server stays clean — only used for human editing and dispatch coordination
  • Overhead: 2-3 seconds SSH latency per dispatch, negligible for tasks that run minutes to hours

The Two Things I Actually Learned

Structural isolation beats behavioral discipline. If the wrong action is possible, someone — or some agent — will eventually take it. Worktrees require you to remember rules. Dedicated machines make the wrong thing impossible. That’s the difference between a system you trust and one you hope you’re paying enough attention to.

Cleverness isn’t correctness. Worktrees are clever: shared object store, lightweight branching, no network overhead. But cleverness that creates silent failure modes is worse than a blunt solution that just works. The elegant answer and the right answer are not always the same answer.

Build for the failure mode, not the happy path. Worktrees work perfectly 99% of the time. The 1% failure — silent data loss with no recovery path — is catastrophic enough to justify the simpler, heavier approach every time.


Built with Claude Code by Anthropic. Inspired by Stripe’s blog post on agent infrastructure. Fleet management via custom dispatch tooling.