I filed four GitHub issues, realized three of them should wait, and calmly removed the label that queues them. Thirty seconds later all four were building.
Here’s the setup, because the trick is in the plumbing. I run a system I call AMA where I describe a feature as a GitHub issue and tag it with a label — ama — and a background watcher (a “poller,” a little loop that wakes up on a timer and checks for new work) notices the tag and hands the issue to a coding agent that actually writes the code. Think of it like a mailroom: I drop labeled envelopes in the tray, and every minute someone sweeps the tray and starts working on whatever’s labeled. My mistake was thinking I could reach back into the tray after the sweep.
The poller runs a 60-second cycle and grabs up to three issues per pass. The first thing it does with an issue is flip the label from ama to ama:working and dispatch a build agent. That relabel is the point of no return — one-way, done, agent’s already running.
So when I filed four, then went to “hold” three by removing the ama label, I was editing a label that no longer existed. The poller had swept, relabeled them to ama:working, and kicked off builds. My remove-label call succeeded against nothing. A clean no-op. All four building in parallel while I sat there thinking I’d stopped them.
The wrong mental model was treating the label like a queue I owned. It wasn’t mine. The moment I filed, the poller and I were both writing to the same field, and it writes faster and more often than I edit. You cannot gate ordering at a piece of mutable state that a fast, one-way consumer also writes. The dispatch fires before your edit lands, and dispatch is irreversible — you can’t un-run an agent.
What actually fixed it: I stopped trying to sequence at the label. In AMA the only step that’s genuinely human-gated is the merge — I review every PR before it lands. That’s the real chokepoint. So let all four build in parallel, generate their four branches, and enforce order where a human already stands guard: I merge them in the sequence I want, one at a time. Upstream runs wild; the gate holds at the door that was already locked.
Where to put the gate in a poller-driven pipeline give me the detail
The failure is a classic read-modify-write race, except I wasn’t even the one modifying. The poller’s transition is atomic-ish from my side:
# conductor poll loop (every 60s, maxPerCycle=3)
for issue in gh.issues(labels=["ama"])[:3]:
gh.set_labels(issue, ["ama:working"]) # <-- one-way, irreversible dispatch
dispatch_build_agent(issue)Any edit I make to the ama label lives or dies on whether my write happens before that set_labels call. With a 60s cycle I have, on average, 30 seconds — and no lock. Don’t design around winning that.
The fix isn’t a faster remove or a lock on the label. It’s recognizing that dispatch_build_agent has no inverse, so ordering can’t live at that stage. Find the stage that is reversible-or-blocking — here, the manual merge — and serialize there:
# builds run in parallel; ordering enforced at the one human-gated stage
ready = [pr for pr in open_prs() if pr.checks_passed]
for pr in sorted(ready, key=lambda p: p.intended_order):
await human_approval(pr) # the real gate
gh.merge(pr)Rule of thumb: in an event-driven pipeline, locate the single stage that blocks on a human (merge, approval, deploy) and enforce sequence there. Everything upstream of it should be free to run in parallel, because you can’t take it back anyway.
If you build these pipelines, the instinct to “cancel by editing state” is the trap. Once you’ve handed work to a consumer that dispatches on sight, that work is gone. Sequence at the gate that was already locked, and let the rest race.