I noticed the A2A message ledger was lying to my agents.
Nothing dramatic — no crash, no exception, no failed query. Just an agent reading its conversation history and occasionally seeing messages in slightly the wrong order. One message a slot too early. Another arriving late. The agent’s picture of what had been said was subtly, silently wrong.
I run a fleet of Claude Code sessions — autonomous agent instances working in parallel across several machines and Anthropic accounts. They talk to each other through an agent-to-agent (A2A) message ledger: an append-only Postgres table. An agent posts a message to another agent, and the recipient reads the conversation back out. Simple.
For months, the ledger was ordered by created_at — the timestamp Postgres stamps on each row when it’s inserted. That worked fine. Until it didn’t.
The kind of bug you catch by feeling something’s off, not by reading a stack trace.
The cause was clock skew. My fleet spans a few laptops that suspend and resume, plus a VM. When a laptop wakes from sleep, or when NTP (the Network Time Protocol — the background service that keeps a machine’s clock synced to internet time servers) corrects a drifted VM clock, the system clock can step backward. By a fraction of a second, sometimes a full second or two.
Now picture this: Agent A inserts a message. A second later, the laptop suspends. It wakes up, NTP corrects the clock backward by 1.5 seconds, and Agent B inserts a reply. That reply gets a created_at timestamp earlier than the message it was replying to. The chronological sort now says B spoke before A. The table lies about what actually happened.
The fix took ten minutes and is permanent: stop ordering by timestamp. Order by the database’s auto-increment integer primary key instead. The row ID is a monotonic counter — it only ever goes up. It has zero relationship to the wall clock, can’t step backward, and reflects actual insertion order by construction, not by measuring a clock I don’t control.
I kept the timestamp column. It’s real metadata — useful for display, for range queries, for knowing when something happened. It just doesn’t get to decide what came first anymore.
This isn’t just about my little A2A ledger. It applies to any append-only log: event streams, audit trails, chat histories, agent message queues. If order matters and the table only grows, the row ID is the right sort key.
Postgres BIGSERIAL, IDENTITY columns, and keyset pagination give me the detail
Use BIGINT GENERATED ALWAYS AS IDENTITY (or BIGSERIAL) as the primary key, and treat created_at as metadata only:
CREATE TABLE agent_messages (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
created_at timestamptz NOT NULL DEFAULT now(), -- metadata, not ordering
sender text NOT NULL,
recipient text NOT NULL,
payload jsonb NOT NULL
);
-- correct ledger order — monotonic, clock-skew-immune:
SELECT * FROM agent_messages
WHERE recipient = $agent_id
ORDER BY id ASC;The same id column gives you keyset pagination for free, which you want on a ledger that grows indefinitely. OFFSET gets slower the deeper you page and can skip or repeat rows when the table is written to concurrently:
-- "give me messages after the last one I saw"
SELECT * FROM agent_messages
WHERE recipient = $agent_id
AND id > $last_seen_id
ORDER BY id ASC
LIMIT 100;One thing worth knowing: Postgres sequences are monotonic but not gapless — a rolled-back transaction leaves a hole in the ID sequence. That’s fine for ordering and cursor pagination; just don’t assume “next ID = previous ID + 1.”
If you’re on a distributed store with no single sequence (sharded database, multi-writer setup), reach for a monotonic ID scheme like a Snowflake ID or ULID rather than falling back to the wall clock. The property you need is “monotonically increasing by construction” — not “high-resolution timestamp.”
The rule is simple: when you need strict ordering, build it on something monotonic by construction. Not on a measurement that can be corrected, adjusted, or skewed. A timestamp is a measurement of a clock I don’t control. A row ID is a counter I do. Reach for the counter.