Reading a service token from /proc came back empty

On Tuesday morning, an agent pane tried to dispatch a workflow to my conductor service. The conductor rejected the HTTP request with a 401 Unauthorized status code. A conductor service is a central daemon that receives tasks and routes them to worker seats. A 401 code means the client failed to present valid credentials. The dispatch script had pulled the bearer token out of thin air, or so it seemed. The token string was completely empty.

If you do not run background services on Linux, the takeaway is simple: never steal secrets from another process through /proc. Store credentials in a secrets manager. Add a lightweight health check that proves the token works before you spawn background work.

The wrong turns

When a service rejects a valid caller, the obvious suspect is a crash. I checked systemctl status conductor. The process was healthy. It had been running continuously for four days without a restart. Next, I checked the master secret in 1Password. I ran a test curl request using the token from the vault. The request succeeded immediately with an HTTP 200 response. The server accepted the secret. The caller sent an empty string.

Why did the caller send an empty string? I opened the helper script that dispatches work. The script found the conductor process identifier, or PID. Then it ran sudo cat /proc/$PID/environ. /proc is a virtual filesystem in Linux. It exposes internal kernel data about active programs. The environ file contains the initial environment variables recorded when a program starts. The helper script ran grep on that file to find CONDUCTOR_TOKEN. I ran the command manually in my terminal:

sudo tr '\0' '\n' < /proc/$PID/environ | grep CONDUCTOR_TOKEN

The terminal printed nothing. I checked the file size with wc -c. The command returned zero bytes. The file was completely empty.

Why /proc/PID/environ went blank

A few weeks ago, I wrote about trusting /proc/$PID/environ over tooling that hides state. That advice holds when you debug your own process. It fails when you rely on it as an inter-process communication bus.

Here is what happens inside the operating system. When Linux starts a process, it copies the initial environment block into the process memory map. The kernel creates the /proc/$PID/environ virtual file as a read-only mirror of that initial block. It is a historical record from fork time, not a live window into memory.

Three different mechanisms can wipe that record clean.

First, runtime engines like Bun and Node.js often sanitize their memory during startup. Some runtimes scrub initial environment pointers to prevent memory inspection attacks. The running code still reads process environment variables from internal heap tables. The initial kernel address table becomes zeroed or unmapped.

Second, systemd can clear environment pointers after execution transitions. When a unit completes service initialization, security policies can isolate process memory.

Third, sandbox restrictions silently deny reads. A script running with sudo might hit AppArmor or namespace boundaries. Instead of returning an error, the kernel returns an empty stream.

The conductor still held the token in its runtime memory. The kernel file that recorded startup state was empty. Reading credentials from /proc is like reading a carbon copy left on a desk. Someone wiped the paper, but the owner still knows the combination.

Fixing the source of truth

Scraping /proc was a lazy shortcut. I had used it because the conductor already knew the token, and the caller wanted to avoid another network call. That shortcut created an invisible coupling. It turned an internal operating system detail into an operational dependency.

I replaced the file scrape with an explicit read from the vault:

CONDUCTOR_TOKEN=$(op read "op://Engineering/conductor/token")

The caller now resolves its own token from the same source of truth the conductor uses. No sudo required. No dependency on kernel memory maps. No mystery when a runtime updates its memory management.

The pre-flight probe

Fixing the secret source exposed a second trap. The caller also had an envelope bug. The conductor expects inputs wrapped inside an inputs JSON key. The caller was sending parameters at the top level of the request payload.

Because the caller failed silently on the empty token, I had not seen the payload error. When you fix authentication, the payload schema error arrives next.

To stop late failures, I added a pre-flight probe before any heavy task spawns:

curl -f -s -H "Authorization: Bearer $CONDUCTOR_TOKEN" \
  http://127.0.0.1:4040/health > /dev/null

If the token is empty or invalid, the pre-flight check fails within 50 milliseconds. The dispatch halts before creating scratch directories, acquiring locks, or logging task records. The error message says exactly what broke: authentication failed at the door.

The named thing

Startup snapshot is not process memory.

/proc/$PID/environ records what a program received when it was born. It does not guarantee what the program holds right now. More importantly, it does not guarantee the kernel will keep showing it to you.

When two services need to talk, give each service its own path to the secret. Treat /proc as an emergency forensic tool for humans, never as an API for automation.