One of my apps has a support bot that reads incoming user messages and, when something looks like a real bug, files a GitHub issue on its own. Some of those issues also get an ama label — a tag that kicks off an auto-dispatch, meaning an agent picks the issue up and starts working it without me. That’s a lot of trust to hand a bot, so I wanted the “which issue-classes are allowed to auto-dispatch” list to be a deliberate decision someone makes on the box, not something baked into the code where it silently ships on.
So I gated it behind an environment variable — think of an env var as a sticky note the running program reads at startup. Mine is AMA_AUTOSTAMP_CLASSES, and it ships empty. Empty means nothing auto-labels. Widening it to a new class of issue is an ops action, on purpose, class by class.
The trick was where to put the flip. My deploys are git pull plus restart, which never rewrites the service definition. If I’d put the flag in the repo, “config” and “code” would blur together and every deploy could quietly change behavior. Instead I dropped it into a systemd drop-in — a small config file that lives at /etc/systemd/system/<service>.d/ on the machine, outside source control entirely. It survives every deploy because deploys don’t touch it. Box config stays on the box.
Then I went to confirm it was actually live, and this is where I burned an hour.
I ran systemctl show and saw my variable. Great. Except systemctl show reports what’s configured, not what the process actually inherited at launch. Those can differ if you forgot to reload or restart. So it tells you the plan, not the reality.
Fine — I’ll check the process directly. I ran pgrep, grabbed a PID, read its environment, and got a flat NOT VISIBLE. My heart sank. Turns out the app spawns several processes off the same binary, and pgrep had handed me a stray worker that never got the env, not the real service. I was reading the wrong process and drawing conclusions about the right one.
The fix was to stop guessing which process to trust. Ask systemd for the one PID it considers the service — the MainPID — and read that process’s actual inherited environment off /proc. Ground truth, no interpretation layer.
Verifying the effective env of a systemd service give me the detail
systemctl show prints the unit’s configured Environment=, not what the live process inherited — and pgrep -f against a multi-process app returns siblings that never got the drop-in. Resolve MainPID, then read the kernel’s copy of that PID’s environment:
# The one PID systemd considers the service
MP=$(systemctl show -p MainPID --value my-support-bot.service)
# /proc/PID/environ is NUL-separated; make it greppable
tr '\0' '\n' < /proc/$MP/environ | grep AMA_AUTOSTAMP_CLASSESThe drop-in itself, at /etc/systemd/system/my-support-bot.service.d/autostamp.conf:
[Service]
Environment=AMA_AUTOSTAMP_CLASSES=bug,crashThen systemctl daemon-reload && systemctl restart my-support-bot.service — the reload alone won’t move the running process’s environment.
The lesson that transfers past systemd: when you want to know what a running program actually got, don’t ask the config layer and don’t ask a name-matcher that returns a crowd. Find the exact process the system owns and read its real state. Configured is a promise. Effective is the truth, and they live in different files.