Observability: Sentry in Every One of the Seven Workers

Photo: Rawpixel (CC0 1.0), via Openverse

The web server and the API had been speaking up for weeks whenever something went wrong. A Sentry event, a stack trace, at best the exact line where it broke — for anything a user touches directly, visibility was good. What struck me this week was that the part of the platform that faces no one was practically blind. The background services ran, did their work, and if they stumbled, I'd find out at best days later through a side effect — a bill that never arrived, a sync that didn't happen.

That's exactly the uncomfortable class of failures: not the ones that crash loudly, but the ones that happen silently. A worker that swallows an exception and simply carries on at the next interval looks healthy from the outside. This week, then, wasn't about a new feature but about giving each of the seven background workers a voice — one that speaks up before anyone else notices the problem.

Why the Workers Were the Blind Spot

The seven workers are the engine room of the platform. They sync health data from connected sources, run through billing cycles, execute privacy sweeps, and keep derived data up to date. Each of them is its own process, often on its own schedule, and none of them has a UI where a failure would show up.

Until now, my only visibility into failures in these services was whatever the OpenTelemetry exports from Aspire provided — useful at runtime, but no tool for reconstructing two days later why a particular run aborted at three in the morning. A log nobody reads isn't monitoring. I wanted to reach the point where an exception in a worker takes the same path as an exception on the web: captured, attributed, with context, and in a place where I actually see it.

One Shared Building Block Instead of Seven Copy-Paste Blocks

The obvious temptation would have been to copy the same Sentry setup code into every worker. Seven near-identical copies, with the usual result: after three months they subtly diverge, because someone applies a fix in only one place. Instead, all the wiring now sits behind a single call — AddXircuitWorkerSentry. Every worker picks up the same configuration this way: the same default tags, the same environment handling, the same way of attaching context to an event.

The benefit isn't just less code. It's that observability becomes a property of the platform, not a property of whichever developer wrote a given worker. If an eighth worker joins next week, the question is no longer "did I forget the monitoring," but a single line in the startup configuration. And if I want to change the behavior for all of them — an extra tag, a different sampling rate — that happens in exactly one place.

One DSN per Worker, Not One for All

The more interesting decision was how finely I wanted to separate the errors. A single Sentry DSN for "the workers" would have been simpler, but it would never have answered the real question: which service has the problem. In a shared pool, a billing error and a sync timeout blur into a diffuse mass of events that then has to be laboriously sorted back apart.

That's why every worker gets its own DSN. An error in the privacy sweep lands in a different project than an error in the billing worker, and the two are distinguishable at a glance. Release health, alert rules, and the question "is this service acting up right now" can be answered per service instead of in aggregate. Environments — development, test, production — are separated within a single project via an environment tag, not via further projects; that keeps the number of DSNs equal to the number of services while keeping the views clean.

Photo: Lorenzo Cafaro · StockSnap (CC0 1.0), via Openverse

The Trick with the Early DSN

One detail caused more head-scratching than it looks like from the outside: when the DSN becomes available. In production, secrets live in Key Vault and are written there by an initialization step. But that means a worker that pulls its DSN exclusively from Key Vault is blind during exactly the window when something is most likely to break — at startup, before initialization has finished.

But boot failures are exactly the ones you least want to lose. A worker that trips at startup over missing configuration or an unreachable dependency otherwise never reports in, because it never gets far enough to load the DSN. The solution is two-stage: the DSN comes from the environment first — early enough that Sentry is already up before anything else initializes — and later from Key Vault, once the regular configuration kicks in. That covers the early startup and keeps ongoing operation cleanly supplied through central secret management. No window in which a failure would be invisible.

So No Failure Stays Silent

The real goal behind the whole mechanism is easy to state: no worker failure should stay silent anymore. If a sync job hits an unexpected response from an external source, if a billing run stumbles over an edge case, if a privacy sweep can't process a record — there's now an event, attributed to a specific service, with the context needed to understand what happened.

That noticeably shifts the way of working. Before, the usual sequence was: someone notices a missing side effect, I work backward to figure out which service is responsible, and dig through logs. Now the order is reversed — the service reports itself, at the moment the failure occurs, not when its consequences become visible. That's the whole point of observability here: pushing the time between "something went wrong" and "I know about it" down to nearly zero.

The Flip Side: A Silence Layer Against the Noise

Monitoring that reports everything is, in its own way, useless. As soon as the seven workers were connected, the expected effect showed up: some of the events weren't real problems but known, harmless exceptions — things that occur during normal operation and require no action. When those fill up the channel, you get used to clicking Sentry alerts away, and that's exactly when you miss the one event that matters.

That's why a silence layer was added afterward: a filter layer that strips out these known, non-critical exceptions before they become events. The bar here is delicate, and I treat it deliberately conservatively — it should get quieter, not deaf. Only what I've understood and classified as harmless gets filtered; everything else comes through. A signal that gets lost in the noise is worth as little as no signal at all, and this last layer is the difference between monitoring you actually look at and monitoring you ignore.

What's Next?

Next week brings a production E2E suite that runs real personas against the live platform with Playwright. The workers now speak up when something goes wrong — the next step is to systematically check whether everything actually works under normal operation, on real production, not just on my machine.