In most projects, observability gets added after the first painful incident: hours of guessing in the dark, then a belated decision that "we need better monitoring." The saner — and far cheaper — approach is to build observability from day one, as part of the definition of done for every new feature. In this article we walk through the three pillars we establish at Grid with every system we build, and why AI systems have made that foundation more urgent than ever.
Watching charts is not observability
The difference is fundamental: traditional monitoring answers questions you decided on in advance — "what's the CPU usage?". Observability lets you ask questions you hadn't thought of when you built the system: "why do this specific customer's requests fail after 8 p.m.?". Real incidents are always the second kind — new questions nobody predicted.
An observable system answers questions that haven't been asked yet — without shipping new code to collect the data.
Pillar one: structured logs, not free text
A text log line like "payment error occurred" is read by one human; a structured log (JSON with stable fields) is queried by machines across millions of lines. From day one we commit to: every log line carries a correlation ID that follows the request across all services, an accurate severity level, and field names that are consistent across the entire system.
The payoff shows in the first investigation: instead of digging through scattered files, one query returns the request's full journey from the gateway to the database.
Pillar two: metrics that matter to the user
CPU and memory are useful, but they don't tell you whether users are suffering. The four golden signals are the foundation:
- Latency — specifically p95 and p99, not the average that flatters the picture.
- Traffic — requests per second, for context and peak awareness.
- Error rate — the share of failing requests, segmented by type and route.
- Saturation — how close resources are to their limits before they tip over.
These four per service, with one dashboard bringing them together, surface 90% of problems before customers report them.
Pillar three: distributed tracing ties the story together
In a multi-service system, logs and metrics tell you something is slow — distributed tracing tells you where. Every request carries a trace context that travels across services, so you see the journey as a waterfall: which service consumed the time, which call repeated needlessly. The OpenTelemetry standard has made this achievable with reasonable effort and no vendor lock-in.
AI systems raise the stakes
With agents and language models, the question is no longer "which service is slow?" but "why did the agent make that decision?". So we extend tracing to cover every run: the prompts used and their version, every model call with its inputs and outputs, every tool call with its result — all tied to a single run ID. Without that trail, debugging agent behavior in production is pure guesswork; with it, every decision is explainable and reviewable.
Alert on symptoms, investigate causes
Too many alerts are more dangerous than too few — the team learns to ignore them. The rule: page only on what touches users (error rate over threshold, noticeable latency, full outage), and make every alert actionable with a direct link to the investigation dashboard or the runbook. Internal-cause warnings — a filling disk, a lagging replica — go to daily review boards, not to waking someone at dawn.
A day-one checklist
- Are your logs structured with consistent fields and a correlation ID on every request?
- Do you measure the four golden signals per service, in percentiles rather than averages?
- Does distributed tracing flow through all your services via OpenTelemetry?
- Do your AI systems record every decision with its calls and outputs?
- Is every human-waking alert tied to real user impact and a clear action?
The bottom line
Observability is an investment that compounds with every incident: what would have taken a night of guessing becomes a minutes-long query. Start it on day one — structured logs, golden signals, distributed tracing, and a full trail of AI decisions — and you'll find that most major incidents were small signals that showed up early and nobody saw.
At Grid we build observability into every system we deliver — from day one, not after the first incident. If your system runs as a black box and you want to see inside it, get in touch.