When an application runs in one process, debugging is relatively direct: open the error record, find the relevant stack trace and inspect the logs around that moment. Once a request travels through an API gateway, identity service, message queue, background worker and third-party provider, the same method becomes insufficient. “A user could not pay” is no longer one error; it is a chain of events across time and components.
Observability makes that chain visible. With a good setup, a team can quickly answer three questions: What happened? Where did it happen? Why did it happen? Logs describe a detailed event, metrics show system-wide behaviour, and traces show the end-to-end path of one request.
Define incident questions first
Observability should not begin with installing a tool. Start by writing the operational questions that must be answered:
- Why did a user request become slow?
- Which endpoint increased the error rate?
- Why are messages in a queue delayed?
- How many people were affected by the same failure?
- Which workflows are affected when an external service slows down?
These questions decide which signal is needed. Finding where latency occurred needs a trace; seeing that error rate rose in the last hour needs a metric; understanding one person’s workflow needs contextual logs. “Log everything” produces an expensive, noisy system that often answers none of these questions.
Give a request a shared identity
The foundation of distributed tracing is a correlation ID or trace context. When a request enters the system, create a trace identifier or accept valid context from the upstream system. Carry that identifier through HTTP calls, queue messages and background jobs.
Every child span must inherit the parent context, keeping database calls, HTTP work and asynchronous consumers in one trace tree.
In queue-based architectures, put context into message headers. When a consumer handles the message, it should continue the message’s trace context rather than begin a new, unrelated transaction. Otherwise asynchronous boundaries break the trace tree precisely where it is needed most.
Traces show journeys; metrics show trends
A trace shows what happened and how long it took in one specific request. A span can represent GET /orders, a PostgreSQL query, a Redis access or a call to a payment provider. Add an operation name, duration, result status and small, bounded attributes to spans.
A metric shows aggregate behaviour rather than the detail of one request. Request count, error count, queue depth, processing duration and active connections can be tracked as time series. Averages alone are misleading for latency. Percentiles such as p95 or p99 often reveal the slow experience users actually see.
Together, the signals are powerful: a dashboard shows p95 latency rising, and a trace explains which dependency or query caused it. Treating traces only as a tool for isolated errors and metrics only as a management view loses that relationship.
Keep logs structured and contextual
Log lines that are only human-readable text are difficult to query. Structured logs, often in JSON, allow fields such as time, level, service name, environment, trace ID, span ID, operation name and error type to be filtered consistently.
Adding context does not mean logging sensitive data. Passwords, access tokens, complete payment data and personal document contents must never enter logs. When a user identifier is needed, a pseudonymous or technical identifier that fits the retention policy is safer. Logging is a diagnostic tool, not a second data store.
For an error record, the exception class, error code, dependency name and retry information can be useful. Instead of generic text such as “operation failed”, choose stable fields that let the team group error types.
Limit operation names and attributes
Poorly named traces are as unhelpful as unsearchable logs. Use GET /users/{id} rather than a route containing a user identifier such as GET /users/123. The same rule matters even more for metric attributes: user IDs, email addresses, random request IDs or unbounded URL paths have high cardinality and make a metrics system slow and expensive.
Good attributes are finite and meaningful: HTTP method, result class, service name, queue name, data centre or feature version. Before adding an attribute, ask: “How many distinct values do I want to group by?” If the answer is unbounded, the value usually belongs in a log or trace attribute, not a metric label.
Connect alerts to user impact
Not every error should raise an alert. A brief single 404 or a temporary error that succeeds after retry may not warrant a late-night page. Alert rules should account for user impact, error budget and persistence.
For example, “p95 latency is above its objective for five minutes while request volume is meaningful” is healthier than alerting on one slow request. For queues, track not only depth but also the age of the oldest message; a low-volume message that has waited for hours can still be a serious business issue.
Every alert should have a short runbook: which dashboard to open, which trace to query, the first safe step and when escalation is needed. An alert should do more than announce a problem; it should reduce the cost of beginning the investigation.
Start with a small contract
Perfect telemetry across the whole system is not required at once. Start with trace context at request boundaries, basic HTTP metrics and structured error logs. Then extend coverage to the two most critical dependencies, queue consumers and the most valuable workflows.
The measure of success is not the number of dashboards. It is the time required to answer which service made a request slow, which dependency failed and how many users were affected. If those questions can be answered in minutes during an incident, the observability foundation is doing its job.