Observability bills have a way of growing faster than the systems they describe. Every new service adds metrics, logs and traces, retention policies default to long periods, and nobody is sure which data anyone reads. Teams then face an uncomfortable choice between cutting visibility and accepting runaway costs. A better answer is to treat telemetry as a pipeline with deliberate decisions at each stage, not as a firehose pointed at a vendor.
Start by measuring what you collect and what you use. Break down volume and cost by service, signal type and team, then compare it with actual consumption: which dashboards are opened, which alerts fire, which log queries are run. It is common to find that a small number of noisy sources account for most of the volume and that large amounts of data have never been queried. That analysis gives you targets that matter and an evidence base for conversations with teams who own the noisy sources.
Metrics cost is driven by cardinality. A single label such as a user identifier or a request path with embedded values can multiply the number of time series by orders of magnitude. Review labels regularly, drop or aggregate those that do not support a decision, and set limits that reject accidental explosions. Pre-aggregating high-volume metrics at the edge, for example computing histograms locally, preserves the shape of the data at a fraction of the cost.
Logs benefit from structure and tiering. Unstructured text is expensive to store and slow to query, while structured events can be filtered and sampled intelligently. Route logs by value: keep errors and audit records at full fidelity, sample high-volume informational logs, and send bulk debug output to cheap object storage that you can query on demand. Dropping health-check noise and duplicate retries at the pipeline often removes a surprising share of the volume without losing any information you would need.
For traces, tail-based sampling keeps the traces that are slow, erroneous or otherwise unusual, and samples the ordinary ones heavily. This retains the diagnostic value while reducing storage by a large factor. Make the sampling rules explicit and visible, and always keep enough baseline traces to understand normal behaviour. Pair this with exemplars that link metrics to specific traces, so engineers can jump from a spike to a representative request without storing everything.
A collector layer gives you control. Using an open standard collector in front of your backends lets you filter, transform, redact and route data in one place, and it avoids lock-in because you can change destinations without re-instrumenting applications. It is also the right location to remove sensitive fields before data leaves your environment, which reduces both cost and compliance risk.
Make cost visible to the people who create it. Show each team its telemetry spend alongside its service dashboards, and set budgets with alerts rather than surprise invoices. Review retention annually, shortening it where long history is not used. The objective is not to collect less for its own sake. It is to make sure that every byte stored is there because someone could plausibly need it, and that the most valuable signals stay available when an incident makes them matter.