A team adds a customer identifier to an API request metric to answer a support question. The next release adds request IDs because they help explain individual failures. Before long the dashboards are slower, the monitoring bill grows, and a perfectly ordinary traffic day looks like a capacity problem. The new labels answered a valid question. The team picked a data source whose cost and query behavior change with every new value.

Start with the decision the metric must support
Write down the decision before adding a label. An on-call engineer deciding whether an incident affects one operation or an entire service needs a different view from someone investigating a single customer report. For the first job, request counts, error rates, and duration by operation can be useful. For the second, a trace or a controlled lookup of structured logs can carry a request identifier if governance and retention rules allow it. The Google SRE book’s monitoring chapter gives a useful starting set of signals: latency, traffic, errors, and saturation.
Consider an illustrative API with 20 routes, four methods, and five outcome groups. Even with no customer label, multiplying those dimensions gives 400 possible combinations before instances or histogram buckets are considered. That is an upper bound for those dimensions, since some routes only accept one method and some outcomes never occur for certain routes. It is enough to show why the label’s possible values matter more than its short name. Adding a key with thousands of possible values can change the scale of the system.
Give each label a small vocabulary
Prefer values the team can enumerate or bound. For an HTTP server, a framework route template such as /orders/{id} groups requests across orders. The full path /orders/12345 lets the number of possible values grow with usage. The OpenTelemetry HTTP metric conventions say a raw URI path must not substitute for http.route. If your framework cannot reliably supply a route template, omit that attribute and retain aggregate service measurements while you improve the instrumentation.
Apply the same question to outcomes. A small list such as timeout, validation failure, dependency failure, and success may support a useful incident drill, provided the categories are defined consistently. A full error message can contain dynamic values and may accidentally expose information your monitoring audience should never see. Keep exception detail in an appropriately restricted tool. In the metric, map it to a stable outcome that someone can use to make a decision.
Put a budget and an owner on each change
For a change proposal, name the dashboard or alert the label will improve, the owner who will look at it, and the expected number of values. Then estimate the increase across targets and check it against observed time series before and after the release. Prometheus’s instrumentation guidance explains that each label set creates another series with resource costs. It makes sense to flag growth in an existing instrument as well as additions: a once-small tenant tier label may quietly become a list of account names if nobody owns its value mapping.
Accelerate Your Engineering Velocity
Looking to scale your infrastructure, implement enterprise AI automation, or secure critical cloud dependencies? Partner with MyNew Technologies LLC to engineer robust technical systems built for enterprise efficiency.
Request Technical ConsultationUse a small rollout to check how the label behaves with client errors, redirects, and requests that never match a route. Inspect the number of distinct values that actually appeared rather than relying on an average request volume. Compare dashboard latency and ingest volume to a representative baseline, and agree who can roll back the instrument if the prearranged budget is exceeded. Leave aggregate alerts running while you validate a replacement so you can still see real production failures.
Keep one path from alert to investigation
Give responders a way to move from a metric that shows a problem to a narrower search. A simple runbook can identify the service and operation to filter on, where to find traces or logs, and which fields are available under the team’s access rules. Try one incident scenario on a quiet day: find a group of timeouts on a dashboard, narrow the time window, then locate a representative request. If that works, adding an identifier to every metric is unlikely to be the best next investment.
To put the idea into practice, review one high-volume metric and remove or redesign any label no alert, dashboard, or runbook uses. If you are setting monitoring priorities across services, our Cloud Computing resources put that decision in the context of architecture and operations.

