← All posts
observabilityalertingsreprometheusthanos

Observability without the noise: how to design alerts your team will actually trust

Most observability stacks fail not because they collect too little data, but because they raise too many false alarms, or stay silent when something really is wrong. A practical playbook for high-signal alerting in production, with the queries.

By Victor Brinkhorst

Most teams we work with don’t have a monitoring problem. They have an alert fatigue problem. The dashboards look great in demos. The pages don’t.

A useful alert tells an on-call engineer three things in the first five seconds:

  1. What is broken? (api-gateway p95 latency, not latency_alert_3)
  2. For whom? (customer-facing, internal, or background?)
  3. Why now? (slow burn, sudden spike, deployment-correlated?)

If your alerts can’t answer those, your team will eventually stop reading them. That’s the point at which observability has actively made your reliability worse.

Three habits that quietly poison alert quality

1. Alerting on symptoms instead of SLOs

A 200ms p95 threshold is meaningless without a target. Alerting on a budget burn rate tied to a real SLO (“99.9% of checkout requests under 400ms over 28 days”) gives you a stable signal that doesn’t require constant tuning.

The mechanism: you have an error budget of 0.1% over 28 days. A burn rate of 1 consumes it exactly on schedule; a burn rate of 14.4 consumes 2% of it in an hour. You alert on the burn rate, not the raw latency.

Start with a recording rule, so the ratio is computed once rather than in every alert and dashboard panel:

groups:
  - name: slo-checkout
    interval: 30s
    rules:
      # Fraction of checkout requests that FAIL the objective
      # (error, or slower than 400ms). One rule per window.
      - record: job:slo_checkout_bad_ratio:rate5m
        expr: |
          (
            sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
            +
            sum(rate(http_request_duration_seconds_count{job="checkout"}[5m]))
            -
            sum(rate(http_request_duration_seconds_bucket{job="checkout",le="0.4"}[5m]))
          )
          /
          sum(rate(http_request_duration_seconds_count{job="checkout"}[5m]))

      - record: job:slo_checkout_bad_ratio:rate1h
        expr: # ...same expression with [1h]

Then alert on two windows at once. The long window says “this is real”; the short window says “this is still happening”, which is what stops an alert firing for an hour after the incident resolved:

      - alert: CheckoutErrorBudgetBurnFast
        # 14.4x burn = 2% of a 28-day budget in 1 hour. Page.
        expr: |
          job:slo_checkout_bad_ratio:rate1h  > (14.4 * 0.001)
          and
          job:slo_checkout_bad_ratio:rate5m  > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Checkout burning error budget 14.4x. 2% gone in the last hour"
          runbook_url: "https://runbooks.internal/checkout-latency"

      - alert: CheckoutErrorBudgetBurnSlow
        # 6x burn over 6h = a real but slower leak. Ticket, don't page.
        expr: |
          job:slo_checkout_bad_ratio:rate6h  > (6 * 0.001)
          and
          job:slo_checkout_bad_ratio:rate30m > (6 * 0.001)
        for: 15m
        labels:
          severity: ticket

Two rules replace a dozen static thresholds, and neither needs retuning when traffic doubles.

2. Pages without a runbook

If the on-call engineer’s first reaction is “what is this even”, the alert isn’t doing its job. Every page should link to a one-paragraph runbook: what it means, where to look first, and how to silence it if it’s a known pattern.

Note the runbook_url annotation above. Make it mandatory. You can enforce that with a query against your own rule metadata, and fail CI when a severity: page rule lacks one.

3. Alerting on the same incident from three places

Latency, error rate, and saturation often spike together. Three pages for one root cause is three times the cognitive load. Group them, route them through a single notification, and let the on-call dig into the correlated signals via the dashboard, not the inbox.

In Alertmanager that is mostly one block:

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s        # collect siblings before the first notification
  group_interval: 5m     # batch updates to an existing group
  repeat_interval: 4h

Grouping by service rather than alertname alone is the part people miss: it collapses different alerts about the same service into one notification, which is exactly the three-pages-one-cause case.

The alert that matters most is the one you never wrote

Everything above assumes your monitoring is working. Here is the failure mode that costs more than all the noisy alerts combined: the metric that returns no data at all.

On a production Kubernetes cluster we inherited, the Thanos compactor had been halted for roughly three months before anyone noticed. Compaction had stopped, which meant long-term storage was quietly degrading and queries over historical data were getting slower and more expensive.

There was an obvious metric for it:

thanos_compact_halted

It returned nothing. Not 0, but nothing at all. Prometheus defined only two scrape jobs of its own, and all real scraping was delegated to Grafana Alloy, whose discovery kept pods carrying prometheus.io/scrape: "true". The Thanos chart shipped no such annotation, so no Thanos component had ever been scraped. The metric that would have caught the problem had never existed.

The fix is an annotation, with the port pinned:

global:
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: /metrics
    prometheus.io/port: "10902"

The port matters. Alloy generates one target per container port and doesn’t filter by name, so without pinning it the gRPC port (10901) gets scraped too and sits permanently down.

But the durable lesson is in the alert rule that came out of it:

      - alert: ThanosCompactorHalted
        expr: max(thanos_compact_halted) > 0
        for: 15m
        labels:
          severity: critical
          component: thanos

…configured with noDataState: Alerting. That single setting is the whole point. The failure mode that hid this for three months was the metric being absent, so “no data” has to page exactly as loudly as “halted”. A rule that only fires on > 0 would have stayed silent for another three months.

So audit for blind spots directly. This finds every target that is down:

up == 0

And this finds the more dangerous case: a job you believe you are scraping that has no targets at all:

absent(up{job="thanos"})

Run absent() against every component you would page on. If it returns a series, you have an alert that can never fire.

While you’re in there: check what you’re paying to store

The same cluster was producing 2h Thanos blocks of 2.8–2.9 GB. At ~36,000 samples/sec and ~275k active series, the arithmetic said they should be 0.3–0.5 GB, six to nine times smaller.

The blocks weren’t malformed; they held several copies of the same data. Alloy emits one target per container port, and the scrape job rewrote all of them to the same address while labelling them by container. That made otherwise-identical targets distinct, so the same endpoint was scraped and stored once per container:

# One pod, three containers, three scrapes of the same :9090 endpoint
up{job="prometheus.scrape.annotated_pods", pod="prometheus-server-0"}

Cluster-wide: 66 targets for 26 pods, 61% of them duplicates. Dropping the container label collapsed them. One pod went from 3 targets to 1, and its head series count from 3 identical copies to 1.

You can check your own cluster for this in one query:

# Any instance being scraped more than once is a duplicate
count by (instance) (up) > 1

This inflated everything downstream: series count, sample volume, S3 block size, Store Gateway load, and the compactor’s scratch disk. Noise isn’t only what pages you at 3am. It’s also what you pay to keep.

Where we usually start

When we’re brought into an environment for an observability rebuild, the first week is rarely about adding dashboards. It’s about deleting alerts. On the cluster described above we retired 57% of the existing alert rules in the first sprint, and the ones that survived got linked runbooks, SLO-aligned thresholds, and clear ownership.

But we start by asking the opposite question: what should be paging and isn’t? Run absent() over every critical component before you delete a single rule. A cluster with 200 noisy alerts and one silent blind spot will hurt you through the blind spot.

The result is a quieter inbox, faster incident response, and a team that actually trusts the signals coming out of their monitoring stack.

If that’s the kind of work you’re looking for, get in touch.