Audit your suppression graveyard before a live page dies in it

A silence set during a maintenance window in 2023 is still active. The host it matched was decommissioned last spring. Every time that host’s replacement throws a warning, the old matcher catches it and drops it, and nobody sees a thing. No one set out to hide a live alert. Someone set a bounded silence for a real reason, forgot to expire it, and the environment moved on around it.

That is the whole failure mode. This installment of the checks franchise is the audit that finds those rows in your own stack.

What you’re dealing with

Every suppression you own, silence, downtime, maintenance window, or acknowledgement, got created for a bounded reason and then nothing forced anyone to revisit it. Over months they pile into an inventory nobody reads. The clutter is not the risk. The risk is a genuine alert firing into an old silence and dying unseen.

A practitioner writeup on Alertmanager hygiene treats a repeatedly-renewed silence as a code smell: the noisy condition was never fixed, just perpetually re-hidden. Upstat’s guide calls these stale suppression rules and recommends treating any suppression tied to a window that closed more than 24 hours ago as suspect by default.

The driver is alert fatigue, which makes muting cheaper than fixing. incident.io’s SRE writeup describes teams mentally discounting alerts when only a fraction are actionable, and that discounting becoming permanent. The Google SRE book frames what erodes: pages have to be actionable and novel, because responders burn out fast. incident.io’s on-call guidance tells teams to cover “what’s muted, why, when it expires” at every shift handoff, which is only necessary because it routinely gets skipped.

One honesty check before the commands. There is no published postmortem naming a stale silence as the root cause of a missed alert. That is probably a reporting gap, not proof the failure mode is rare. The closest verifiable analogue is GitLab’s January 2017 database-loss postmortem: not a suppression rule, but the same shape. Backup-failure notifications were rejected by a DMARC misconfiguration, and in GitLab’s words, “there was no ownership, as a result nobody was responsible for testing this procedure.” Read it as a parallel, not a match. Separately, the 2025 SANS detection survey found 73% of organizations name false positives their top detection challenge. That measures the scale of false-positive fatigue, not suppression hygiene itself.

The three flags that find bad rows

Any one of these flips a suppression into the cleanup pile.

• NO_EXPIRY — the suppression has no real end. Alertmanager and Grafana have no null-end state, so the tell is a far-future date, not an absent one. Datadog is the opposite: on the current v2 schema, end: null literally means begins now and never ends.
• NO_OWNER — creator is blank, a deleted account, or a generic service account with no linked ticket.
• DEAD_TARGET — the matcher or scope no longer resolves against live host, service, or monitor inventory.

How to check, tool by tool

Every command below is illustrative. Test it against your own endpoint and version before you trust the output.

Prometheus Alertmanager. amtool lists active silences by default; the API exposes every field for the far-future check.

# CLI: active silences (add --expired to flip; --within=48h for "expiring soon")
amtool silence query --alertmanager.url=http://localhost:9093
amtool silence query --created-by=svc-deploy -q      # IDs only, filtered by creator

# REST API — active silences with the fields that matter
curl -s http://localhost:9093/api/v2/silences \
  | jq '.[] | select(.status.state=="active") | {id, endsAt, createdBy, comment}'

Bad answer: an endsAt years out, or a blank createdBy. There is no null-end here, so a distant date is the NO_EXPIRY tell. Note that expired silences auto-delete five days after expiry, but the permanent debt lives in the config-level constructs. Inhibition rules have no time dimension at all; mute time intervals are recurring windows with no expiry on the rule itself. Both persist until someone edits the static config, and neither shows up in the silence list above.

Grafana Alerting. Embedded Alertmanager, identical schema under Grafana’s path prefix.

curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  "https://<grafana-host>/api/alertmanager/grafana/api/v2/silences" \
  | jq '.[] | select(.status.state=="active") | {id, endsAt, createdBy, comment}'

Same endsAt / createdBy / status.state. Swap grafana for the datasource UID to hit an externally-connected Alertmanager.

Datadog. end: null is the literal indefinite flag.

curl -s "https://api.datadoghq.com/api/v2/downtime?current_only=true&include=created_by" \
  -H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
  | jq '.data[] | {id, status: .attributes.status,
                   end: .attributes.schedule.current_downtime.end,
                   scope: .attributes.scope,
                   creator: .relationships.created_by.data.id}'

Bad answer: end is null. relationships.created_by.data.id is your owner check; scope is what you cross-check against live monitors for DEAD_TARGET.

PagerDuty. The one tool where suppressions can’t be indefinite by schema. Maintenance windows are always bounded and the service auto-exits when the window ends.

curl -s -G "https://api.pagerduty.com/maintenance_windows" \
  -H "Authorization: Token token=$PD_TOKEN" --data-urlencode "filter=ongoing" \
  | jq '.maintenance_windows[] | {id, start_time, end_time, description,
                                  created_by: .created_by.summary}'

start_time and end_time are always present, so a “never-ending” window here means someone keeps recreating it. Owner and dead-target checks carry the weight. The real config-level debt lives in event-orchestration suppress rules, which drop matching events silently with no expiration until a human edits the ruleset.

Icinga2. Downtimes require a bounded end_time; the never-expires pattern lives in sticky acknowledgements, which clear only when the object returns to OK/UP.

# Scheduled downtimes
curl -k -s -u api-user:pass -H 'Accept: application/json' \
  'https://icinga-host:5665/v1/objects/downtimes' \
  | jq '.results[].attrs | {end_time,start_time,author,comment,fixed}'

# Acknowledgements with NO expiry (from the official API docs' worked example)
curl -k -s -u root:icinga -H 'Accept: application/json' \
  -H 'X-HTTP-Method-Override: GET' -X POST \
  'https://icinga-host:5665/v1/objects/comments' \
  -d '{"joins":["service.name","service.acknowledgement","service.acknowledgement_expiry"],
       "attrs":["author","text"],
       "filter":"service.acknowledgement!=0 && service.acknowledgement_expiry==0","pretty":true}'

The second query returns every acknowledgement with acknowledgement_expiry==0, which is the NO_EXPIRY set that never clears on its own.

The audit table

Dump one row per active suppression into the same schema across every tool, then flag it.

Column	Field per tool
suppression_id	tool’s native ID
tool	alertmanager / grafana / datadog / pagerduty / icinga
matcher_or_scope	matchers[] / scope / services[] / host+service filter
created_at	createdAt / created / start_time / entry_time
creator_owner	createdBy / creator_id / created_by / author
expires_at	endsAt / end / end_time, or literal NEVER
status_state	tool-native
linked_ticket	present? parsed from comment / message / description
target_still_exists	yes / no, cross-checked against current inventory

A row is a cleanup candidate if NO_EXPIRY (expires_at null or a far-future placeholder), NO_OWNER (creator_owner blank, deleted, or a service account with no linked_ticket), or DEAD_TARGET (matcher_or_scope no longer resolves) is true. Done means every remaining active row has a real expiry, a named owner, and a target that still exists.

What to change so it stops piling up

The audit clears today’s debt. Three config changes keep it from refilling.

1. Default short TTL. Cap every suppression at hours to a couple of days unless someone actively extends it. OneUptime’s guide recommends flagging silences past roughly 24 hours for review and keeping them as short as possible. No platform enforces this for you; Grafana’s docs state plainly there is no maximum silence duration.
2. Required owner and linked reason. Grafana’s best-practices guide says every alert must have an owner. Extend that to suppressions: an owner plus a ticket ID for anything outliving the short default.
3. Standing audit cadence. Schedule a recurring review that clears the NO_EXPIRY / NO_OWNER / DEAD_TARGET rows. Quarterly is defensible, more often at high volume. CIS Control 8, Safeguard 8.11 requires reviewing audit logs weekly; it governs log review, not suppression review, so read it as backing the discipline by implication, not mandating it.

Track this out of band. PagerDuty’s own docs note its audit trail does not capture maintenance-window changes at all, so the native history is insufficient and suppression state belongs in a ticket, IaC, or a spreadsheet-of-record.

The NO_EXPIRY rows are embarrassing. The DEAD_TARGET rows are the ones that cost you an incident, because they are still catching real alerts for infrastructure that was supposed to be gone. Run the enumeration this week. Those rows have been there longer than you think.

Sources

• brtkwr — Silencing alerts properly in Alertmanager — 2025-12-10
• Upstat — Alert suppression best practices — 2026-01-26
• incident.io — SRE alerting best practices — 2026-03-26
• Google SRE Book — Practical Alerting — foundational
• incident.io — On-call best practices — 2026-02-27
• GitLab — Postmortem of database outage of January 31 — 2017-02-10
• Stamus Networks — 2025 SANS Detection & Response survey summary — 2025-12-16
• Grafana — Configure silences — current
• Datadog — Downtimes — current
• amtool.1 manpage — current
• Prometheus — Alerting configuration — current
• PagerDuty — Maintenance Windows — current
• PagerDuty — Event Orchestration — current
• Icinga 2 — Object types — current
• Icinga 2 — API docs — current
• OneUptime — Grafana silence rules — 2026-01-30
• Grafana — Alerting best practices — current
• CIS Controls Assessment Spec v8.1 — Control 8 — current
• PagerDuty — Audit Trail Reporting — current