Incidents

An incident is the durable record of “something is broken.” Every page, every escalation, every status-page update hangs off an incident. They’re created two ways:

Auto: when a monitor linked to a service in auto mode starts failing, Site Qwality opens an incident with auto_created: true. A service has at most one open auto-created incident, and every other monitor of the service that fails joins it. Monitors with no service never open incidents. See auto-created incidents.
Manual: when you create one in the dashboard or via POST /incident, for events no monitor catches (“partial Stripe outage breaking webhooks,” “a cron job took down the queue”). See create manual incidents.

Either way the lifecycle is the same.

Anatomy of an incident

An incident has a title, a severity, two states (below), the monitors it is about, the people working it, an optional service, the status pages it is published on, and a timeline of updates. Status pages show its public_title, so an internal title like db-7 failover never reaches customers. Each update is public (on status pages) or internal (a note for your team). Every field is in the incident reference.

Status vs responder_status

An incident has two parallel states:

status is the external / customer-facing progress: “we’re still investigating,” “we know the cause,” “fix is rolled out, we’re watching.” It’s what shows on the status page, and only public updates and resolving move it.
responder_status is the internal / responder state: “no one’s on this,” “someone has it,” “we’re done.” It’s what drives escalation and ack tracking.

An incident can be status: monitoring and responder_status: acknowledged at once: customers read “we’re keeping an eye on it” while a responder still owns it.

Severity guidance

Severity	Use for
`minor`	Single non-critical endpoint flapping; degraded performance below SLA but functional.
`major`	Customer-facing feature broken for a subset of users; one region down with traffic still served from another.
`critical`	Full outage; data loss; security incident.

Severity sets the urgency of the page: critical and major incidents page at high urgency, minor at low, and each responder’s notification rules decide how each urgency reaches them. It does not choose a different escalation policy: the policy comes from the incident’s service. Severity also drives display priority on status pages and dashboards.

Raising an incident’s severity from minor to major or critical pages the current escalation level again. An acknowledged incident goes back to triggered, so an earlier acknowledgment does not silence the new page.

Lifecycle: a typical auto-created incident

HTTP check prod-api, linked to the API service in auto mode, flips from success to failure at 03:14:22.
Incident opens automatically with auto_created: true, status: investigating, responder_status: triggered, and the service’s default severity.
The service’s escalation policy pages the on-call through their notification rules. Separately, the check’s notification groups send their usual monitor alert.
The on-call engineer receives an SMS at 03:14:30. They open the dashboard and click Acknowledge at 03:15:10.
responder_status flips to acknowledged. Escalation stops. With no lead yet, the engineer becomes the incident’s lead.
They post an incident update at 03:18: status: identified, message "Database failover stuck. Manually promoting standby."
The fix lands at 03:24. The check goes green on the next tick, and prod-api records its recovered_at.
Every monitor on the incident has recovered, but the incident was acknowledged and the service uses the default auto-resolve setting, unless_acknowledged. It stays open for the engineer to resolve at 03:27: status: resolved, responder_status: resolved, resolved_at set.

Had nobody acknowledged it, the incident would have resolved itself at 03:24. If prod-api fails again within the service’s reopen window after that, the same incident reopens instead of a new one opening. If you resolve the incident yourself first, auto-resolution is skipped.

Acknowledgment

Acknowledging an incident is not resolving it. Ack stops escalation, but the incident stays open until someone resolves it.

The first person to acknowledge an incident that has no lead becomes its lead participant.

An incident that a monitor outage opened automatically is linked to that outage’s own acknowledgment. Acknowledging the outage acknowledges the incident if it is still triggered, and acknowledging the incident records the outage acknowledgment. Incidents you create by hand are never linked to an outage.

Working an incident with others

Participants. Record who is working the incident and in what role (lead, responder or comms), and page more people from a user, an on-call schedule or an escalation policy. See participants.
Affected monitors. An incident lists the monitors it is about, when each joined and recovered, and how badly each is affected. See affected monitors and impact.
Merging. Fold duplicate incidents into one. See merge incidents.
Moving. Change an incident’s service when it was filed under the wrong one. See move an incident to another service.