PSI Alerting Standard

Canonical, normative. Every PSI system that raises an alert — web app, ETL job, PLC watchdog, scheduled script, CI/CD pipeline, IoT sensor — conforms to this page. If you are wiring an alert and this page and any other doc disagree, this page wins.

The mechanics of how to send live in Teams Notifications. This page governs whether to send, what the alert must say, and who owns it afterwards.

1. The one-sentence rule

An alert is a request for a specific human to do a specific thing, right now.

Everything in this standard follows from that sentence. If nobody needs to do anything, it is not an alert — it is a log line, a metric, or a digest. If the recipient can’t tell what to do from the message alone, the alert has failed even if it was delivered.

The three failures we are engineering against

Failure	What it looks like at PSI	What this standard does about it
Cryptic alerts	`Alert: SYNC ERR 0x8004` at 2am with no link, no owner, no next step	§4 makes `impact`, `action`, and `runbookUrl` required fields the ingest endpoint rejects without
Alert fatigue	A channel nobody reads because 90% of its traffic is noise	§3 severity gate + §7 noise budget — a source that over-pages is auto-demoted
Orphaned alerts	An alert firing for 8 months that no current employee understands	§6 ownership — every alert names an owner and a runbook, or it is deleted

2. What qualifies as an alert — the four-question gate

Before wiring anything, answer all four. A “no” at any step means it is not a Teams alert.

#	Question	If no →
1	Does a human have to do something? Not “would be nice to know” — has to act.	Log it / emit a metric. Put it on a dashboard, not in a chat.
2	Does it have to happen now? Would waiting until the next working day cause real harm?	File a ticket or add it to a digest. Never a page.
3	Is the person receiving it the one who can fix it?	Route it to whoever can, or don’t send it. Broadcasting to a channel of bystanders is noise.
4	Is the condition real and specific? Would this fire on a transient blip, a retry that later succeeded, or a threshold nobody calibrated?	Add hysteresis / retry-tolerance first. A flapping alert trains people to ignore the channel.

Alert on symptoms, not causes. “Line 3 operators cannot scan travelers” is an alert. “CPU on prgjsmes-prod is at 88%” is a dashboard metric — unless high CPU has been proven to precede an outage, in which case alert on the outage symptom and put CPU in the detail.

Things that are explicitly NOT alerts

Job success notifications (“nightly sync completed”). Alert on failure and on silence; success is a metric.
Anything a person cannot act on before the next business day.
Duplicate coverage — if Azure Monitor already pages for it, do not add a second path that pages for the same thing.
Debug output, stack traces, or raw provider payloads as the primary message body. Those belong in detail or behind a link.

3. Severity — four tiers, and what each one is allowed to do

Severity is not a mood. It is a contract about who gets woken and how fast we expect a response.

Tier	Means	Channels	Response expectation	Runbook
`critical`	Production stopped, safety risk, data loss, or customer-visible outage. Money or people are affected now.	Teams + voice call (press-1 ack) + escalation	Ack within 5 min, 24/7	Required
`high`	Degraded and getting worse; will become `critical` if untouched. Single machine/line down, ETL stale past SLA.	Teams with on-call @mention	Ack within 1 hour, business hours	Required
`warning`	Needs a human this week. Anomaly, growing backlog, expiring credential, budget drift.	Teams channel post	Triaged within 5 business days	Recommended
`info`	Situational awareness only.	Digest only — never its own message	None	Not required

Rules that bind these tiers:

critical wakes people. It requires central approval. Self-service rule creation is capped at warning + Teams-only (see Consumer onboarding). Waking someone at 2am is a decision with a human in the loop, permanently.
The sender proposes severity; the routing rule caps it. The system raising the alert knows the most about how bad it is, so it declares severity in the envelope. The rule may lower it or restrict channels — it may never raise a self-service source to critical.
info never gets its own message. If it were worth interrupting someone for, it wasn’t info.
A tier with no runbook cannot be critical or high. Ingest enforces this (§4).
Expiry credentials, certs, and licences alert at warning 30 days out and high at 7 days. Never let one become critical — that means we already failed.

Response windows — two profiles, not a per-source setting

“Business hours” means two different things at PSI: when the company is staffed, and when the plant is running. Rather than one wrong global rule or thirty unmaintained per-source ones, a source picks one of exactly two profiles:

Profile	Window	For
`business` (default)	Mon–Fri 07:00–17:00 ET	IT systems, web apps, ETL, integrations — anything a day-shift responder owns
`production`	Whenever the plant is running (shift-dependent)	Shop-floor and machine-adjacent systems where a 2nd-shift failure is a real failure

Two profiles is a closed set on purpose — it stays maintained, and the name tells the responder what to expect. A high raised outside its window queues and delivers when the window opens; it does not evaporate. Anything that genuinely cannot wait for the window is not high — it is critical, which is precisely what the tier boundary is for.

4. The Alert Envelope — the required format

Every alert is a versioned JSON envelope, not a blob of text. The transport (POST /api/alerts/inbound) validates it and rejects non-conforming alerts with 400 and a reason. This is the mechanism that makes “no cryptic alerts” real instead of aspirational — you cannot ship one past the door.

{
  "v": 1,
  "source":     "psi-datasync",              // registered system id — owns the alert (§6)
  "severity":   "high",                       // critical | high | warning | info
  "title":      "Customer file sync stalled for 3 hours",   // plain language, ≤120 chars
  "impact":     "Engineering can't see new customer drawings on \\\\DFS\\DATA; 14 files queued.",
  "action":     "Check the Egnyte token in ps-certificates-kv, then restart psi-datasync.",
  "runbookUrl": "https://wiki.progressivesurface.com/development/datasync-runbook#sync-stalled",
  "dedupeKey":  "psi-datasync/egnyte-stp/stalled",           // stable identity of the CONDITION
  "detail":     "Last successful sync 2026-08-10T03:12Z. Last error: 401 from Egnyte token refresh.",
  "links": [
    { "label": "Live dashboard", "url": "https://..." },
    { "label": "Log",            "url": "https://..." }
  ],
  "env":           "prod",                    // prod | test — non-prod never pages
  "firstSeenUtc":  "2026-08-10T03:12:00Z",
  "correlationId": "…",                       // optional: ties related alerts together
  "metric":  { "name": "minutes_since_sync", "observed": 187, "threshold": 60 }   // optional
}

The four fields that kill cryptic alerts

Field	Answers	Rejected if
`title`	What is wrong?	Empty, >120 chars, a bare error code (`0x8004`, `E_FAIL`), a stack trace, or a generic non-title (`Error`, `Failed`, `Alert`, `Exception`)
`impact`	Why should I care? Who is affected?	Missing. Must be in business terms — a person, line, job, or customer — not a component name
`action`	What do I do first?	Missing. Must be an imperative first step, not “investigate” or “check the logs”
`runbookUrl`	Where’s the full procedure?	Missing when severity is `critical` or `high`. Must resolve (checked at rule-registration time)

The 2am test. Read only title, impact, and action aloud. If a competent colleague who has never seen this system could take the first correct step from those three lines, the alert passes. If they’d have to ask someone, it fails. This test is the standard — the field list is just how we make it enforceable.

Rendering — one house format, every channel

Consumers do not hand-write HTML. The envelope renders identically everywhere so recipients learn one shape:

🔴 CRITICAL · psi-datasync                         ← severity + owning source
Customer file sync stalled for 3 hours             ← title

Impact   Engineering can't see new customer drawings on \\DFS\DATA; 14 files queued.
Action   Check the Egnyte token in ps-certificates-kv, then restart psi-datasync.
Runbook  ▸ DataSync → Sync stalled
Detail   Last successful sync 03:12Z. Last error: 401 from Egnyte token refresh.

Live dashboard · Log · Alert #4471          Raised 06:12 · Ack (press 1 on the call)

Voice renders severity + source + title + action only — a phone call cannot carry detail. SMS renders severity + title + runbook short-link.

Never send an alert whose body is a raw provider payload, a JSON dump, or a stack trace. Those go in detail (truncated, with a link to the full log) — never in title or impact.

5. Lifecycle — alerts must close

An alert that never resolves is a scary message that ages into wallpaper. Every alert has a lifecycle, and the source that raised it is responsible for clearing it:

raised → notified → acked → resolved
                 ↘ suppressed (cooldown)     ↘ expired (auto-close, 24h)

Transition	Who triggers it	How
`notified`	Engine	Fan-out completed on at least one channel
`acked`	Responder	Press 1 on the voice call, or the Ack button on the Teams card
`resolved`	The source system	`POST /api/alerts/resolve` with the same `dedupeKey`
`expired`	Engine	Auto-close after 24h with no activity, recorded as `expired` not `resolved`

Recovery is a first-class event. When a source recovers it must say so, and the original Teams message is updated in place to ✅ RESOLVED after 14m rather than leaving the red one sitting there. A monitoring system that only knows how to say “broken” teaches people to distrust it.

Escalation applies to critical only: unacknowledged after 5 minutes → next person in the on-call order gets the voice call. Everyone in the rotation is paged at most once per alert.

6. Ownership — no unowned alerts, ever

Every source is registered with a named owner before its first alert is accepted. An alert whose owner has left PSI, or whose runbook 404s, is a defect — treat it like a failing test.

Obligation	Owner must…
Runbook	Keep `runbookUrl` resolving and accurate — it is read at 2am by someone who isn’t them
Noise	Keep the source inside its noise budget (§7)
Recovery	Emit `resolve` when the condition clears
Review	Confirm the source’s rules are still wanted at the quarterly review

Who is on call today — there is no rotation

Stated plainly so nobody plans around something that doesn’t exist: on-call is currently a fixed list, not a rotation. GroupMembers.OnCallOrder is a static order, so escalation always walks the same people in the same sequence. Today the primary recipient for essentially everything is the Director of IT, who is the platform’s main customer while it is being dialled in.

Two consequences that the design must respect until a rotation exists:

Escalation must not assume more than one person. With a single pageable recipient, “escalate to the next tier” has nowhere to go — so it re-pages the same person rather than silently doing nothing. A page that quietly stops is the worst possible failure mode.
Service and test accounts do not belong in a paging group. They consume an escalation tier and answer nothing.

A real weekly rotation (primary/secondary, with swap overrides) is a near-future addition, not a gap in this standard. When it lands, this section is what changes.

Routing is declared as code, in the source’s own repo (psi-alerts.yml), not typed into an API by hand. Routing then gets reviewed in a PR like everything else, and drift is visible:

# psi-alerts.yml — lives in the alerting system's repo, applied by CI
source: psi-datasync
owner:  adevereaux@progressivesurface.com
runbookBase: https://wiki.progressivesurface.com/development/datasync-runbook
rules:
  - name: Egnyte sync stalled
    match:    { dedupeKeyPrefix: "psi-datasync/egnyte-stp/" }
    severity: high            # capped: this source is not approved for voice
    channels: [teams]
    group:    IT On-Call
    noiseBudget: 3/week

Quarterly alert review

Once a quarter, every source’s owner reviews, in the Alert Console (§8):

Alerts raised, acked, and auto-expired (expired = nobody acted = probably not an alert).
Alerts with no runbook click-through (the runbook isn’t useful, or the alert isn’t).
Top noisy conditions.
Delete or demote anything that didn’t earn its place. Deleting alerts is a success metric, not a regression.

7. Noise budget — the system defends itself

Alert fatigue is not a discipline problem, it’s a design problem. So the platform measures it.

The platform counts and reports noise; it does not automatically silence anything. A human decides whether a chatty source gets turned down. Automatic demotion is deliberately not built yet: with one source routing through the engine and one recipient, there is no baseline to set a threshold from, and a demotion firing wrongly would teach everyone to distrust the platform before it has earned any trust. Revisit once there is a quarter of real data — the threshold then comes from measurement rather than from a guess.

Each source declares a noise budget; the console shows actual volume against it per source, split by severity.
Breaching it raises a warning to the source’s owner and shows red on the scorecard. Nothing is demoted or suppressed on the platform’s own authority.
Volume is counted in distinct conditions (dedupeKey), not messages — otherwise storm damping and the budget double-count one incident.
Storm damping: >10 alerts from one source in 5 minutes collapse into one “psi-datasync raised 37 alerts in 5 min” summary with a link to the console. One incident, one page. (This is automatic — it changes presentation, not priority, and nothing is lost.)
Cooldown: the same dedupeKey re-pages at most every 15 min while unacked, 60 min once acked.
Maintenance windows: declared in the console or psi-alerts.yml; alerts are recorded but not delivered, and the window’s owner is named on the suppressed alert.

8. Every alert is logged — the Alert Console

All alerts are persisted, whether or not anyone was notified — including suppressed, storm-damped, maintenance-window, and rule-matched-nothing alerts. An alert that reached nobody is exactly the one you need to find later.

The record lives in the alerts DB (procserv-proddata) and is surfaced by the Alert Console — the single web UI for everything on this page:

Surface	Answers
Live board	What is on fire right now, who acked it, how long it’s been open
History	Full searchable log — every alert, delivery attempt, and per-recipient channel outcome
Source health	Per-source volume, noise-budget standing, MTTA/MTTR, auto-expiry rate
Rules & routing	Read-only view of what each `psi-alerts.yml` resolved to; conflicts flagged
On-call	Groups, escalation order, maintenance windows
Quality scorecard	% with runbooks · % acked vs auto-expired · % auto-resolved · noisiest conditions

“Was an alert sent?” must always be answerable from the console, never from someone’s memory of a Teams channel. Teams is a delivery channel; it is not the record.

The console is a PSI web app built on the platform SDK — see PSI Notify Bot for its place in the architecture.

9. How to send — one contract, four clients

The envelope over HTTP is the contract. Client libraries are thin conveniences over it, so a new language never means a new format. Never hand-roll the token flow and payload again — three PSI consumers have already done that independently, in three languages, and they drifted.

Runtime	Use	Status
Node / TypeScript	`@progressivesurface/notify`	Planned — see §11
PowerShell	`PSI.Notify` → `Send-PsiAlert`	Module exists; envelope cmdlet planned
Python	`psi_notify`	Planned (csm-board has a partial port to harvest)
.NET	`PSI.Notify.Client`	Planned
CI/CD	`progressivesurface/notify-alert@v1` composite action	Planned
Anything else	`POST /api/alerts/inbound` with the envelope	Live

Every client offers the same three calls and nothing more: sendAlert(envelope), resolveAlert(dedupeKey), lintAlert(envelope).

lintAlert runs in your CI, not at 2am. POST /api/alerts/lint dry-runs the envelope: it returns the rejection reasons and the rendered message without delivering anything. Wire it into your test suite so a cryptic alert fails a PR instead of failing a responder.

10. Compliance checklist — wiring a new alert

11. Current state vs this standard

This standard is normative and current; the implementation is catching up to it. Live today: the ingest endpoint, rule matching, dedupe/cooldown, Teams + voice fan-out with press-1 ack, escalation, and the full Alerts/AlertDeliveries audit trail.

The gaps between the engine as built and this page — envelope validation, the resolve path, per-group Teams routing, sender-declared severity, noise budgets, the console, and the client packages — are specified in docs/alert-platform-design.md in the psi-notify-bot repo and tracked as GHE issues there.

Rebuild current state from living sources, never from this page: the repo’s BUILD_LOG.md, gh issue list --repo ProgressiveSurface/psi-notify-bot, and the psi-notify-listener Function App settings.

PSI Knowledge Base

Explorer

PSI Alerting Standard

PSI Alerting Standard

1. The one-sentence rule

The three failures we are engineering against

2. What qualifies as an alert — the four-question gate

Things that are explicitly NOT alerts

3. Severity — four tiers, and what each one is allowed to do

Response windows — two profiles, not a per-source setting

4. The Alert Envelope — the required format

The four fields that kill cryptic alerts

Rendering — one house format, every channel

5. Lifecycle — alerts must close

6. Ownership — no unowned alerts, ever

Who is on call today — there is no rotation

Quarterly alert review

7. Noise budget — the system defends itself

8. Every alert is logged — the Alert Console

9. How to send — one contract, four clients

10. Compliance checklist — wiring a new alert

11. Current state vs this standard

See Also

Graph View

Table of Contents

Backlinks