Infrastructure Control Server · Alert center
From detection to acknowledgment
Proposed interface concept Not released. A design for discussion, drawn in HTML and CSS; not a screenshot of working software.
Incidents by severity, with an open incident showing what was detected, the recovery attempted, which notifications went out through which channel, who has it, and the controls to acknowledge, snooze, escalate or resolve. The figures shown are example data.
Proposed interface concept Not released · example data · subject to change
- CriticalINC-204702:14 Outbound mail relay failing on mail-02 AcknowledgedDO
- WarningINC-204601:52 ceph-b write latency above 20 ms OpenSR
- WarningINC-2044Fri 18:20 Cluster A: memory no longer survives a host loss OpenDO
- NoticeINC-2043Thu 16:05 Identity container restarted, recovered ResolvedUnassigned
- WarningINC-2042Wed 01:40 Database backup took twice as long as usual ResolvedSR
- InformationalINC-2041Tue 09:00 Mail service 1.14.3 security update available ClosedUnassigned
- EmergencyINC-20313 Nov Cluster B lost quorum after a rack power fault Resolved · 41 minDO
CriticalINC-2047Acknowledged
Outbound mail relay failing on mail-02
Mail service · container ou-mail-smtp · VM mail-02 on pve-a2 · open 27 min
- 02:14:08DetectedSMTP submission check failed 3 times in a row on mail-02
- 02:14:20DiagnosedDatabase, DNS and identity healthy · spool volume 98% full
- 02:14:35Automated restart attemptedPolicy allows 3 container restarts in 15 minutes
- 02:16:10Verification failedCheck still failing after 3 restarts · restart loop stopped
- 02:16:15Pushover and email to D. OkaforPrimary on-call · both delivered
- 02:26:15Not acknowledged in 10 minutesSMS sent to D. Okafor · delivered 02:26:19
- 02:31:02Acknowledged by D. OkaforFrom the SMS link · escalation timer keeps running until resolved
- 02:36:40Volume expansion requestedExpand mail-02 spool by 20 GB · waiting for a second approver
DODana OkaforAssigned · primary on-call
- 1Pushover and email02:16
- 2SMS after 10 min02:26
- 3S. Reyes, secondary02:46 if unresolved
- Restart ou-mail-smtp, 3 timesAutomatic · verification failed
- Restart loop stoppedPolicy · prevents flapping
- Expand spool volume +20 GBNeeds 1 more approvalReview
| Channel | Recipient | Sent | Status |
|---|---|---|---|
| PushoverD. Okafor · phone | D. Okafor · phone | 02:16 | Delivered |
| Emaildana@ridgeline-it.example | dana@ridgeline-it.example | 02:16 | Delivered |
| SMS+1 ··· ··· 0142 | +1 ··· ··· 0142 | 02:26 | Delivered |
| Webhook#ops-alerts chat channel | #ops-alerts chat channel | 02:16 | Accepted |
| PushoverS. Reyes · next level | S. Reyes · next level | 02:46 | If unresolved |
Delivered through an independent provider — not the platform’s own mail. Alerts still reach people when Office Unbound mail is the thing that failed.
Shape this before it is built
This screen is a proposal. MSP design partners review concepts like this one against how they actually run customers, and say what is missing.