Infrastructure · Monitoring & alerting
Know before it breaks.
From infrastructure failures to predictive capacity warnings, Office Unbound is being designed to keep administrators informed with real-time monitoring, configurable alerts, automated recovery and escalation that follows the rules you set.
Project status Planned. The Infrastructure Control Server is in the roadmap’s Infrastructure Operations track; nothing described here has been built.
The right notification, through the right channel, to the right person.
Monitoring that produces a hundred alerts a day trains people to ignore them. Monitoring that stays silent when the mail server is down is worse. The ICS is designed around the whole alert lifecycle, not just message delivery: what counts as a problem, how serious it is, who needs to know, how they are told, what happens if nobody responds, and what was done about it.
It is also designed to keep working in the situation where alerts matter most: when the platform itself, including its own mail, is failing.
Proposed interface concept Not released · example data · subject to change
CPU, % of 64 coresMemory, % of 512 GBAlert threshold 85%
Peak CPU 63% yesterday afternoon; the rise 2 hours ago is the nightly backup. Memory steady at 76–79%, no swap.
- Mail · IMAPLogin test, mail-01OK 140 ms
- Mail · SMTPSubmission on port 587, mail-02Failing · INC-2047
- Mail · queueOutbound queue12 messages
- Identity / SSOOIDC sign-in and token testOK 210 ms
- FilesWebDAV 1 MB upload and downloadOK 380 ms
- DocumentsOpen a sample documentOK 1.2 s
- CalendarCalDAV queryOK 95 ms
- DNSns1 and ns2 serials matchOK
- TLSNext certificate expiry23 days · auto-renew
| Host | CPU | Memory | Disk latency | Network | Uptime |
|---|---|---|---|---|---|
| pve-a1 | 1.8 ms | 2.1 Gb/s | 41 d | ||
| pve-a2 | 1.6 ms | 1.7 Gb/s | 41 d | ||
| pve-a3 | 1.9 ms | 1.2 Gb/s | 12 d | ||
| pve-b1 | 9.4 ms | 0.8 Gb/s | 63 d | ||
| pve-b2 | 24 ms | 0.9 Gb/s | 63 d |
| Container | CPU | Memory | Restarts 24 h |
|---|---|---|---|
| ou-search-indexsearch-01 | 41% | 6.8 GB | 0 |
| ou-db-primarydb-01 | 18% | 11.4 GB | 0 |
| ou-docs-collaboradocs-01 | 12% | 7.9 GB | 0 |
| ou-files-appfiles-01 | 9% | 3.2 GB | 0 |
| ou-mail-imapmail-01 | 6% | 2.1 GB | 0 |
| ou-mail-smtpmail-02 | 0% | 0.2 GB | 3 · loop stopped |
Mail serviceSMTP failing on mail-02, IMAP healthy
- Databasedb-01 primary · replica in sync
- DNSMX, SPF and DKIM resolve
- IdentityToken test OK
- Storage · mail-02 spool volume98% full · likely cause
- Storage · ceph-bWrite latency 24 ms
Alerts from services that depend on Mail are held back while this incident is open.
OutageDegradedPlanned maintenance
- MailOutage · Sat 02:14, ongoing
- IdentityOutage · Thu 16:05, 2 min, recovered automatically
- Files and syncDegraded · Mon 09:40, 18 min
- DocumentsPlanned · Tue 22:00, 25 min
- StorageDegraded · Sat 01:52, ongoing
Monitoring and observability
Metrics, health checks and history for each layer, collected by established open-source components rather than a monitoring engine written from scratch. Metrics and alert-routing projects such as Prometheus, Alertmanager, Grafana and Zabbix are candidates under evaluation.
- Concept An idea or design direction; not yet committed to the roadmap
- Under evaluation Being assessed; no choice made yet
- Planned Committed to the roadmap; work not started
- In development Being worked on now
- Implemented Code exists in a public repository
- Tested Passed published tests
- Production-ready Shipped in a supported release
| Capability | What it means | Status |
|---|---|---|
| Infrastructure metrics | CPU, memory, disk, network and storage for hosts, VMs and containers, kept as history. | Planned |
| Application health checks | Synthetic tests that use each service the way a user would: sign in, send, upload, open. | Planned |
| Dependency map | Which services rely on which, so the root cause and its downstream effects are shown together. | Planned |
| Outage timeline | When a problem started, what changed around it and when service returned. | Planned |
| Thresholds and baselines | Warning and critical thresholds per metric, and comparisons with normal behavior for that time of day. | Planned |
| Metrics and alerting engines | Established open-source components for collection, storage and routing; specific projects not yet chosen. | Under evaluation |
What can raise an alert
The planned alert conditions, by category. Every threshold is intended to be adjustable per service, host or cluster.
| Category | Conditions |
|---|---|
| Host and infrastructure | Host offline; high CPU or RAM; memory exhaustion; CPU oversubscription; disk space thresholds and storage growth; disk latency and IOPS; disk failures and SMART warnings; ZFS pool degradation; RAID failures; network congestion, packet loss and interface failures; cluster health problems and redundancy loss; VM crashes and unexpected reboots |
| Applications | Service failures; unhealthy containers; API failures and HTTP errors; database connectivity and replication problems; mail delivery failures and queue buildup; spam filtering failures; file synchronization problems; authentication failures; chat and meeting failures; telephony failures; certificate expiration; DNS problems |
| Security | Repeated login failures; suspicious administrator access; unauthorized configuration changes; unexpected service changes; critical vulnerabilities; unsupported software; expiring credentials; failed security scans |
| Backups and recovery | Backup failures and missed backups; outdated recovery points; replication failures; backup storage exhaustion; restore verification failures; unhealthy disaster-recovery targets |
| Maintenance | Security and critical patches available; unsupported versions; failed updates and failed rollbacks; maintenance overruns; compatibility warnings |
| Predictive | Projected storage exhaustion; insufficient resources for forecast growth; capacity thresholds approaching; resource contention trends; redundancy risks; scaling requirements (see capacity) |
Five severity levels
Each level has its own notification behavior. The defaults below are examples; every one is configurable.
| Severity | Typical meaning | Example default behavior |
|---|---|---|
| Informational | Something happened that is worth recording | Dashboard and history only |
| Notice | Worth a look during working hours | Dashboard, plus a daily summary by email |
| Warning | Will become a problem if left alone | Notify the owning team; respect quiet hours |
| Critical | Service degraded or failing now | Notify on-call immediately; repeat until acknowledged; escalate |
| Emergency | Major outage, data at risk, or security incident | All configured channels to on-call and a second administrator; ignore quiet hours |
Proposed interface concept Not released · example data · subject to change
- CriticalINC-204702:14 Outbound mail relay failing on mail-02 AcknowledgedDO
- WarningINC-204601:52 ceph-b write latency above 20 ms OpenSR
- WarningINC-2044Fri 18:20 Cluster A: memory no longer survives a host loss OpenDO
- NoticeINC-2043Thu 16:05 Identity container restarted, recovered ResolvedUnassigned
- WarningINC-2042Wed 01:40 Database backup took twice as long as usual ResolvedSR
- InformationalINC-2041Tue 09:00 Mail service 1.14.3 security update available ClosedUnassigned
- EmergencyINC-20313 Nov Cluster B lost quorum after a rack power fault Resolved · 41 minDO
CriticalINC-2047Acknowledged
Outbound mail relay failing on mail-02
Mail service · container ou-mail-smtp · VM mail-02 on pve-a2 · open 27 min
- 02:14:08DetectedSMTP submission check failed 3 times in a row on mail-02
- 02:14:20DiagnosedDatabase, DNS and identity healthy · spool volume 98% full
- 02:14:35Automated restart attemptedPolicy allows 3 container restarts in 15 minutes
- 02:16:10Verification failedCheck still failing after 3 restarts · restart loop stopped
- 02:16:15Pushover and email to D. OkaforPrimary on-call · both delivered
- 02:26:15Not acknowledged in 10 minutesSMS sent to D. Okafor · delivered 02:26:19
- 02:31:02Acknowledged by D. OkaforFrom the SMS link · escalation timer keeps running until resolved
- 02:36:40Volume expansion requestedExpand mail-02 spool by 20 GB · waiting for a second approver
DODana OkaforAssigned · primary on-call
- 1Pushover and email02:16
- 2SMS after 10 min02:26
- 3S. Reyes, secondary02:46 if unresolved
- Restart ou-mail-smtp, 3 timesAutomatic · verification failed
- Restart loop stoppedPolicy · prevents flapping
- Expand spool volume +20 GBNeeds 1 more approvalReview
| Channel | Recipient | Sent | Status |
|---|---|---|---|
| PushoverD. Okafor · phone | D. Okafor · phone | 02:16 | Delivered |
| Emaildana@ridgeline-it.example | dana@ridgeline-it.example | 02:16 | Delivered |
| SMS+1 ··· ··· 0142 | +1 ··· ··· 0142 | 02:26 | Delivered |
| Webhook#ops-alerts chat channel | #ops-alerts chat channel | 02:16 | Accepted |
| PushoverS. Reyes · next level | S. Reyes · next level | 02:46 | If unresolved |
Delivered through an independent provider — not the platform’s own mail. Alerts still reach people when Office Unbound mail is the thing that failed.
Notification channels
Planned channels, chosen per severity, team and person; voice calls are a later idea. Integrations are intended to avoid lock-in to any single provider where practical.
| Channel | Notes | External account or fees |
|---|---|---|
| Through the platform’s mail or an independent outbound relay, so alerts still go out when platform mail is down | May apply (external relay) | |
| SMS | Through configurable SMS providers, not one fixed gateway | May apply |
| Pushover | Push notifications with priority and acknowledgment | May apply |
| Mobile push | Notifications to an administrator’s phone | May apply (push service) |
| Browser | Notifications while the ICS console is open | No |
| Dashboard | The alert center in the ICS console | No |
| Webhooks | Any system that accepts an HTTP request, including ticketing and automation tools | No |
| Slack | Messages to a channel | May apply |
| Microsoft Teams-compatible integrations | Messages to a Teams channel through its incoming integrations | May apply |
| Discord | Messages to a server channel | May apply |
| Incident-management services | Optional hand-off to an external on-call platform the operator already uses | May apply |
| Voice call | Automated phone calls as a later escalation step Concept | May apply |
Third-party providers may require their own accounts, subscriptions or per-message fees, set by those providers.
Alert lifecycle and escalation
The planned rules for what an alert does after it fires. Escalation timing, severity, recipients and automatic actions are all designed to be configurable.
Rules
Custom thresholds; service-specific rules; host and cluster rules; per-tenant policies where authorized.
Routing
Team-based routing; personal notification preferences; on-call rotations; quiet hours.
Noise control
Deduplication; correlation of related alerts; dependency-aware suppression, so one storage failure raises one incident rather than forty; grouping; suppression during scheduled maintenance.
Response
Acknowledgment; snoozing; escalation rules; repeat notifications for critical events; resolution notices.
Incidents
Ownership; a timeline of everything that happened; incident reports; optional ticket creation in clientst0r.
Accountability
Notification history; delivery tracking where the provider supports it; test notifications; an audit of every escalation.
Example: mail service failure
One possible policy. Recovery is attempted first, within what the policy allows; people are notified when it fails; escalation continues until someone takes ownership.
Reading the example
- Step 1 uses an application-level check, so a mail server whose process is running but whose logins fail is still caught.
- Step 3 runs only if the operator has authorized that action for that service. In observe-only mode it becomes a recommendation.
- Step 4 re-runs the same check. A restart that did not fix the problem is not reported as a recovery.
- Steps 5 to 7 use channels that do not depend on the failing mail service.
- The waiting periods, channels and recipients are examples. A business with one administrator would configure this differently from an MSP with an on-call rotation.
Independent alert delivery
If Office Unbound’s mail servers fail, the monitoring system still has to reach an administrator. Alerts are designed to go out through notification providers that do not depend on the platform’s own mail: an SMS provider, Pushover, a push service, a webhook, or an outbound mail relay outside the platform.
The operator chooses which independent channel to configure. At least one is recommended for every deployment.
Watching the watcher
A controller that has stopped cannot report that it has stopped. The ICS is designed to send a regular heartbeat to an external monitor running somewhere else: another site, another provider, or a dead-man check service. If the heartbeats stop, that monitor raises the alarm through an independent channel.
Production services keep running while the controller is down; what is lost is visibility, which is why the heartbeat matters. See ICS design principles.
How does your on-call work today?
Which channels, how many people and how escalation is decided: describing it helps the project design alert policies that fit real teams.