Infrastructure · Recovery & maintenance
Detect. Diagnose. Recover.
The Infrastructure Control Server is designed to find the cause of a failure, carry out the recovery actions an operator has approved, check that they worked, and escalate to a person when they did not. Updates follow the same principle: applied on the operator’s schedule, with checks before and after.
Project status Planned. The Infrastructure Control Server is in the roadmap’s Infrastructure Operations track; nothing described here has been built.
Recovery that knows its limits
Many failures in a self-hosted platform have a known fix: a container that has stopped, a service that needs a restart after a dependency comes back, a queue that needs flushing once DNS resolves again. Doing those things at 3 a.m. is work a system can do, as long as it does only what it has been allowed to, checks the result, and stops when the fix is not working.
Not every failure can be repaired automatically. A failed disk, corrupted data, a misconfiguration or a bug in a new release needs a person. The ICS is designed to recognize when automated recovery has not worked and hand the incident to an administrator with the diagnosis it has gathered.
Three automation modes
How much the ICS is allowed to change is the operator’s decision, and can differ by service, host or customer environment.
Observe only
Monitoring, alerts and recommendations. The ICS makes no changes to infrastructure.
Suits a new deployment, or an operator who wants to see what the ICS would do before trusting it.
Assisted operations
The ICS proposes a specific action, such as restarting the mail service, and an administrator approves it before it runs.
Keeps a person in every decision, with the diagnosis and the command prepared.
Policy-controlled automation
Specifically approved operations run automatically, within configured safety boundaries such as attempt limits and time windows.
Anything outside those boundaries falls back to assisted mode or escalation.
In every mode
Every action, proposed or taken, is recorded with its reason and result.
High-impact and destructive actions always respect approval, backup, health-check and rollback policies.
Planned recovery capabilities
- Concept An idea or design direction; not yet committed to the roadmap
- Under evaluation Being assessed; no choice made yet
- Planned Committed to the roadmap; work not started
- In development Being worked on now
- Implemented Code exists in a public repository
- Tested Passed published tests
- Production-ready Shipped in a supported release
| Capability | What it means | Status |
|---|---|---|
| Dependency-aware detection | A failure is traced to the component that caused it, so restarting mail is not attempted when the database underneath it is down. | Planned |
| Automatic diagnostics | Logs, recent changes, resource usage and dependent checks gathered when an incident opens. | Planned |
| Restart policies | Container and service restarts with limits on attempts and spacing between them. | Planned |
| Recovery playbooks | Predefined remediation actions per service, reviewed and approved by the operator before they can run. | Planned |
| Post-recovery verification | The same application-level check run again; a recovery counts only if the service works. | Planned |
| Restart-loop prevention | A service that fails again soon after each restart is stopped from cycling and escalated instead. | Planned |
| Escalation when recovery fails | Administrators notified with the diagnosis and what was already tried. | Planned |
| Incident history and analytics | What failed, how often, how it was resolved and how long recovery took. | Planned |
| Configuration drift alerts | Differences between the intended and actual configuration of a service, reported before they cause a failure. | Planned |
| Approved corrective maintenance | Fixes that need a maintenance window queued for one, rather than run immediately. | Planned |
It is not a promise of uptime, and it is not a substitute for redundancy, backups or an administrator. Automating a fix that hides a recurring fault can make the fault harder to see; this is why every recovery is recorded and repeated recoveries are reported as a problem in their own right.
Your infrastructure.
Your maintenance schedule.
With a cloud subscription, the vendor decides when application updates arrive. Office Unbound is designed so the business or its MSP decides when updates are applied: in which window, to which systems first, and after which checks.
The ICS is intended to do the preparation around that decision: find what has changed, classify it, check compatibility, back up, apply, verify and roll back where that is possible.
Deferring a critical security patch leaves a known vulnerability open. The ICS is designed to keep the severity of every pending update visible, raise critical patches as alerts, and show when software has reached the end of its supported life. Running supported versions is a requirement, not a preference.
Discover
Update discovery and container version inventory, with vulnerability advisories and security patch classification.
Assess
Software dependency analysis and compatibility checks against the rest of the deployment.
Schedule
Maintenance windows and customer-defined patch schedules, with staged update rings: test, pilot, broad.
Prepare
Pre-update backups or snapshots, and pre-update validation that the system is healthy before anything changes.
Apply and verify
Deployment health checks and failed-update detection after each ring.
Roll back
Rollback where supported. Some updates change stored data and cannot simply be reversed; the release notes are intended to say so.
Record
Update history and maintenance reports, per environment and per customer.
Proposed interface concept Not released · example data · subject to change
| Component | Severity | Affected services | Approval | Schedule and status |
|---|---|---|---|---|
| Mail serviceou-mail · 1.14.2 → 1.14.3SMTP, IMAP, filtering · Needs approvalProposed Sat 21 Nov 02:00 | HighOU-SA-2027-004 | SMTP, IMAP, filtering | Needs approvalReview | Proposed Sat 21 Nov 02:00 |
| Identity (Keycloak)ou-identity · 3.2.0 → 3.2.1Sign-in for every service · Approved · S. ReyesRing 0 done · ring 1 Tue 17 Nov | MediumOU-SA-2027-005 | Sign-in for every service | Approved · S. Reyes | Ring 0 done · ring 1 Tue 17 Nov |
| Document engine (Collabora Online)ou-docs-collabora · 2.8.4 → 2.9.0Documents · Approved · D. OkaforRing 1 · Sun 15 Nov 02:00 | Feature | Documents | Approved · D. Okafor | Ring 1 · Sun 15 Nov 02:00 |
| Proxmox VE host packagespve-a1 – pve-a3 · 14 packages, kernelAll VMs on cluster A · Approved · S. ReyesTue 17 Nov 22:00 · rolling | MediumVendor advisory | All VMs on cluster A | Approved · S. Reyes | Tue 17 Nov 22:00 · rolling |
| PostgreSQL minor releaseou-db · Next minorEvery service with data · Needs approvalNeeds failover to db-02 | Low | Every service with data | Needs approvalReview | Needs failover to db-02 |
Security fixes are never installed silently: High and Critical advisories raise an alert until they are scheduled, and an unsupported version is flagged on the overview.
Update ringsRollout of Document engine 2.9.0 · each ring waits for the one before it to stay healthy
Ring 0 · StagingInstalled
Ridgeline IT · Staging environment
12 Nov · healthy for 48 hInstalls once approved
Ring 1 · CanaryScheduled
Production · docs-01 only
Sun 15 Nov, 02:00–03:00After ring 0 is healthy for 48 h
Ring 2 · ProductionWaiting
Production · all document engine VMs
Earliest Wed 18 NovAfter ring 1 is healthy for 72 h
- Backup of docs-01 verifiedToday 01:20 · test restore passed
- VM snapshotTaken at the start of the window
- Compatibility checkFiles 4.6 and WOPI settings compatible
- Users toldBanner from Sat 18:00 · editing paused 5 min
- Ring 0: discovery endpointResponds · version 2.9.0
- Ring 0: open 20 sample documentsAll opened, layouts compared
- Ring 0: two-person co-editing testSaved and reopened
- Ring 0: error rate for 48 hAt or below the 30-day baseline
- Previous image keptou-docs-collabora 2.8.4
- Automatic if checks failWithin the window · no data migration
- Last rollback: Files 4.6.12 Nov · health check failed · back in 6 min
Two kinds of update
The ICS is designed to handle updates to the infrastructure it manages, such as container images, service versions and host packages, as well as Office Unbound releases themselves. Office Unbound releases are intended to be signed and checked before installation, and to go through the same review, ring and approval steps.
The full release path, from signature check to the last ring, is drawn in Flow J: customer-controlled updates. MSPs can apply update policies across customers, with an emergency path for critical fixes; see security controls.
How do you schedule maintenance today?
Windows, rings and approvals vary widely between businesses and MSPs. Describing yours helps the project design policies that fit.