How a Microsoft Outlook outage spread across Exchange servers
Microsoft’s Exchange Online network began failing in Switzerland last Thursday, then spread to Mexico and other countries as failover systems redirected email traffic. More than 80 percent of Exchange servers were unresponsive within a couple of hours, according to sources familiar with the outage.

The disruption began around 10 a.m., about an hour after Microsoft employees arrived on campus for a normal day of email and meetings. Employees, businesses and governments lost access to Outlook emails for nearly 24 hours, in what the source describes as one of Microsoft’s worst outages in history.
Microsoft classified the incident as “sev0” in its internal tracking system, the highest severity level, indicating a catastrophic service disruption, sources said. Hundreds of engineers worked to identify the cause and restore service. Failover systems shifted traffic away from servers that were offline, but in this case those systems also helped spread the problem.
Engineers examined crash reports, trace logs and kernel dumps. Some servers had entered maintenance mode and stopped responding under unusually high CPU loads. Attempts to bring them back online were quickly overwhelmed by traffic waiting to reconnect.
Exchange servers boot and run an operating system over a network using Preboot Execution Environment, or PXE. The process lets Microsoft support staff and engineers manage operating-system updates remotely.
Comments (0)
No comments yet. Be the first!