Rackspace hosted Exchange suffered a catastrophic interruption starting December 2, 2022 and is still continuous since 12:37 AM December 4th. At first described as connection and login issues, the assistance was ultimately updated to reveal that they were handling a security occurrence.
Rackspace Hosted Exchange Issues
The Rackspace system went down in the morning hours of December 2, 2022. Initially there was no word from Rackspace about what the issue was, much less an ETA of when it would be fixed.
Clients on Buy Twitter Verification reported that Rackspace was not responding to support emails.
This has been quite the day with #Rackspace. Every hosted exchange client has actually been down for 14 hours approximately. Support isn’t reading/responding to tickets. Updates are unhelpful.
I am worried now that they came down with something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace consumer independently messaged me over social media on Friday to relate their experience:
“All hosted Exchange customers down over the past 16 hours.
Not exactly sure the number of companies that is, however it’s significant.
They’re serving a 554 long hold-up bounce so individuals emailing in aren’t aware of the bounce for numerous hours.”
The main Rackspace status page used a running upgrade of the interruption however the initial posts had no information aside from there was an outage and it was being investigated.
The first official update was on December 2nd at 2:49 AM:
“We are investigating a concern that is affecting our Hosted Exchange environments. More information will be posted as they become available.”
Thirteen minutes later Rackspace started calling it a “connectivity problem.”
“We are investigating reports of connectivity problems to our Exchange environments.
Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their e-mail client(s).”
By 6:36 AM the Rackspace updates described the continuous problem as “connection and login concerns” then later on that afternoon at 1:54 PM Rackspace announced they were still in the “examination stage” of the failure, still trying to figure out what failed.
And they were still calling it “connectivity and login issues” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
4 hours later Rackspace referred to the circumstance as a “substantial failure”and started offering their consumers free Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround up until they understood the problem and might bring the system back online.
The main assistance mentioned:
“We experienced a substantial failure in our Hosted Exchange environment. We proactively closed down the environment to prevent any further problems while we continue work to bring back service. As we continue to resolve the root cause of the problem, we have an alternate option that will re-activate your capability to send out and receive e-mails.
At no cost to you, we will be providing you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 up until additional notification.”
Rackspace Hosted Exchange Security Event
It was not until almost 24 hours later at 1:57 AM on December 3rd that Rackspace officially announced that their hosted Exchange service was experiencing a security occurrence.
The statement even more revealed that the Rackspace technicians had powered down and disconnected the Exchange environment.
“After additional analysis, we have actually identified that this is a security occurrence.
The known effect is isolated to a part of our Hosted Exchange platform. We are taking needed actions to evaluate and safeguard our environments.”
Twelve hours later on that afternoon they upgraded the status page with more information that their security group and outdoors professionals were still dealing with solving the failure.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has actually not launched information of the security event.
A security occasion normally involves a vulnerability and there are two severe vulnerabilities presently in the wile that were covered in November 2022.
These are the 2 most present vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Demand Forgery (SSRF) attack allows a hacker to read and alter data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an opponent has the ability to run destructive code on a server.
An advisory published in October 2022 explained the effect of the vulnerabilities:
“A confirmed remote attacker can perform SSRF attacks to intensify benefits and perform arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted versus Microsoft Exchange Mailbox server, the assailant can possibly access to other resources by means of lateral motion into Exchange and Active Directory environments.”
The Rackspace blackout updates have not shown what the particular problem was, just that it was a security incident.
The most present status upgrade since December 4th mentioned that the service is still down and consumers are motivated to migrate to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in attending to the incident. The accessibility of your service and security of your data is of high importance.
We have actually dedicated extensive internal resources and engaged world-class external knowledge in our efforts to reduce unfavorable effects to customers.”
It’s possible that the above kept in mind vulnerabilities relate to the security incident impacting the Rackspace Hosted Exchange service.
There has been no statement of whether customer info has actually been compromised. This occasion is still ongoing.
Included image by Best SMM Panel/Orn Rin