HubSpot allows users to connect their third-party inboxes (such as Outlook) to sync emails and send 1:1 messages. When a connection error occurs, HubSpot classifies it as either recoverable or permanent:
- Recoverable errors resolve without user intervention.
- Permanent errors require manual reauthentication, and are typically triggered by password changes or users manually disconnecting their inboxes in the third-party provider.
When a permanent error occurs, HubSpot automatically disconnects the inbox and notifies users via email, prompting them to manually reconnect.
What Happened
On Saturday, March 1, 2025 at 8:40 PM UTC, a global Microsoft outage resulted in permanent authentication errors when HubSpot attempted to sync Outlook inboxes. This caused:
- A mass disconnection of connected Outlook inboxes from HubSpot.
- Automated email notifications informing affected users of the disconnect and prompting them to manually reconnect.
- Failure to sync incoming emails and an inability to send scheduled 1:1 emails during the outage.
Once Microsoft restored service at 9:45 PM UTC, HubSpot worked on an automated solution to reconnect inboxes without user intervention. Given the scale of our email infrastructure, with over 1.7 million connected inboxes processing more than 105 million emails daily, restoring all disconnected inboxes required extra time to ensure that each reconnection was successful and did not introduce further instability. Our priority was to restore service safely and prevent any additional disruptions.
By Sunday, March 2, 2025 at 6:22 AM UTC, all affected inboxes were successfully restored.
On Monday, March 3, many customers saw the automated email notification prompting them to take action to reconnect their inboxes. This led to some confusion since we’ve already reconnected inboxes on their behalf. ,
Timeline of Events (all times in UTC)
- March 1, 8:40 PM: Microsoft reported a global outage affecting Microsoft 365 services.
- March 1, 8:40 PM: HubSpot detected permanent authorization errors, disconnected Outlook inboxes, and sent automated email notifications to affected customers.
- March 1, 9:45 PM: Microsoft reported the outage to be resolved.
- March 2, 1:12 AM: HubSpot posted a status page related to the incident.
- March 2, 1:40 AM: HubSpot engineers began implementing an automatic inbox reconnection solution.
- March 2, 6:22 AM: All affected inboxes were reconnected to HubSpot.
Learnings and Actions Taken
We conduct detailed post-incident reviews for all incidents to ensure that we responded appropriately and proportionally to the situation. We use these reviews to identify opportunities to reduce the likelihood of similar events happening in the future and apply learnings to our future product reliability efforts.
Improved Outage Detection
- Issue: HubSpot did not correctly recognize this as a third-party outage, leading to unnecessary inbox disconnections, customer notifications, and confusion.
- Action: We are enhancing our outage detection mechanisms to prevent automatic inbox disconnections when external services experience widespread disruptions.
Faster Outage Recovery
- Issue: The time required to fully restore inbox connections exceeded the duration of the Microsoft outage.
- Action: We are improving our automated inbox recovery processes to ensure inboxes reconnect as soon as the third-party service is restored.
Clearer Customer Communication
- Issue: Customers received an automated email prompting them to manually reconnect their inboxes, even though the action was not available during the outage and most inboxes were already reconnected automatically after the outage.
- Action: We are improving visibility into the status of connected inboxes during and after an outage, ensuring users can clearly understand their inbox status and whether action is needed.