On Thursday, August 7, 2025, between approximately 1:45 PM EDT and 2:05 PM EDT, HubSpot customers experienced widespread errors when accessing CRM records. With approximately 95% of requests failing at peak impact, this resulted in “All is not lost” errors across all record pages, object previews, and workspace preview sidebars.
The issue was caused by a programmatic system configuration change that had dependencies on code updates in other services, which had not yet been deployed everywhere they were required. There was no data loss during this time.
We understand that reliable access to CRM data is essential for daily workflows, and we sincerely apologize for the disruption this caused. Below is a detailed account of what happened, why it occurred, and the actions we are taking to prevent similar issues in the future.
We deployed a configuration update to our CRM system that depended on related code changes across multiple services. However, those supporting changes hadn't been fully deployed across all relevant services. While the change appeared to pass standard validation checks in both our Test and Production environments, those validations did not confirm that the dependent code was fully deployed across all relevant services.
Because the CRM Service did not have defensive safeguards to handle incomplete configuration rollouts, the change caused it to return errors for all requests dependent on the updated configuration once deployed to Production.
1. Defensive Service Hardening
Harden configuration processing in CRM services to handle incomplete configuration changes gracefully.
2. Improved Validation and Deployment Safeguards
Enhance validation checks to confirm that all dependent code changes are fully deployed across relevant services before approving configuration changes.
Implement slower, more gradual rollouts for critical CRM updates with enhanced monitoring at each stage.
Strengthen automated testing to catch cross-service dependency issues before they reach customers.
Improve service alerting sensitivity for critical customer-facing functions.
This incident enabled us to improve how we validate and roll out configuration changes with cross-service code dependencies. By strengthening validation, improving defensive programming, and narrowing the scope of impact from future deployments, we are reducing the likelihood of similar events in the future.
We know this disruption impacted customer workflows, and we are committed to making changes that improve the reliability and safety of updates across the HubSpot platform.