Product & Engineering Roundtable: HubSpot's INBOUND Product Announcements
It's time for INBOUND, the premier conference that brings together business, marketing, and sales leaders to network, learn, and grow. This year, ...
It was back in 2011 when then-HubSpotter Yoav Shapira wrote a blog post to celebrate an important milestone: more than 1,000 releases of our software. In his post, Yoav also manifested his dream that HubSpot would one day embrace Continuous Deployment, which he foresaw would make "everyone more productive, and deploy safer". That day is upon us.
Just last month, fellow HubSpotter Ken Sykora shared an insightful overview of the foundational work that our Infrastructure and Platform teams have been doing to make Continuous Delivery a reality at HubSpot. In his blog post, Ken discusses the challenges that come with adopting such a system at HubSpot's current scale, with over 50,000 deploys per day across 3 data centers.
In this blog, I want to tell the other side of Ken's story. I will share how our Growth Onboarding Frontend team has embraced Continuous Deployment to increase its efficiency and productivity, and how that has translated in incremental value delivered to customers.
The journey of a code change through our CI/CD pipelines
Let's begin by peeking behind the curtains of how Continuous Integration / Continuous Deployment ("CI/CD") pipelines work in the context of Frontend development at HubSpot. But first, some definitions.
Continuous Integration ("CI") refers to the use of automated testing to validate changes to a codebase. Continuous Deployment ("CD")—also referred to as "Continuous Delivery"—, makes it so that, when these tests pass, and the build is successful, an application is automatically deployed to a production environment. A CI/CD pipeline is the combination of these two systems, such that code changes can be automatically delivered to users. Let's have a look at how these concepts work in practice in the context of how our Frontend teams ship changes to their applications.
At HubSpot, the CI/CD process begins even before the code leaves an engineer's machine. First, we use TypeScript to check for type safety at compile-time. In addition, our toolchain includes husky pre-commit hooks that automatically review code syntax and style. These rules enforce agreed-upon standards in the way we write code.
Example of a pre-commit error from a failed syntax check
The next checks happen during the build process. Frontend teams at HubSpot embrace the testing practices outlined in the Testing Trophy: Unit tests to assess business logic, Integration tests to verify functional requirements, and End-to-end Acceptance tests to check that everything in the system works together as expected from the perspective of the end user.
After code changes are committed to a remote repository (we use Git for version control), our in-house build system immediately runs the application's unit and integration tests. The build will fail if any of the test suites fails. Instead, successfully built artifacts can be previewed on feature branches, which can be further tested running the end-to-end tests:
End-to-end acceptance tests can easily be run on feature branches via a custom GitHub integration
When code changes pass all tests—as well as the code review process—they are usually merged to the master branch. Orion, the HubSpot deploy system, then deploys them to a shared QA environment. This is already a form of continuous deployment (with automatic builds deploy), albeit one that stops at the QA environment. Orion will also relaunch any end-to-end acceptance test associated with the freshly deployed app. Declaring which test suites should run post app deploy is incredibly simple thanks to our custom deploy configs:
Finally, builds deployed to QA which pass their end-to-end acceptance tests are automatically deployed to production—ready to be used by our customers.
The benefits of Continuous Deployment
As outlined above, CI/CD requires a heavy investment in automation throughout all deployment stages. But, with our current process, an engineer's manual involvement in the deploy process can be minimal, leading to increased developer productivity. In the HubSpot Product org, we believe in moving quickly and iterating fast, so that we can constantly evolve with our customers’ needs. With hundreds of Frontend developers and projects at HubSpot, Continuous Deployment has become indispensable to enable fast-paced iteration and a consistent user experience for the end user. In fact, the benefits of CD are plentiful and can be reaped both by product teams and end users.
By automating the deployment process, teams can avoid the errors and inconsistencies that can occur when deploying manually. This can improve the reliability and stability of Frontend applications. Continuous Deployment can also help teams to improve collaboration and communication: rather than spend time on manual deployment tasks, engineers can focus on writing and reviewing code. This can help to improve the overall efficiency and productivity of product teams. In addition, when teams deploy their changes often, they also minimize the risks that come with large batch deployments. Finally, CI/CD fosters testing practices that help to reduce the number of errors and bugs released to customers.
For end users, the main upside of Continuous Deployment is that it allows for faster and more frequent releases of new features and updates. This means that users can benefit from new and improved functionality sooner. Additionally, because CI/CD can help teams to identify and fix bugs and issues quickly, end users are less likely to encounter problems in the application. This translates to a better overall user experience and to improved user satisfaction. Thus, the level of service and value delivered to final customers increases.
Summary of the possible benefits of CD for Product Teams and End Users
For our Growth Frontend teams, shipping frequently also increased the rate and speed of our experimentation and learning. In the words of HubSpot's Mike Champion, getting rapid feedback and data points from users translates into higher value delivered to customers.
An example of how Frontend teams can gradually adopt Continuous Deployment
Continuous Deployment is not a switch that can just be flipped on overnight. HubSpot's Frontend infrastructure empowers teams with state-of-the art, homegrown developer tools that make CI/CD very approachable. However, the burden and responsibility of putting in place the automated tests ultimately falls on the engineering teams. It took our team about 9 months and approximately 20 new tests suites across 4 repositories to embrace Continuous Deployment. But the approach that we followed can be a blueprint for Frontend teams thinking of adopting CI/CD.
Screenshot of our deploy platform showing multiple per-day production deploys of one of our Frontend applications
We started with a cost/benefit analysis. We wanted to identify projects where the adoption of Continuous Deployment would benefit our users and our team. We deprioritized apps and libraries in "maintenance mode" and with infrequent changes, as well as those with declining user adoption or shrinking strategic value. For these repositories, the foreseeable upsides didn't justify the costs of tackling technical debt and strengthening their tests.
We then transitioned to the diagnosis phase. For each candidate repository, we tried to determine which and how many tests to write. Finding the right amount and mix of tests is never straightforward, but, as Kent C. Dodds—creator of React Testing Library—suggests, "the thing you should be thinking about when writing tests is how much confidence they bring you that your project is free of bugs". We also tried to avoid some common pitfalls:- Too many tests can lead to long build and wait times for your developers, slowing iteration down
- An excessive amount of end-to-end tests can also exacerbate type I false-positive errors
- Not enough tests can let errors creep into your user experience, and bugs to go unnoticed. This could lead to Type II errors, and let issues in your code affect your users
- Your testing environment should mirror the production one very closely, such that it were representative of the user experience
- Instrument your app to keep track of JavaScript client- and server-side errors
- Monitor your SLA metrics (latency, availability, and so on) closely, and add alerts to be notified if they drop
- Keep an eye on your product business metrics (conversion rates, flow progressions, etc.) to measure how they are affected by your product releases
- Consider using feature flags to gradually roll out big changes
- Implement an easy rollback process to quickly redeploy an older build if something goes wrong
Wrapping Up
"Our team went from deploying some of our apps once or twice per week to releasing new changes to production once or twice a day!"
CI/CD cannot simply be an afterthought for engineering teams. In fact, it needs to become an integral part of how a team operates. While there are non-negligible costs associated with writing and maintaining the level of unit, integration, and acceptance tests imposed by CI/CD, automating the deployment process can result in tangible benefits for product teams and end users alike.
Continuous Deployment has been a huge win for our Growth Onboarding teams. It has been speeding up our iteration cycles, and has enabled us to move fast without breaking things. Our team went from deploying some of our apps once or twice per week to releasing new changes to production once or twice a day! This has been translating into incremental value created for our customers, helping them discover and adopt HubSpot features to help them grow their business.
If your engineering team is not already using Continuous Deployment, now is the time to get started. Getting started with CI/CD may look daunting, but the benefits are well worth the effort, and the testing and deployment tools available are getting better by the day. Don't miss out on the many advantages that continuous deployment has to offer—start to take on this challenge today!
Interested in working on challenging projects like this? Check out our careers page for your next opportunity! And to learn more about our culture, follow us on Instagram @HubSpotLife.