Purple Teaming with the Threat Hunters

For many SaaS companies, responsibility for an application’s security is shifting to the application teams themselves, and HubSpot is no exception. HubSpot has always engaged with penetration testers from external security firms—in addition to security researchers via our bug bounty program—and will continue to do so. These external engagements do provide real value in terms of independent testing and verification, but there are advantages to Engineering taking direct responsibility for a portion of security testing as well. With external testers, HubSpot engineers have to spend time providing the right context, setting up access, and helping to troubleshoot through the engagement. With thousands of deploys daily, we need the freedom and flexibility to tackle high value changes without a ton of overhead.

To help meet this goal, HubSpot has a new Infrastructure Security team in town: the Threat Hunters. With a mix of internal and external hires, the members of the Threat Hunters team bring a wealth of security experience. The mission of our team is to systematically reduce HubSpot’s attack surface and continue to improve the maturity of HubSpot’s product security. This means working with the development teams across HubSpot to make the product as secure as possible for our customers. One way we’re doing this is through an idea called “micro purple teaming.”

Why Purple?

The idea behind purple teaming is to create a feedback loop between red teamers—ethical ‘hackers’ who emulate an adversary and simulate attacks in an environment—and blue teamers, the defenders who continuously assess the security in an environment and attempt to mitigate and prevent attacks. Micro purple teaming is the HubSpot version of purple teaming, where folks from the Threat Hunters team work with the owners of various features and applications within product to attack our product and then provide ideas for mitigations and security improvement before an attack comes from a real threat actor. 

Process

The Threat Hunters team starts the process by meeting with a development team to understand more about their system and tackle the logistics of the testing. When will the features be ready? Are they gated or available to everyone? Should this be tested in QA or Production? We learn about how the features are meant to be used and the potential areas that could be abused. The development team has time to share their fears and the areas they’re unsure about. A testing time is put on the calendar.

In the week leading up to the planned test, the Threat Hunters take time to review the codebase, noting high risk areas and preparing the tools we’ll need for testing. Having this plan ready allows for maximum results during a relatively short testing window. During testing, the team keeps detailed notes on the process, so that found vulnerabilities can be re-created by other team members and re-tested after fixes. While we are on the lookout for areas that can be exploited, we’re also keeping “blue team” defensive tactics in mind. Did any exceptions or alerts warn the team that an exploit could be happening? Could the activities of an attacker be accurately re-created from a service’s logs? 

Once the test is concluded, team members review the findings and write up a detailed report with information on areas of concern. Once the report is shared with the development team, HubSpot’s Mainsail process is used to track the items. Reports are available for all HubSpot engineers to review and learn from.

Value to HubSpot 

While external penetration testers can offer only generic advice for possible fixes, an in-house purple team knows exactly which tools are available in HubSpot’s environment to address a vulnerability. It can offer pointers to strategies other internal teams have used in similar situations. When the same vulnerability appears multiple times, the team can create new training videos, deliver a presentation to the team during HubSpot’s weekly “Tech Talks,” or develop tools that can detect the issue sooner in the development process. 

Using tools such as Portswigger’s Burp Suite for manipulating input, in addition to code review and homegrown scripts, the Threat Hunters have helped multiple teams with security improvements before new features made it to production. In some cases, these were minor improvements: better error messages that avoid leaked information, or improved auditing that could help investigators during a security incident. Other discoveries, however, revealed flaws in business logic or situations where a backend service put too much trust in the inputs it received. These could have seriously weakened security protections were they not caught early.

By working as an extension of the development teams, HubSpot’s purple team can develop a trusted relationship with them. Dev teams know there is a resource for asking questions, or offloading nagging security worries that they aren’t sure about. An internal team also understands the internal pressures and processes that might lead to a particular bug. Identifying these culture- or environment-related issues can lead to security improvements for every team.

If solving problems like these and creating better application security solutions interests you, check out our open Senior Software Engineer position.