--- title: Monitoring Driven DevOps description: Let monitors drive your devops work and avoid being too late getting work done. image: https://product.hubspot.com/hubfs/BlogCode_Green-Oct-01-2020-12-25-41-04-AM.png --- [Engineering](https://product.hubspot.com/blog/topic/engineering) # Monitoring Driven DevOps [ Jeremy Katz ](https://product.hubspot.com/blog/author/jeremy-katz) on Jun 27, 2011 Last week, I attended the Velocity Conference on web performance and operations out in California. The conference itself was really good; a ton of good talks on topics like post-mortems, building systems for failure, failures that people have hit, how to successfully embrace developer ownership of product operations. Basically a lot of the things that we embrace and encounter every day at HubSpot. On Tuesday night, there was a set of Ignite talks. If you've never been to an Ignite talk, the idea is that you have 5 minutes to give a talk. Pressure, right? Well, to make it more fun, you also have 20 slides and they auto-advance every 15 seconds. Okay, lots of pressure. And somehow I talked myself into submitting a proposal which was accepted so I then had to write a talk. **[Just Too Late](http://www.slideshare.net/katzj/just-too-late)** View more [presentations](http://www.slideshare.net/) from [katzj](http://www.slideshare.net/katzj). As with many of the Velocity talks, mine was a story. About six months ago, I shifted to running a team that is responsible for automating, standardizing and improving the infrastructure that powers HubSpot. Part of this has involved a bit of a mindshare shift from that of just being a developer to caring a bit more about the operations side of things. In doing so, I've noticed some differences that have made me feel a bit more like the work I'm doing is reactive and just a little bit too late. Part of this is because the goal of operations work is that nothing goes wrong... this means that you build something and nothing happens! So you don't get to some of the preventative work you'd like. But then, inevitably, the bad thing occurs. And then you do some sort of post-mortem. One of the things we've been finding in a lot of ours is that we could have found the problem faster instead of having our customers report the problem. So a lot of our corrective actions end up being things which would allow us to do so: alerts in our monitoring system, automated testing and similar things to find out sooner. From here, I talked a bit about how we fared with the Amazon EBS problems a couple of months ago. Overall, we fared pretty well. But there were some areas where we had problems. One of these was that we had a database which was not being backed up according to our standards. We were lucky and it wasn't a lasting problem. But as part of the post-mortem, I went ahead and backed up the server. I then put a monitor in place to ensure the server was backed up. And then I generalized it out so that it applied to all of our servers. And this is when we discovered that there were a few other servers on the brink of disaster. So then I backed them up. Coincidentally, my team was working on the automated build process for our database servers of the future, so we went ahead and built it into that process so that servers would get it automatically in the future. But I had a little bit of an inspiration here. As a developer, I would do test driven development; I'd write a unit test and then get my code to work to make the test pass. As an infrastructure developer, I was doing monitor driven infrastructure development; I wrote a monitor and then made the infrastructure match that state. Pretty powerful; I could do it with our old stuff or with new things we were building. And adding a monitor to start with is cheap; from there you can make easier value judgements on what the cost is of not making preventative improvements. You do have to be careful that your monitors aren't noisy in the process, but it's working. I finally don't feel like I'm always behind and getting to things just too late. Instead, I'm getting to things before they explode which in the end means a more stable system, happier developers, and happier customers using our product.   [![so-hiring](https://d1n2i0nchws850.cloudfront.net/portals/51294/6b9a27b2-9f52-41f9-9d92-488c8f59765a-1333593960679/so-hiring.jpg?v=1333593960.98)](https://www.hubspot.com/jobs) [Share on Twitter](https://twitter.com/intent/tweet?url=https%3A%2F%2Fproduct.hubspot.com%2Fblog%2Fbid%2F65871%2Fmonitoring-driven-devops)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fproduct.hubspot.com%2Fblog%2Fbid%2F65871%2Fmonitoring-driven-devops)[Share via email](mailto:info@example.com?&subject=&body=https%3A%2F%2Fproduct.hubspot.com%2Fblog%2Fbid%2F65871%2Fmonitoring-driven-devops)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fproduct.hubspot.com%2Fblog%2Fbid%2F65871%2Fmonitoring-driven-devops) ## Recommended Articles [Culture](https://product.hubspot.com/blog/topic/culture)[UX](https://product.hubspot.com/blog/topic/ux)[Product](https://product.hubspot.com/blog/topic/product) ## [Day in the Life: Erin Couse, Content Design Lead](https://product.hubspot.com/blog/day-in-the-life-erin-couse) Meet Erin Couse, Content Design Lead at HubSpot! Erin's career started out in marketing and social media, and led her all the way to a role in UX and ... [ Erika Cote ](https://product.hubspot.com/blog/author/erika-cote) on Jul 19, 2022 [Culture](https://product.hubspot.com/blog/topic/culture)[UX](https://product.hubspot.com/blog/topic/ux)[Product](https://product.hubspot.com/blog/topic/product)[Product Management](https://product.hubspot.com/blog/topic/product-management) ## [Day in the Life: Dana Land, Deliverability Consultant](https://product.hubspot.com/blog/day-in-the-life-dana-land) Meet Dana Land, Deliverability Consultant at HubSpot. She started her career in tech at a local Chicago startup before moving to HubSpot. Since ... [ Erika Cote ](https://product.hubspot.com/blog/author/erika-cote) on Jul 27, 2022 [UX](https://product.hubspot.com/blog/topic/ux) ## [How Implementing a Design Sprint Led Our Team to Quick Results](https://product.hubspot.com/blog/design-sprint-explore-week) Introduction Imagine that planning season is happening, and your team has decided on the top priority for 2023. It’s been given the go-ahead by ... [ Stormi Knight ](https://product.hubspot.com/blog/author/stormi-knight) on Nov 2, 2022 ## Join our subscribers Sign up here and we'll keep you updated on the latest in product, UX, and engineering from HubSpot. [Subscribe to the newsletter ](https://product.hubspot.com/blog/bid/65871/monitoring-driven-devops#subscription-popup) [Back](https://product.hubspot.com/blog/bid/65871/monitoring-driven-devops#) ## Get Connected The latest engineering, UX, and product news from the HubSpot Product Blog, straight to your inbox. Next Not using [HubSpot](https://www.hubspot.com/) yet? Search Term: ** Search HubSpot.com ** Search the blog Load more