Kafka at HubSpot: Critical Consumer Metrics
Apache Kafka is an integral part of our infrastructure at HubSpot. We often have long pipelines of workers that consume from and publish to Kafka ...
Neha Narkhede is the co-founder and head of engineering at stream data platform company Confluent, and one of the original authors of Apache Kafka. Earlier this week, Neha joined us at HubSpot for Boston Kafka Night where she talked to us about Kafka's architecture, discussed how companies are leveraging it for advanced real-time data processing, and discovered Samza on the Rocks.
Neha walked through the evolution of LinkedIn's data systems from an increasingly coupled architecture to separate systems linked with Kafka as a message bus. Similarly, Kafka is now an instrumental part of HubSpot's infrastructure. After Neha's presentation, two of our engineers, Ze'ev Klapow and Mike Axiak, spoke on how we've operationally integrated Kafka. Mike spoke about how HubSpot has dealt with availability and failover, and Ze'ev covered the consumer metrics we've developed to gauge our system's performance and health.
Save the date: The Apache Kafka Summit is in San Francisco on April 26th, 2016.