Challenges of Deploying Kafka / Elastic Pipeline in Kubernetes

Aug 28, 2019 · Mountain View, United States of America

We are excited that Chas Honton will be sharing his experience deploying Elastic pipeline and Kafka in Kubernetes.

Doors open at 6pm and talks start at 6:30pm.

Talk Abstract:

Message Intelligence Service (MIS) collects connection and message metadata from thousands of production email clusters for near real-time monitoring, reporting, and searching. The service processes billions of messages a day and manages hundreds of terabytes of indices with a standard 30 day retention.

MIS consists of conventional data pipeline components from the Kafka and Elastic ecosystems. What might be unusual is the choice of using Kubernetes to deploy these components.

Kubernetes is the current choice for deploying stateless services in the cloud. Unfortunately, many of the simplifying assumptions that makes Kubernetes outstanding for deploying stateless services cause complications for stateful services such as Kafka and Elasticsearch.

This presentation discusses some of these challenges and how we dealt with them.

Speaker: Chas Honton

Chas Honton works with Proofpoint, "protecting the Very Attacked Person". He has previously worked with startups and established players such as Amazon Web Services and Intuit. He is an Apache Commons contributor and actively supports his open source productivity components on GitHub.

Event organizers
  • Elastic Silicon Valley User Group

    This group is for anyone interested in open source search, logging, analytics & data visualization. We primarily focus on these open source projects — Elasticsearch, Kibana, Logstash, and Beats — but we welcome talks on any aspect of data exploration. This Meetup community adheres to the Elastic Code of Conduct - https://www.elastic.co/community/codeofconduct. Attendance to events run as part of this Meetup group means you agree to be an awesome human and engage by these rules

    Recent Events
    More

Are you organizing Challenges of Deploying Kafka / Elastic Pipeline in Kubernetes?

Claim the event and start manage its content.

I am the organizer
Social
Rating

based on 0 reviews

Featured Events