Kafka Meetup with our friends from Zalando

Nov 7, 2017 · Berlin, Germany


We are joining forces with our friends form the Zalando Tech Events Meetup Group who are hosting this meetup.

For this Kafka Meetup, we welcome Ricardo De Cillo, Dmitriy Sorokin, Andrey Dyachkov, Max Schultze, Daniel Truemper, Ian Duffy, and Conor Clifford with a series of short talks highlighting tales from the trenches. Topics will be focused on operating large, critical Kafka clusters in the cloud - and hope to foster discussion and share best practices for running Kafka without losing sleep. 

*Please register on Eventbrite to guarantee entrance: Kafka Meetup Registration


17:30 - Doors Open - Drinks + Snacks 
18:20 - Welcome and Intro 
18:30 - Nakadi Event Broker - Lionel Montrieux
18:45 - Operating Kafka and Zookeeper on AWS - Ricardo De Cillo
19:00 - Bubuku - A Supervisor to Run Kafka on AWS - Dmitriy Sorokin 
19:15 - Break - Drinks, Snacks, and Discussions 
19:45 - Kafka and EBS - Andrey Dyachkov 
20:00 - The Tricky Thing about Offsets - Max Schultze 
20:15 - Kafka-powered Microservice - Daniel Truemper 
20:30 - Networking + Drinks 
21:45 - Event ends

For more details on topics and speakers, please read below:

Nakadi Event Broker
Speakers: Lionel Montrieux

Nakadi is an event broker, built on top of Kafka-like queues, that provides a REST API for easy integration with microservices. In this talk, we will briefly introduce Nakadi, and then focus on one of its most interesting features, Timelines.

Timelines allow administrators to transparently move topics between different clusters, without interrupting producers or consumers. This is useful to move between clusters, possibly using different technologies, but also as a fail-over for publishers in case of downtime on a cluster. Timelines can also be used to move event types between brokers on the same cluster, which is an alternative to rebalance operations. Contrary to rebalance operations, using timelines does not require additional data copy between brokers, which can slow them down. We also have plans to use timelines for re-partitioning, i.e. changing the number of partitions in an event type.

Operating Kafka and Zookeeper on AWS 
Speaker: Ricardo De Cillo 

Deploying and operating Kafka and Zookeeper reliably on AWS is a non trivial task. Most teams learn it the hard way. In this talk we are going to share our experience growing a Kafka cluster to ingest 5 TB of data per day with 99,999% availability.

You can expect to hear about: Deployment, Monitoring, Recovering from various failure scenarios, Scaling, Hardware choices, Important configurations

There is no single solution to this challenge, so let's talk. What choices did you make regarding operations and why?

Bubuku - A Supervisor to Run Kafka on AWS 
Speaker: Dmitriy Sorokin

Operating a Kafka cluster on a cloud platform such as AWS comes with its challenges - finding exhibitor nodes, updating the configurations, reacting to lost instances without downtime, rebalancing data across brokers, and many others). Daily operations on large clusters, with thousands of topics, are not trivial, and can take up a lot of time.

To simplify operations, we have built Bubuku, a supervisor for Kafka. In this talk, we will explore some of Bubuku's most useful features, such as rebalances and migration of data, updates of Zookeeper, and rolling restarts of Kafka brokers. We will share our experience in running a large cluster in production using Bubuku, and discuss future plans for the project. 

Kafka and EBS
Speaker: Andrey Dyachkov (github)

Kafka cluster is able to grow to huge amount of data stored on the disks. Hosting Kafka, requires support of instance termination (on purpose or just because `cloud provider` decided to terminate the instance), which in our case introduces a node with no data, the rebalance of the whole cluster has to accomplished in order to evenly distribute the data among the nodes, taking hours of data copying. In this talk, we will present how we avoid rebalance after node termination in Kafka cluster hosted on AWS.

The Tricky Thing about Offsets
Speaker: Max Schultze (twitter)

The Data Lake at Zalando is all about archiving data and making it accessible efficiently. For that, we consume a lot of data from Zalando's internal event bus Nakadi, which is based on Kafka. Additionally, we run another internal Kafka that data flows through first before it gets archived. This talk will be about presenting our current infrastructure, as well as talking about the concept you can mess up the most with during manual operations: offsets.

Kafka-Powered Microservice
Speaker: Daniel Truemper (twittergithub)

Microservices have fueled an industry-wide transition toward massively distributed architectures with many more people working on the same systems. They also allow for greater individual ownership, freedom of deployment, and diverse technology choices.

In this distributed landscape, easily creating and maintaining consistency among various views of business-critical data, decoupling reads from writes, decoupling microservice go-lives, while adhering strictly to incrementality emerge as important topics.

This presentation will demonstrate how Kafka can function as a backbone for the microservice world thanks to log compaction, which makes it unique compared to other event brokers. We’ll show how log compaction is essential in delivering a consistent user experience, and how this can empower teams developing their own autonomy. We will walk through a migration path from a hypothetical monolithic shop to a set of microservices.

Event organizers
  • Berlin Apache Kafka® Meetup by Confluent

    **** Join our Slack Team for live, global interaction! **** This Meetup Group will focus on the Apache Kafka® technology and its ecosystem, also allowing its members to interact and share their use cases, do and don't, how to's, etc. We will also hear about the Confluent Platform and topics like Kafka's Connect API and streaming data pipelines, Kafka’s Streams API and stream processing, Security, Microservices and anything else related to Apache Kafka. If you want to hear about a particular topic please le

    Recent Events

Are you organizing Kafka Meetup with our friends from Zalando?

Claim the event and start manage its content.

I am the organizer

based on 0 reviews