"The Magical Rebalance Protocol of Apache Kafka" by Gwen Shapira
Very few people know that inside's Apache Kafka's binary protocol for publishing and retrieving messages hides another protocol - a generic, extensible protocol for managing work assignments between multiple instances of a client application.
When multiple Kafka consumers in the same consumer group subscribe to a set of topic partitions, Kafka knows how to assign a subset of topic partitions to each consumer and how to handle failover automatically. What is less known is that this assignment is determined by the consumer client itself and that the same protocol can be used by any application for both leader election and task assignment.
In this session we'll dive into the internals of this little-known assignment protocol -- the binary network protocol and the Java APIs. We'll look in detail at how Kafka Consumers, Connect and Streams API use this protocol for task management. And finally we'll show how you too can extend this protocol to implement task assignment in your application with an algorithm of your choice - even if it doesn't use Kafka for anything else.
Speaker: Gwen Shapira
Watch video "The Magical Rebalance Protocol of Apache Kafka" by Gwen Shapira online without registration, duration hours minute second in high quality. This video was added by user Strange Loop Conference 16 October 2018, don't forget to share it with your friends and acquaintances, it has been viewed on our site 15,188 once and liked it like people.