Connecting the {Dots}

This post is the continuation of my previous post on building Kafka client:   https://www.jobinesh.com/2021/10/how-to-get-your-apache-kafka-client.html. In case you have not yet read it, would encourage you to take a look at it before proceeding with this post. In this post, I will share a simple real-life Kafka client implementation.  In case you have not yet built a Kafka client of production quality, please read this article https://www.confluent.io/blog/kafka-consumer-multi-threaded-messaging/. The client implementation used in this example is heavily inspired by the concepts discussed in the above-mentioned article. Some features that you may find interesting in the Kafka client example shared along with this post are: Listen to Kafka rebalancing and handle it as appropriate Error handler for async Kafka message processing task Rate limiting the incoming messages by waiting for the currently running Kafka record processing threads to finish. About this example H...

Recently I have got an opportunity to work on a Kafka client implementation for an interesting use case. Till then my assumption was writing a Kafka client was as easy as we see in many examples on the net :) Although it's true for many see cases, we cannot say the same simple client work for all. Depending upon the complexity of the use case that you may deal with, the client's implementation might change and complexity may increase.  Please note that this is not an introductory article on Kafka, also expected you to have some basic understanding of Kafka's client. I found the following article very useful while learning the basics of Kafka client:  https://www.confluent.io/blog/kafka-consumer-multi-threaded-messaging/  .  This post will refer to various topics from this article as we move forward.  In this post, I am sharing three common scenarios that you may need to deal with while using Kafka client in an message heavy system and possible solutions or patte...