Kafka: The WhatsApp for Microservices Communication

We all lead busy lives. Some of us are buried in office presentations, while others are tackling assignments from the comfort of our beds. Yet, we always find time to catch up with each other, thanks to quick messaging apps that keep us connected. With these apps, your favorite person living across the globe is just a message away. But have you ever thought about the microservices running 24×7 inside their containers? They work tirelessly, handling all your exceptions, logging their "feelings" which you ignore until something breaks. Once upon a time, they were a big monolithic family, but some ingenious developers, in the name of optimization, split them into smaller, isolated services—microservices. Have you ever wondered how they communicate with each other? Enter Kafka—the WhatsApp of microservices, enabling them to chat from the safety of their individual containers.

What is Kafka?

Think of Kafka as a WhatsApp Community Group. While the admins have the power to publish content, the other group members are focused on consuming it. In Kafka terms, this group is known as a topic. A Kafka cluster consists of multiple topics. A Kafka producer acts like the admin, producing and publishing content. The Kafka consumer, akin to a group member, subscribes to a topic to consume the published content. The messages sent are referred to as events in Kafka terminology. The concept is straightforward—producers publish events to a Kafka topic, and consumers subscribe to a topic to consume these events. Depending on the use case, multiple actions can be taken on the received events.

For example, consider a WhatsApp Community Group focused on planning upcoming events in a residential society. The society chairman, acting as the admin, informs other members about upcoming events. The finance head consumes these messages to handle budget allocation. The PR head starts drafting invites for society members, while the Catering head identifies catering requirements. Although the admin published a single message, different members consumed it differently. This is how Kafka serves as a brokering interface, facilitating communication between microservices.

An order confirmation event might be processed by a payment-confirmation service to store payment details, an analytics service to extract information for analytics, and a logistics service to further process the order. Kafka enables all this communication to happen in real-time, concurrently. Its capabilities allow it to handle millions of such events effortlessly and asynchronously. You might be wondering, why Kafka?

Why Kafka?

Why not Kafka? It passes the trinity test of microservices helper services—it is highly efficient, reliable, and scalable. It enables different services to communicate with each other. Without such a communication mechanism, microservices architecture would have been deprecated long ago. How else would you transfer data between two different services running in separate places? While making inter-service API calls is an option, it comes with the limitation of how many concurrent calls you can make when you need to communicate the same information to multiple services.

Kafka in action

Here's a small code snippet showing how to create a Kafka producer and consumer using Python. The producer publishes a message to a topic, and the consumer reads the message from the topic.

Kafka Producer

from kafka import KafkaProducer

# Initialize Kafka Producer
producer = KafkaProducer(bootstrap_servers='localhost:9092')

# Send a message to the topic 'void_ness_topic'
producer.send('void_ness_topic', b'Hello, Kafka!')

# Ensure all messages are sent before closing the producer
producer.flush()
producer.close()

Kafka Consumer

from kafka import KafkaConsumer

# Initialize Kafka Consumer
consumer = KafkaConsumer('void_ness_topic', bootstrap_servers='localhost:9092')

# Read messages from the topic
for message in consumer:
    print(f"Received message: {message.value.decode('utf-8')}")

For GUI lovers - Conduktor

I get it. The terminal is cool. But sometimes, when you're five hours into debugging and still can't find the bug, you would appreciate if reading/publishing to a Kafka topic was as simple as clicking a button. This is where Conduktor comes into play. In simple terms, it helps you connect to a Kafka cluster and read/write content to a topic with just a few clicks. There are two ways to run a Conduktor instance locally:

1. Install the GUI as a desktop app. The download link and installation steps can be found on their website.

2. Alternatively you can consider running a little docker-compose file and you will have the Conduktor up and running on a local server. You can then access the GUI by opening the link using your browser. It is as simple as that and the suggested way of accessing Conduktor as per their team. You will find more details about setting it up from over here.

Conclusion

So this is all for now. We covered the loneliness of microservices and how they stay connected with each other with the help of Kafka. Kafka makes it possible to process millions of concurrent messages between different microservices in real-time. I hope you enjoyed reading the analogy between how Kafka operates similarly to WhatsApp community groups for us. If you did, don’t forget to like this article. Do let me know in the comments below what other tech topics you would like to read about next. Till then, have a great time playing ping-pong with microservices :)