Kafka MirrorMaker Replication Guide: Zero-Downtime Strategies for Seamless Source and Target Cluster Consumption

Apache Kafka’s MirrorMaker is a vital tool for achieving data replication between Kafka clusters, ensuring uninterrupted operations and data availability. This guide delves into core Kafka concepts and presents strategies for establishing efficient and seamless Kafka MirrorMaker replication.

Before getting into MirrorMaker setup, Let us first understand some Kafka core concepts. This will help us better understand how we can configure our applications to benefit from MirrorMaker setup

Kafka core concepts

In Kafka world ,

• Producers are those who publish(write) messages to Kafka
• Consumers are those who consume(read) messages from Kafka
• Cluster is a set of machines serving a common purpose.
• Topic is similar to a folder in file system and Events are files in that folder.
• Topics are partitioned(split) into multiple nodes for high performance.
• Offset is similar to byte/character position in a file which helps us track upto which point a consumer has consumed message in a partition

If we want to create a file, we need to know directory where we want to create this file with information. In similar manner while sending messages to Kafka, we need to know which topic we wish to send message.

If we want to get some particular information from a file , we need to know it’s directory , sub_directory, byte position. In similar manner when consuming from Kafka cluster, we need to know Topic , partition, Offset.

I would recommend going through this documentation for more details regarding Kafka

Now let’s understand what is MirrorMaker in brief points.

MirrorMaker

Apache Kafka MirrorMaker is a tool used for replicating data between two Kafka clusters. It’s designed to enable the seamless transfer of messages from a source Kafka cluster to a target Kafka cluster. This replication is essential for various purposes, such as disaster recovery, data migration, load balancing, and data sharing between different environments.

I would recommend this documentation for more details about MirrorMaker :

Now, that we understood the basics of Kafka and MirrorMaker .Let’s try to understand some common concerns about MirrorMaker setup in relation to our application configuration.

Basic scenario without replication

Let’s start with scenario where we don’t have MirrorMaker setup

Now, from above diagram we see that Producer configured to write to topic driver_topic. Consumer is configured to read from topic driver_topic.

producer needs to write to topic driver_topic.

Consumer needs to read messages from driver_topic , part0/part1 with some offset.

Original flow

Let’s take simplest form of above architecture as shown in below

Now, In our application we can have below configuration

producer -> driver_topic

consumer -> driver_topic

Now, let’s add target cluster for replication using MirrorMaker as shown in below figure. Here comes the first question : What should be Topic configuration for producer and Consumer 🤔?

Scenario 1: Common topic name for Producer and Consumer

1. Producer -> driver_topic
2. Consumer -> driver topic

Issue:

If we just give original topic name to both producer and consumer then it works only if they producer or consume from source cluster.They will receive topics not found error if they try to connect with Target cluster in Disaster Recovery situation.

So , this configuration is not suitable for us

Scenario 2: Regex pattern for producer and consumer topic names with *

1. Producer -> ‘*’ topic
2. Consumer -> ‘*’ topic

It solves our problem of not able to find topics in Target cluster but it comes with a new problem.

Issues:

It will produce / consume topics which it’s not supposed to as we gave * regex which matches with every topic in cluster.

So , this configuration also not suitable for us.

Scenario 3: Regex pattern for producer and consumer topic names with ‘*driver_topic’

1. Producer -> ‘*driver_topic’
2. Consumer -> ‘*driver_topic’

In this scenario everything is working good. Producer are producing to only driver related topics and Consumers are consuming only from driver topic.

Issues:

The problems come when we want bi directional replication between clusters.There is no backward replication support between topics for MirrorMaker. So, we won’t be able to replicate back.

So, we won’t be able to restore Source cluster from Target cluster post Disaster Recovery.

Now, you may think why isn’t MirrorMaker allow backward replication in above scenario 🤔? Let’s see what would happen if MirrorMaker allows backward replication

As you can see from above diagram, If MirrorMaker allows backward replication from replicated topics , it will result into infinite loop which results into cluster failure.

Scenario 4: Updating configuration of application when Disaster happens

We may think , why not update the application configuration on DR scenarios? Let’s see what happens if we decide that we will apply configuration whenever we switch between clusters to match topic names.

In non DR situation we can have like below:

1. Producer -> driver_topic
2. Consumer -> driver_topic

In DR situation like below

1. Producer -> Source.driver_topic
2. Consumer -> Source.driver_topic

Issues:-

This may solve reading from appropriate topics but in this scenario as well it doesn’t cover backward replication problem where we are unable to restore Source cluster form Target cluster

It also needs application configuration changes each time we switch to different cluster or application restart to get appropriate cluster topic names.This will also lead to application downtime.

Scenario 5: Add similar topics in Target cluster with replication enabled to Source cluster

Let’s say we create same topics in target cluster as well and enable replication back to source cluster.This solves our problem of not able to replicate back the data from target cluster but still this doesn’t solve our producer , consumer topic configuration.

Scenario 6: Add similar topics in Target cluster with replication enabled to Source cluster and have Producer and Consumer with different topic configuration

Let’s see a common topic configuration which works even if we switch between clusters. In this scenario we are keeping replication strategy from before but we are updating our producer and consumer with different topic configuration.

1. Producer -> driver_topic
2. Consumer -> *driver_topic

Non DR situation/using Source cluster:

Producer -> driver_topic.

Consumer -> driver_topic , Target.driver_topic (as we gave *driver_topic as topic configuration)

Consumer will not find any data in Target.driver_topic as we are using Source cluster and mainly get data from driver_topic

DR situation:

Producer -> driver_topic

Consumer -> driver_topic , Source.driver_topic

Consumer will find data in Source.driver_topic if any unconsumed messages are there before DR. Once Source.driver_topic becomes empty , consumers will mainly get data from driver_topic

DR switch back:

Producer -> driver_topic.

Consumer -> driver_topic , Target.driver_topic

Consumer will find data in Target.driver_topic if any unconsumed messages are there before switch back in Target. Once Target.driver_topic becomes empty , consumers will mainly get data from driver_topic

Conclusion:

In conclusion, the strategies presented in this guide offer significant advantages for Kafka MirrorMaker replication scenarios:

1. Simplified Configuration Management: By maintaining a common topic configuration for both normal and disaster recovery situations, the need for frequent application configuration changes is eliminated.
2. Elimination of Custom Logic: The necessity to implement custom replication logic to remove prefixes from topics is entirely avoided, streamlining the replication process.
3. Seamless Transition: The consistency in topic names between source and target clusters means that applications do not need to be restarted during topic name changes, ensuring a smooth transition.
4. Automated Bidirectional Replication: The automatic bidirectional replication capability ensures a worry-free cluster restoration process, even in complex disaster recovery scenarios.

By adopting these strategies, organisations can not only ensure zero-downtime replication but also simplify their configuration management and enhance the overall reliability of their Kafka clusters.