ReviseAlgo Logo

Security & Multi-Cluster Deployments

Mirror Maker 2 — Cross-Cluster Replication

Active-passive and active-active setups for disaster recovery.

Introduction

For mission-critical applications, relying on a single datacenter is a single point of failure. If an entire region goes offline, your stream processing halts. MirrorMaker 2 (MM2) is Kafka's primary tool for replicating data streams and translating consumer offsets across distinct physical Kafka clusters, enabling cross-region disaster recovery and high availability.

Why It Matters

Standard Kafka topic replication only operates within the boundaries of a single cluster. MirrorMaker 2 spans datacenters, allowing architectures to implement **Active-Passive** (disaster recovery failover) or **Active-Active** (global read-write locality) designs. Furthermore, MM2 translates consumer group committed offsets between clusters, allowing consumer applications to fail over without re-processing historical data or skipping messages.

Real-World Analogy

Think of MirrorMaker 2 like a global shipping company syncing warehouse catalogs. You have a main catalog warehouse in New York (Source) and a backup in London (Target). A team of runners (MirrorMaker Connectors) constantly watches the New York shelves and packs exact copies of incoming packages to ship over to London (Topic Replication). Additionally, they keep track of which box the London customers had last finished reading (Offset Translation) so if New York suffers a blackout, London customers can resume reading from the exact same box.

How It Works

MirrorMaker 2 is built on the **Kafka Connect** framework and deploys three dedicated connectors:

  1. MirrorSourceConnector: Replicates topics, partitions, and message logs from the source cluster to the destination cluster, prefixing topic names (e.g. source.topic_orders) to prevent circular loops.
  2. MirrorCheckpointConnector: Regularly translates and propagates consumer group offsets between clusters, writing checkpoints to the destination metadata store.
  3. MirrorHeartbeatConnector: Emits periodic heartbeat records to verify link connectivity and latency between source and target clusters.

Internal Architecture

Because partition offsets are not identical across clusters (due to timing, partition count changes, or record drops), consumer groups cannot simply use their raw source offset on failover. The MirrorCheckpointConnector translates offsets dynamically. It correlates timestamps and mapping offsets, translating consumer progress (e.g. offset 1050 in Source translates to offset 1012 in Target), writing this mapping data to the destination's __consumer_offsets topic.

Visual Explanation

Below is an ASCII representation showing how MirrorMaker 2 replicates data logs and translates consumer progress between source and target clusters: 

 [Source Cluster (US-East)]                [Target Cluster (US-West)]
┌──────────────────────────┐              ┌──────────────────────────┐
│ Topic A (Log End: 105)   │ ──(MM2 Log)─>│ us-east.Topic A (Log: 98)│
│                          │              │                          │
│ Consumer Group Alice     │ ──(MM2 Off)─>│ (Offset translated to    │
│  - Committed Offset: 80  │              │  Group Alice Target: 75) │
└──────────────────────────┘              └──────────────────────────┘

Practical Example

Here is a configuration file connect-mirror-maker.properties setting up MirrorMaker 2 in dedicated mode to replicate data from a primary cluster (us-east) to a backup cluster (us-west):

Common Mistakes

  • Disabling topic naming prefixes in active-active topologies: Attempting to replicate with identical topic names (e.g. Source orders to Target orders) when replicating both directions. This triggers endless echo replication loops, quickly overloading brokers. Always preserve prefixes.
  • Forgetting to sync offset commit states on client libraries: Failing to verify that target clusters have the sync.group.offsets.enabled option set to true. During DR failover, consumers will restart from offset 0 or the latest record, causing data duplication or loss.

Quick Quiz

Q1: Why are partition offsets not exactly identical between replicated topics on source and target clusters?

Because record delivery order, replication latency, network failures, or varying partition allocations can alter the write sequence in the target log.

Q2: What is the role of the MirrorCheckpointConnector in MM2?

It monitors consumer group committed offsets on the source cluster, translates them to the corresponding target cluster partition offsets, and commits them.

Scenario-Based Challenge

The Challenge: Executing a seamless failover and failback for a payment processing group

Your primary region (US-East) experiences a fiber cut, and you fail over consumers to US-West. After US-East recovers, you need to fail them back to US-East. Since they processed data on US-West in the meantime, how do you prevent re-processing identical payments during failback?

Solution Tip: Ensure MirrorMaker 2 runs bidirectionally (US-East <-> US-West) with offset synchronization enabled on both sides. When US-East recovers, MM2 will replicate the newly processed payment offsets back to US-East. Stop consumers on US-West, wait for the replication queue to drain, and then safely start consumers on US-East.

Debugging Exercise

During DR drill execution, your consumer groups fail over to the target cluster, but they start reading from the beginning of the topic (offset 0), processing thousands of historical duplicate records.

The Fix: Check the MirrorCheckpointConnector configuration. Verify that sync.group.offsets.enabled is set to true and the consumer group name patterns are matched by the property sync.group.offsets.groups. Confirm that the MM2 process has sufficient ACL permissions to write offsets to target consumer groups.

Interview Questions

1. How does MirrorMaker 2 prevent infinite circular loops in bidirectional topologies?

MM2 automatically prepends the source cluster alias prefix to replicated topic names (e.g. us-east.orders). MM2 connectors will not pull or replicate topics containing their own cluster alias prefix.

2. What is the difference between active-passive and active-active multi-cluster topologies?

Active-Passive replicates data unidirectionally from a live source to a standby target (read-only standby). Active-Active replicates data bidirectionally, allowing clients to read and write locally to both regions simultaneously.

Production Considerations

Deploy MirrorMaker 2 close to the destination cluster to optimize write latencies. Always monitor network throughput constraints and scale MirrorMaker Connect tasks across multiple worker nodes to handle peak traffic.