Kafka Connect
Kafka Connect Architecture — Workers, Tasks, Connectors
Distributed vs standalone mode, tasks scaling, and failover.
Interview: High-value interview topic for senior backend and platform engineering roles — tests your understanding of how Kafka Connect achieves scalability, fault tolerance, and operational simplicity.
Introduction
Kafka Connect's power comes not just from its connector ecosystem, but from its distributed, fault-tolerant runtime architecture. Understanding the three-layer model — Workers, Connectors, and Tasks — is essential for operating Connect clusters in production and explaining its design in interviews.
The Three-Layer Architecture
┌─────────────────────────────────────────────────────────────────┐
│ KAFKA CONNECT CLUSTER │
│ │
│ ┌───────────────────┐ ┌───────────────────┐ │
│ │ Worker JVM 1 │ │ Worker JVM 2 │ │
│ │ │ │ │ │
│ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │
│ │ │ Connector A │ │ │ │ Connector B │ │ │
│ │ │ (config) │ │ │ │ (config) │ │ │
│ │ └──────┬──────┘ │ │ └──────┬──────┘ │ │
│ │ │ │ │ │ │ │
│ │ ┌──────▼──────┐ │ │ ┌──────▼──────┐ │ │
│ │ │ Task 0 │ │ │ │ Task 0 │ │ │
│ │ │ Task 1 │ │ │ │ Task 1 │ │ │
│ │ │ Task 2 │ │ │ │ Task 2 │ │ │
│ │ └─────────────┘ │ │ └─────────────┘ │ │
│ └───────────────────┘ └───────────────────┘ │
│ │
│ Internal Kafka Topics: connect-configs, connect-offsets, │
│ connect-status │
└─────────────────────────────────────────────────────────────────┘
Layer 1: Workers
A Worker is a JVM process running the Kafka Connect runtime. It is the compute unit of the Connect cluster. Workers are responsible for:
- Hosting and executing connector instances and their tasks.
- Exposing the REST API on port 8083 for connector management.
- Participating in group rebalancing when workers join or leave the cluster.
- Reading connector configurations from the
connect-configsinternal topic. - Publishing status updates to the
connect-statusinternal topic.
Workers in the same cluster share the same group.id configuration, forming a Kafka consumer group that uses standard Kafka rebalancing protocols to distribute work.
Layer 2: Connectors
A Connector is a logical entity — it represents the configuration and plugin implementation for a specific integration. The connector does NOT directly process data. Its responsibilities are:
- Configuration validation: Checks that required config properties are present and valid.
- Task planning: Determines how many tasks to create and how to split work among them.
- Task configuration generation: Provides each task with its specific partition/table/shard assignment.
Layer 3: Tasks
Tasks are the actual units of work. Each task runs in a worker thread and directly handles data movement. For a JDBC source connector with 3 tables and tasks.max=3, three tasks might each handle one table. Tasks are the parallelism unit — more tasks = more throughput.
| Concept | Worker | Connector | Task |
|---|---|---|---|
| What it is | A JVM process | A logical plugin + config | A thread doing actual I/O work |
| Processes data? | No (hosts tasks) | No (plans tasks) | Yes |
| Scaled by | Adding new JVM instances | One per integration (not scaled) | Increasing tasks.max |
| Failure impact | Its tasks rebalanced to other workers | Connector reconfigured | Task restarted on a healthy worker |
Distributed vs Standalone Mode
| Property | Standalone Mode | Distributed Mode |
|---|---|---|
| Worker Count | Single worker process | Multiple workers forming a cluster |
| Offset Storage | Local file on disk (not durable) | Kafka topic (durable, replicated) |
| Config Storage | Local .properties file | Kafka topic (shared, REST-managed) |
| Fault Tolerance | None — single point of failure | Auto-rebalances tasks on worker failure |
| REST API | Limited | Full REST API on all workers |
| Use Case | Local development, testing | All production deployments |
Internal Kafka Topics
Kafka Connect uses three internal Kafka topics to coordinate the cluster. These topics must use high replication factors (typically 3) in production:
Fault Tolerance and Rebalancing
When a worker fails, the Connect cluster automatically redistributes its tasks to the remaining healthy workers:
Before failure:
Worker-1: [MySQL-Source Task-0, MySQL-Source Task-1]
Worker-2: [MySQL-Source Task-2, ES-Sink Task-0]
Worker-3: [ES-Sink Task-1, ES-Sink Task-2]
Worker-1 fails → Rebalance triggered:
Worker-2: [MySQL-Source Task-0, MySQL-Source Task-1, MySQL-Source Task-2, ES-Sink Task-0]
Worker-3: [ES-Sink Task-1, ES-Sink Task-2]
Tasks resume from last committed offset — no data loss.
Scaling Tasks for High Throughput
Common Mistakes
- Setting
tasks.maxhigher than available partitions: Excess tasks remain idle. The effective parallelism is bounded by the number of source partitions (tables, shards) or Kafka topic partitions for sink connectors. - Low replication factor on internal topics: If
connect-offsetsloses data due to broker failure with replication-factor=1, source connectors lose their position and re-process data from the beginning. - All workers on the same host: Defeats the purpose of distributed mode. Spread workers across different availability zones for true fault tolerance.
Quick Quiz
Q1: You have a Connect cluster with 3 workers and a connector with tasks.max=6. Worker-2 fails. What happens?
Kafka Connect detects the worker failure via the consumer group heartbeat mechanism. A rebalance is triggered, and the tasks that were running on Worker-2 are redistributed to Worker-1 and Worker-3. Each reassigned task resumes from the last committed offset stored in the connect-offsets topic — ensuring exactly-once or at-least-once processing depending on configuration.
Q2: What is the role of the connect-configs internal topic?
It stores all connector configurations as JSON records. When you POST a new connector via the REST API, the configuration is written to this topic. Any worker in the cluster can read from it, enabling any worker to host any connector — enabling seamless rebalancing.
Scenario-Based Challenge
Production Incident: Connector FAILED state
Your Elasticsearch sink connector transitions to FAILED state at 3 AM. The on-call alert fires. How do you diagnose and recover?
View SolutionStep 1 — Check connector status: GET /connectors/es-sink/status — look at both connector state and individual task states and error messages.
Step 2 — Check worker logs: grep -i "error|exception" connect-worker.log — look for deserialization errors, connection refused, or schema mismatch exceptions.
Step 3 — Restart failed tasks: POST /connectors/es-sink/tasks/0/restart (per task) or POST /connectors/es-sink/restart (all).
Step 4 — Configure dead-letter queue: Set errors.deadletterqueue.topic.name=es-sink-dlq so poison-pill records don't permanently block the connector.
Interview Questions
1. Explain the difference between a Worker, a Connector, and a Task in Kafka Connect.
A Worker is a JVM process that hosts the Connect runtime. A Connector is a logical configuration + plugin that defines an integration (but does not process data). A Task is the actual unit of work — a thread that polls data from a source or writes to a sink. One connector can spawn many tasks for parallelism, and those tasks run on workers.
2. How does Kafka Connect ensure fault tolerance without a centralized coordinator?
Kafka Connect uses Kafka's own consumer group protocol for worker coordination — no separate ZooKeeper or coordinator process is needed. All state (configurations, offsets, status) is stored in replicated Kafka topics. When a worker fails, surviving workers detect it via heartbeat timeout, trigger a rebalance, and take over the failed worker's tasks by reading their last committed offsets from the connect-offsets topic.