Kafka Connect
What is Kafka Connect and Why It Exists
Codeless data integration framework overview.
Interview: Frequently asked in backend and data engineering interviews to assess your understanding of scalable, configuration-driven data integration without writing custom producer/consumer code.
Introduction
Every real-world Kafka deployment needs to move data between Kafka and the outside world — databases, data warehouses, search engines, object stores, and legacy systems. Writing custom Kafka producers and consumers for every data source or destination is tedious, error-prone, and hard to operate at scale.
Kafka Connect is a scalable, fault-tolerant, and configuration-driven framework built into the Apache Kafka ecosystem for streaming data between Kafka and external systems. Instead of writing code, you declare your integration in a JSON or properties file, and Kafka Connect handles the rest — including offset tracking, schema evolution, error handling, and parallel scaling.
The Problem Kafka Connect Solves
Before Kafka Connect, engineers had to manually write, maintain, and operate integration code for every data pipeline:
- Custom producers to read from MySQL and publish to Kafka.
- Custom consumers to read from Kafka and write to Elasticsearch.
- Manual offset management to track which rows or CDC events had already been processed.
- No standardized error recovery — each team implemented retries and dead-letter queues differently.
- No built-in scaling — increasing throughput required manually spawning more consumer instances.
Kafka Connect eliminates all of this boilerplate by providing a plug-in architecture where connectors (pre-built libraries) handle the logic, and operators simply provide configuration.
Real-World Analogy
Think of Kafka Connect like a USB standard for data systems. Before USB, every peripheral (keyboard, mouse, printer) needed its own custom port and driver. USB created a universal standard so that any peripheral just "plugs in" and works. Similarly, Kafka Connect defines a universal standard for data connectors — any system that implements the Connect API can be plugged into Kafka without writing custom integration code.
Core Concepts
| Concept | Description |
|---|---|
| Connector | A reusable plugin (JAR file) that knows how to communicate with a specific external system (e.g., JDBC, S3, Debezium). |
| Task | A unit of work spawned by a connector. Multiple tasks run in parallel for throughput. |
| Worker | A JVM process that hosts and executes one or more connectors and their tasks. |
| Offset | A persistent record of how far a source connector has read, stored in a Kafka internal topic. |
| Transforms (SMTs) | Single Message Transformations — lightweight, configuration-driven record transformations applied inline. |
How It Works
- Deploy a Worker Cluster: Start one or more Kafka Connect worker JVM processes, configured to connect to your Kafka cluster.
- Submit a Connector Config: POST a JSON configuration to the Connect REST API specifying the connector class, topics, and system-specific settings.
- Connector Spawns Tasks: The connector plugin determines how many parallel tasks to create and assigns work to each one.
- Tasks Run Continuously: Source tasks poll the external system, produce records to Kafka. Sink tasks consume from Kafka, write to the target system.
- Offsets are Committed: Progress is checkpointed automatically to internal Kafka topics, enabling fault-tolerant resume after failure.
Deploying a Simple Connector
Here is a minimal example of starting a Kafka Connect worker and deploying the built-in FileStream source connector via the REST API:
Any new line appended to /tmp/demo-input.txt will automatically appear as a record in the connect-demo Kafka topic — no Java code required.
Key Advantages Over Manual Code
| Feature | Manual Code | Kafka Connect |
|---|---|---|
| Offset Tracking | Developer must implement manually | Built-in, stored in Kafka topics |
| Parallel Scaling | Manual instance management | Configure tasks.max, auto-balanced |
| Fault Recovery | Developer must implement retries | Auto-restart failed tasks on healthy workers |
| Schema Handling | Custom serialization code | Native Schema Registry integration |
| Monitoring | Custom metrics implementation | Built-in JMX and REST status endpoints |
Common Mistakes
- Using standalone mode in production: Standalone mode stores offsets locally on disk. If that worker node fails, progress is lost. Always use distributed mode in production.
- Ignoring
tasks.max: Leaving it at the default of 1 creates a serial bottleneck. Set it equal to the number of partitions in your source topic for maximum parallelism. - Conflating connectors with tasks: A connector is the logical configuration. Tasks are the actual JVM threads doing work. One connector can spawn many tasks.
Quick Quiz
Q1: What is the primary benefit of Kafka Connect over writing custom Kafka producers/consumers?
Kafka Connect provides built-in offset tracking, parallel task scaling, automatic fault recovery, and standardized schema handling — all without writing custom integration code. Operators only need to provide a JSON configuration file.
Q2: Where does Kafka Connect store its source connector offsets?
In internal Kafka topics (typically named connect-offsets), making offset state durable and accessible by any worker in the cluster — enabling fault-tolerant resume.
Scenario-Based Challenge
The Challenge: Migrate from custom consumer to Kafka Connect
Your team maintains a Python script that reads from a PostgreSQL table every 5 minutes and publishes new rows to a Kafka topic. The script has a bug: if it crashes mid-batch, it re-processes the entire batch on restart. How would Kafka Connect solve this, and what connector would you use?
View Solution
Use the Confluent JDBC Source Connector with mode=incrementing or mode=timestamp+incrementing. The connector automatically tracks the last-seen row ID or timestamp in a Kafka offset topic. On restart, it resumes exactly from where it left off, eliminating re-processing bugs entirely — no code changes required.
Interview Questions
1. Conceptual: What is Kafka Connect, and how does it differ from writing a raw Kafka producer?
Kafka Connect is a framework for building and running connector plugins — reusable integration components. A raw producer is custom application code that manually handles serialization, offset tracking, and error recovery. Kafka Connect provides all of those capabilities declaratively through configuration, allowing engineers to integrate external systems without writing Java boilerplate.
2. Design: Your company integrates with 15 external data systems. Why is Kafka Connect more operationally scalable than 15 separate custom producer services?
Kafka Connect centralizes offset management, error handling, monitoring, and scaling into a single cluster. Each integration is a declarative config, not a separate deployable service. Operational changes (restart, scale, reconfigure) are made through a unified REST API rather than deploying 15 separate codebases.