Kafka Connect
Source Connectors vs Sink Connectors
Ingesting from DBs and outputting to Elastic/S3/Warehouses.
Interview: Core Kafka Connect concept tested in data engineering and backend system design interviews — understanding data flow direction and common production connectors.
Introduction
Every Kafka Connect integration falls into one of two categories based on the direction of data flow: Source Connectors that pull data into Kafka, and Sink Connectors that push data out of Kafka. Understanding the distinction — and knowing which connectors exist for common systems — is essential for designing end-to-end data pipelines.
Source Connectors — Ingesting Into Kafka
A Source Connector reads data from an external system and publishes it as records to one or more Kafka topics. The source connector is responsible for polling the external system and tracking what data has already been sent, using Kafka Connect's offset infrastructure.
Data Flow Direction
External System ──────────────Kafka Topic
(MySQL, PostgreSQL, (append-only
S3, REST APIs, event log)
MongoDB, etc.)
Source Connector Ingestion Modes
Source connectors typically use one of three strategies to detect new or changed data:
| Mode | How It Works | Best For | Limitation |
|---|---|---|---|
| incrementing | Tracks a monotonically increasing integer column (e.g., id) |
Insert-only tables with auto-increment IDs | Cannot detect updates or deletes |
| timestamp | Tracks a timestamp column (e.g., updated_at) |
Tables with reliable last-modified timestamps | Can miss fast-changing rows; clock skew issues |
| CDC (Debezium) | Reads the database binary log (binlog/WAL) for every row-level change | Full change capture: inserts, updates, deletes | Requires DB configuration (binlog enabled) |
Example: JDBC Source Connector Configuration
Sink Connectors — Exporting From Kafka
A Sink Connector consumes records from one or more Kafka topics and writes them to an external system. It behaves like a Kafka consumer group internally, using standard Kafka offsets to track progress and supports exactly-once or at-least-once delivery depending on the target system's capabilities.
Data Flow Direction
Kafka Topic ──────────────External System
(event log) (Elasticsearch,
S3, BigQuery,
PostgreSQL, etc.)
Example: Elasticsearch Sink Connector
Popular Production Connectors Ecosystem
| System | Source Connector | Sink Connector |
|---|---|---|
| MySQL / PostgreSQL | Debezium CDC, JDBC Source | JDBC Sink |
| Elasticsearch | — | Elasticsearch Sink |
| Amazon S3 | S3 Source | S3 Sink (Parquet/JSON/Avro) |
| Google BigQuery | — | BigQuery Sink |
| MongoDB | MongoDB Source (Change Streams) | MongoDB Sink |
| Apache Cassandra | — | Cassandra Sink |
| Snowflake | — | Snowflake Sink |
| HTTP / REST APIs | HTTP Source | HTTP Sink |
Single Message Transforms (SMTs)
Both source and sink connectors support SMTs — inline record transformations applied without writing code. Common use cases:
Common Mistakes
| Mistake | Impact | Solution |
|---|---|---|
| Using JDBC source for CDC | Misses deletes and fast updates; causes data inconsistency | Use Debezium CDC connector for full change capture |
| Not handling tombstone records in sink | Delete events (null value records) crash or are silently dropped by sink connectors | Configure behavior.on.null.values=ignore or handle explicitly |
| Schema mismatch between source and sink | Sink connector throws deserialization errors | Use Schema Registry with Avro to enforce compatibility |
Quick Quiz
Q1: You need to replicate MySQL order records — including updates and deletes — into Elasticsearch in real time. Which connector type and which ingestion mode should you use?
Use a Debezium MySQL Source Connector (CDC mode) to capture all row-level changes from the MySQL binlog and publish them to Kafka. Then use an Elasticsearch Sink Connector to consume from that topic and index documents. JDBC source in timestamp mode would miss deletes.
Q2: A sink connector is consuming records from Kafka but the target Elasticsearch cluster is temporarily unavailable. What happens?
The sink connector tasks will retry based on configured retry policies. Kafka consumer offsets will not advance (not committed) until the records are successfully written to Elasticsearch, ensuring at-least-once delivery. The task enters a FAILED state after max retries, which the Connect REST API will report.
Scenario-Based Challenge
Design a Data Archival Pipeline
Your company generates 500,000 user events per minute. These events flow through Kafka. You need to: (1) archive all events to S3 in Parquet format partitioned by date, and (2) make the last 24 hours searchable in Elasticsearch. Design the connector configuration.
View Solution
Deploy two sink connectors consuming the same Kafka topic (user-events): (1) S3 Sink Connector with format.class=io.confluent.connect.s3.format.parquet.ParquetFormat and partition.duration.ms=86400000 (daily partitions). (2) Elasticsearch Sink Connector with an ILM policy to auto-delete indices older than 24 hours. Both connectors independently track their own consumer group offsets in Kafka — neither interferes with the other.
Interview Questions
1. What is the difference between a source connector and a sink connector?
A source connector reads data from an external system and publishes it to Kafka topics (external → Kafka). A sink connector reads from Kafka topics and writes to an external system (Kafka → external). They differ in data flow direction, offset semantics, and failure behavior.
2. Why is CDC (Debezium) preferred over JDBC polling for database replication?
JDBC polling can only detect changes by querying the current state, which misses hard deletes and requires a reliable timestamp/ID column. CDC reads the database transaction log (binlog/WAL), capturing every insert, update, and delete at the row level with sub-second latency, making it the only reliable approach for full data replication.