ReviseAlgo Logo

Schema Registry & Data Contracts

Avro, Protobuf, and JSON Schema — Which to Choose

Compare serialization efficiency and developer workflow.

Introduction

When storing structured events in Kafka, developers must select a serialization format. The three primary options supported by the Confluent Schema Registry are Apache Avro, Protocol Buffers (Protobuf), and JSON Schema.

Why It Matters

Selecting the wrong serialization format leads to high CPU utilization, excessive network bandwidth consumption, and complex code generation pipelines. Engineers must weigh the trade-offs of each format in terms of serialization speed, payload sizes, and developer tooling.

Real-World Analogy

Think of JSON Schema like a printed tax form where every single box has a written label (e.g. "Age: 30"). It is easy to read but uses a lot of paper. Think of Avro/Protobuf like a dense grid card containing values only (e.g., "John", "Smith", "30"). To read the card, you must overlay a transparent grid sheet (the schema) that defines what each cell represents.

How It Works

The three formats validate and structure binary bytes differently:

  • Apache Avro: A row-oriented binary format. It relies entirely on schema definitions, writing only value bytes to the stream, making it highly compact.
  • Protobuf: A tagged binary format developed by Google. Records use field numbers (tags) to identify fields, supporting fast, backward-compatible parsing.
  • JSON Schema: Validates human-readable JSON payloads. It is easy to inspect but consumes significant network bandwidth because keys are serialized with every message.

Internal Architecture

Avro stores no type or field tags in the serialized payload, which makes it impossible to parse without the exact reader schema. Protobuf, by contrast, encodes numeric tag IDs alongside binary values, allowing consumers to skip unrecognized fields during deserialization without needing the exact schema version.

Visual Explanation

Below is an ASCII representation comparing payload size overhead across the three serialization formats: 

JSON Schema Payload:
{"user_id": "usr-1", "age": 30, "status": "active"} ──> (51 Bytes, heavy keys)

Protobuf Binary Payload:
[Tag 1][usr-1][Tag 2][30][Tag 3][active]            ──> (22 Bytes, medium tags)

Avro Binary Payload:
[usr-1][30][active]                                 ──> (15 Bytes, values only!)

Practical Example

Below is a comparison of Avro IDL and Protobuf file structures for defining a matching customer registration contract:

Common Mistakes

  • Manually embedding Avro schemas in payloads: Writing the schema text into the Kafka record payload itself instead of utilizing the Schema Registry. This negates all serialization efficiency benefits, bloating disk usage.
  • Reordering Protobuf tag numbers: Changing the tag numbers in a .proto file (e.g. changing id = 1 to id = 2) during schema updates. This scrambles field decoding for existing consumers, corrupting data.

Quick Quiz

Q1: Which serialization format yields the smallest binary payload size by omitting field tags entirely?

Apache Avro.

Q2: Why is modifying tag numbers in a Protobuf file considered a breaking change?

Because Protobuf uses tag numbers to match serialized bytes to message fields; reordering tags results in mismatched field decoding.

Scenario-Based Challenge

The Challenge: Choosing a serialization standard for high-frequency IoT sensors

You design an IoT gateway processing 500K sensor updates per second. Devices are low-power and communicate over cellular networks with strict data limits. Which format do you choose?

Solution Tip: Choose Apache Avro. By using Schema Registry, Avro sends only the raw sensor reading value bytes preceded by a 5-byte header, minimizing cellular bandwidth overhead and reducing network costs compared to JSON Schema or Protobuf.

Debugging Exercise

Your Java build fails during compilation with the error: cannot find symbol: class Customer, even though you just defined the schema file in your project folders.

The Fix: Define code generation plugins in your build config. For Avro, configure the avro-maven-plugin or Gradle equivalent. For Protobuf, integrate the protobuf-maven-plugin. Execute mvn clean compile to trigger the compiler tool and generate the Java classes.

Interview Questions

1. What are the key differences between Avro and Protobuf serialization?

Avro relies on the schema to decode values and stores no field tags. Protobuf stores tag identifiers in the binary stream, allowing decoders to skip unknown fields.

2. When would you prefer JSON Schema over binary options like Avro?

When human readability and immediate debuggability of raw topic events are prioritized over network and storage optimization.

Production Considerations

Use Avro or Protobuf for core domain events. Integrate code generation checks directly into your CI/CD pipelines to prevent malformed schema code commits.