ReviseAlgo Logo

Schema Registry & Data Contracts

Schema Evolution and Compatibility Rules

Backward, forward, full compatibility modes and broker validation.

Introduction

Schema Evolution is the process of modifying event schemas over time as business requirements change. Centralized registries enforce compatibility rules (e.g. BACKWARD, FORWARD, FULL) to ensure updates do not break active clients.

Why It Matters

In production, microservices are deployed and upgraded independently. A producer might update its schema before consumers are ready, or a consumer might upgrade first. If compatibility rules are not strictly managed, schema evolution will result in message parsing exceptions and system downtime.

Real-World Analogy

Think of compatibility like updating a multiplayer video game. Backward compatibility ensures that a new server version can load save-files created by old clients. Forward compatibility ensures that old game client versions can still connect and play with the upgraded server. Full compatibility guarantees both are true, letting players connect in any order.

How It Works

The Schema Registry enforces compatibility modes when new schemas are submitted:

  • BACKWARD (Default): Consumers using the new schema can read data written with the old schema. You can delete fields, or add optional fields with default values. Upgrade consumers first.
  • FORWARD: Consumers using the old schema can read data written with the new schema. You can add new fields, or delete optional fields. Upgrade producers first.
  • FULL: Both backward and forward compatibility are enforced. You can only add or delete optional fields. Clients can be upgraded in any order.
  • NONE: Compatibility checks are disabled, allowing any schema change (not recommended for production).

Internal Architecture

When a client tries to register a new schema via POST requests to /subjects/{subject}/versions, the Schema Registry fetches the previous versions from the _schemas topic. It runs compatibility checks against the configured rules. If validation fails, the Registry blocks the write and returns a 409 Conflict response.

Visual Explanation

Below is a mapping table displaying schema reading safety across different registry configurations: 

Compatibility | New Consumer + Old Data | Old Consumer + New Data | Upgrade Order
──────────────┼─────────────────────────┼─────────────────────────┼──────────────
  BACKWARD    |          SAFE           |         BLOCKED         | Consumers 1st
  FORWARD     |         BLOCKED         |          SAFE           | Producers 1st
  FULL        |          SAFE           |          SAFE           |   Any order

Practical Example

Below are curl commands demonstrating how to configure and check compatibility rules on the Schema Registry dynamically: 

# Set global compatibility configuration to FULL
curl -X PUT -H "Content-Type: application/json" \
  --data '{"compatibility": "FULL"}' \
  http://localhost:8081/config

# Check compatibility config for a specific topic subject
curl -X GET http://localhost:8081/config/customer-events-value

# Manually test compatibility of a new schema change before registering
curl -X POST -H "Content-Type: application/json" \
  --data '{"schema": "{\"type\":\"record\",...}"}' \
  http://localhost:8081/compatibility/subjects/customer-events-value/versions/latest

Common Mistakes

  • Adding fields without defaults under BACKWARD: Adding a mandatory field without a default value. If a consumer using the new schema reads old data, it fails because it cannot find the new field, crashing the application.
  • Changing data types in-place: Attempting to modify a field type (e.g. changing an int to a string). This violates all compatibility settings, requiring topic recreation or subject naming strategies.

Quick Quiz

Q1: Under BACKWARD compatibility rules, which components should you upgrade first?

The consumers (since new consumers are guaranteed to read old data safely).

Q2: Why must you provide a default value when adding a field under BACKWARD compatibility?

So that the new consumer can fill in the missing field value when reading old events where the field did not exist.

Scenario-Based Challenge

The Challenge: Upgrading a shipping order schema to support international addresses

You need to add an optional country_code field to your order schema. The topic is configured with BACKWARD compatibility. How do you roll out this change without disrupting consumers?

Solution Tip: Define the country_code field in your schema with a default value (e.g., "US"). This ensures it complies with BACKWARD rules. Register the schema, then deploy the consumers before upgrading the producers.

Debugging Exercise

A consumer application crashes on startup with a SchemaProjectorException: Writer schema is incompatible with reader schema during a blue-green deployment.

The Fix: This means you deployed new producers (writing new schemas) before upgrading the consumers on a topic configured with BACKWARD compatibility. Roll back the producers, upgrade the consumers, and then re-deploy the new producers.

Interview Questions

1. Compare Backward and Forward compatibility modes.

Backward compatibility ensures new consumers can read old data. Forward compatibility ensures old consumers can read new data.

2. What are the compatibility requirements for setting a topic to FULL compatibility?

You can only add or delete optional fields, and all added/deleted fields must define default values.

Production Considerations

Use the Schema Registry Maven/Gradle plugin in your CI/CD pipelines to run compatibility checks during code builds before merging pull requests.