Kafka Schema Evolution: A Complete Guide
Schema evolution is one of the most underestimated challenges in Kafka deployments. One incompatible schema change can silently corrupt data pipelines for days. This guide covers every compatibility mode, migration strategy, and pitfall you need to know.
Why Schema Evolution Is Hard
In a traditional database, you run an ALTER TABLE statement and all queries instantly see the new schema. Kafka topics are different: messages are immutable and can persist for days, weeks, or indefinitely. Producers and consumers often deploy independently. This creates a distributed schema compatibility problem.
The Three-Party Problem
Compatibility Modes Explained
BACKWARD (Default)
RecommendedNew schema can read data written with the previous schema. Consumers can be upgraded before producers. This is the most common and safe default for most teams.
- Delete a field
- Add an optional field with default
- Add required field without default
- Change field type
- Rename a field
FORWARD
Old schema can read data written with the new schema. Producers can be upgraded before consumers. Useful when consumers are slower to deploy.
- Add a new field (ignored by old consumers)
- Delete an optional field
- Delete a required field
- Change field type
FULL
Most RestrictiveBoth backward and forward compatible. Producers and consumers can be upgraded in any order. Only adding optional fields with defaults and deleting optional fields is permitted. Recommended for shared platform topics consumed by many independent teams.
NONE
Use with CautionNo compatibility checks. Schema Registry accepts any schema update. Appropriate only for development environments or when you own all producers and consumers and can coordinate their deployments atomically.
Safe Schema Evolution Examples
Adding an Optional Field (Safe)
{
"type": "record",
"name": "Order",
"fields": [
{"name": "id", "type": "string"},
{"name": "amount", "type": "double"},
{"name": "status", "type": "string"}
]
}{
"type": "record",
"name": "Order",
"fields": [
{"name": "id", "type": "string"},
{"name": "amount", "type": "double"},
{"name": "status", "type": "string"},
{
"name": "currency",
"type": ["null", "string"],
"default": null
}
]
}Renaming a Field (Requires Strategy)
Direct renames break all existing consumers. The safe approach is a deprecation cycle:
// Step 1: Add new field alongside old field (deploy producers)
{"name": "user_id", "type": "string"}, // keep old
{"name": "customer_id","type": ["null","string"],"default": null} // add new
// Step 2: Update consumers to read both fields, prefer new
const customerId = record.customer_id ?? record.user_id;
// Step 3: Producers write both fields
// Step 4: After all consumers deployed, remove user_id
// (requires FORWARD or NONE compatibility temporarily)Breaking Change Migration Strategies
Strategy 1: New Topic + Dual Write
The safest approach for incompatible changes. Create a new topic with the new schema, have producers write to both topics simultaneously, and migrate consumers one by one.
# Topic naming convention for versioned topics orders.v1 # existing consumers stay here orders.v2 # new consumers target this topic # After all consumers migrated: # 1. Stop dual writes to orders.v1 # 2. Archive or delete orders.v1 after retention period
Strategy 2: Schema Registry Compatibility Override
Temporarily set compatibility to NONE, perform the migration, and restore the target mode. This requires coordinating all producer and consumer deployments within a tight window.
# Override compatibility for a single subject
curl -X PUT http://schema-registry:8081/config/orders-value \
-H "Content-Type: application/json" \
-d '{"compatibility": "NONE"}'
# Register new schema
# Deploy all producers and consumers atomically
# Restore compatibility mode
curl -X PUT http://schema-registry:8081/config/orders-value \
-H "Content-Type: application/json" \
-d '{"compatibility": "BACKWARD"}'Common Pitfalls to Avoid
Registering schemas manually in production
Schema registration should be part of your CI/CD pipeline with compatibility checks. Manual registration bypasses safety checks and can break consumers immediately.
Using NONE compatibility globally
Setting NONE at the registry level removes all guardrails. Scope NONE overrides to individual subjects and restore BACKWARD after the migration.
Assuming Protobuf is always safer than Avro
Protobuf field numbers create their own evolution traps. Reusing a deleted field number with a different type is a silent data corruption bug.
Not versioning the topic name
Topics like 'orders' with no version make it impossible to do clean breaking changes. Establish a naming convention (orders.v1, orders.v2) early.
Ignoring schema documentation fields
Adding doc fields to schemas is not free — some serialization formats include them in the wire format, subtly changing the schema fingerprint.
Key Takeaways
Manage Schema Evolution with KLogic
KLogic's Schema Registry integration gives you a visual history of every schema version, compatibility mode settings per subject, and alerts when a compatibility check fails — before it reaches production.
Request a Demo