Schema Registry and safe contract evolution

A schema is structural, not the whole contract

Kafka stores bytes; a schema defines how structured data becomes and returns from those bytes. It can describe field names, types, records, enum symbols and defaults. It cannot fully express ownership, business meaning, data classification or the unit represented by an amount.

Compatibility therefore proves a structural relationship, not semantic safety. Changing an amount from pounds to pence can pass a structural check while still breaking every consumer.

Distinguish subjects, versions and IDs

A subject groups an ordered schema history and owns a compatibility policy. A version is a subject-local revision number. A schema ID identifies registered schema content in the Registry and is what Registry-aware records normally carry on the wire.

Subject naming strategy matters. Topic-based naming commonly creates separate <topic>-key and <topic>-value subjects; record-based strategies group schemas differently. Confirm the producer strategy instead of inferring a subject from the topic name.

Follow writer-schema lookup

A Registry-aware producer resolves or registers its writer schema, receives a schema ID, encodes the datum and prefixes the payload with Registry framing. Kafka stores those bytes without inspecting the schema.

A consumer reads the ID, obtains the writer schema and decodes the datum. It may then resolve the writer schema into its own reader schema. The record usually does not contain its subject or subject version.

Read compatibility in the right direction

Backward means a new reader can consume data written with the previous schema. Forward means an old reader can consume data written with the new schema. Full requires both directions. Transitive variants check every prior version instead of only the latest.

Deployment order follows the direction: backward compatibility supports upgrading consumers before producers; forward compatibility supports upgrading producers before consumers. Full compatibility gives structural flexibility in either order, but still requires behavioural testing.

Apply Avro evolution rules deliberately

• Adding a field with a valid default supports a new reader consuming old data.
• Fields present only in the writer are ignored by readers that do not define them.
• Numeric promotions such as int to long are directional, not interchangeable.
• Adding an enum symbol can break an old reader if the new symbol is produced.
• A default is used by a reader during resolution; it does not mean a new writer may omit the field.
• Aliases can assist renames in one direction but do not automatically make a rename fully compatible.

{
  "name": "hubCode",
  "type": ["null", "string"],
  "default": null
}

Review a proposal in dFlow~IQ

1. Open Schema Registry → Schemas and select the subject.
2. Confirm the compatibility mode and the currently deployed version.
3. Use Text view for exact fields, defaults, unions and enum changes.
4. Use Graph view to follow nested records and shared named types.
5. Compare the candidate with the latest version and, for transitive requirements, older relevant versions.
6. Open Validate proposal and run the Registry compatibility check.
7. Test mixed-version traffic before changing producers.

Prefer staged migration over false defaults

A direct field rename is rarely safe in both directions. A safer pattern is to introduce the new field, dual-populate old and new fields, move consumers, wait for replay requirements to pass, and remove the old field later.

Do not invent a convenient default such as zero or UNKNOWN when it changes meaning. If no truthful compatibility bridge exists, a new topic and contract version can be safer than weakening Registry checks.

Avoid common contract failures

• Treating compatibility as proof that business meaning is unchanged.
• Checking only the latest version when consumers replay older retained data.
• Deleting schema history while Kafka records still refer to it.
• Letting production serializers auto-register unreviewed schemas.
• Ignoring key-schema changes that affect partitioning.
• Reviewing only the root file and missing referenced-schema changes.