Serialization and Kafka wire formats

Kafka stores bytes, not application objects

A serializer converts a typed key or value into bytes; a deserializer reverses that operation. Key and value are separate boundaries and may use different formats. Headers are bytes too.

Serialized key bytes influence partition selection. UTF-8 text 42 is not the same key as a four-byte integer 42, so changing key serialization can move related records and disrupt ordering, joins or compaction.

UTF-8 string "42"  = 34 32
32-bit integer 42  = 00 00 00 2a
Avro long 42       = 54

Use JSON as a text wire format

JSON on Kafka is normally UTF-8 text. It is easy to inspect and interoperable, but the bytes do not identify a schema version unless an additional framing or contract system is used.

Whitespace, member order and numeric spelling can produce different bytes for semantically similar documents. Large integers can also lose precision in runtimes that map every JSON number to floating point.

Avro binary requires a writer schema

Avro binary omits most field names and separators because the schema already defines structure. Records are encoded in schema order, integers use variable-length encoding and unions include a branch index.

Raw Avro datum, Registry-framed Avro, Avro Object Container File and Avro single-object encoding are distinct formats. A consumer built for one cannot assume another is compatible.

Recognise Registry framing conceptually

The common Registry frame starts with a format marker, then a four-byte schema ID, then the encoded datum. The full schema is not copied into every record. Consumers use the ID against the expected Registry environment.

A leading zero byte is only a clue. Confirm that the next bytes resolve to a real ID and that its schema can decode the remainder before concluding a payload is Registry framed.

+------------+------------------+-------------------+
| marker     | 4-byte schema ID | Avro binary datum |
+------------+------------------+-------------------+

Match dFlow~IQ formats to the consumer contract

Base64 and hex are input notations for bytes, not extra wrappers. If a consumer expects the literal characters aGVsbG8=, use String; choosing Base64 sends the decoded bytes for hello.

Debug from exact bytes outward

1. Separate key and value and identify which side failed.
2. Capture topic, partition, offset, lengths, headers and null status for one exact record.
3. Inspect raw hex or Base64 before trying another structured decoder.
4. Verify the producer's actual serializer configuration and release version.
5. For Registry data, resolve the frame's ID in the correct Registry environment.
6. Use the exact writer schema, not simply the latest subject version.
7. Check primitive width and byte order.
8. Compare old and new offsets for mixed-format topic history.

Produce safely in dFlow~IQ

1. Confirm workspace, cluster and topic before opening the producer.
2. Inspect existing records to identify key and value formats.
3. Select key and value serializers independently.
4. For Registry-backed data, select and review the intended subject and version.
5. Validate JSON, Avro types, enum symbols, unions and numeric ranges locally.
6. Send one canary and wait for its broker acknowledgement.
7. Read the exact partition and offset back with the intended deserializer.
8. Observe downstream errors before using Batch Upload.