Kafka fundamentals: records, partitions and consumer position

Think in distributed event logs

Kafka stores events in append-oriented logs. Producing adds a record to a partition; consuming reads that record without removing it. Retention and compaction decide when stored data can be removed, independently of any consumer's progress.

This allows several applications to read the same stream at different speeds. A fraud detector, fulfilment service and analytics pipeline can each keep a separate position without competing for a single shared copy.

Understand the record boundary

A Kafka record has a key, value, timestamp and optional headers. Kafka stores keys and values as bytes; serializers give those bytes meaning. Topic, partition and offset identify the stored position after the broker accepts it.

Keys commonly represent an entity such as an order or account. Reusing a key is normal: it creates an ordered history for that entity when the records map to the same partition.

topic: orders.events.v1
key: customer-42
headers:
  event-type: OrderPaid
  trace-id: 96a20d4c8fb74136
value:
  {"orderId":"ord-1842","amountMinor":7995,"currency":"GBP"}

Ordering exists inside a partition

A topic is divided into partitions. Each partition is an independent ordered log and can live on a different broker. Partitioning creates storage and processing parallelism, but it also defines the boundary of Kafka's ordering guarantee.

Two offsets from different partitions cannot be compared to establish business order. If entity order matters, choose a stable key that keeps related events together. Increasing the partition count later can change default key-to-partition mapping for new records.

Separate log offsets from consumer progress

An offset is a partition-local position assigned when a record is appended. The log start offset marks the earliest retained position; the log end offset points just beyond the newest available data. A consumer group's committed offset records where that group should resume.

Consumers usually commit the next offset to read. If processing of offset 811 succeeds, committing 812 means resume after that record. A crash after processing but before committing can cause offset 811 to be delivered again.

• Offsets are not globally unique without topic and partition.
• Compaction can leave gaps; offsets are positions, not a count of visible records.
• Seeking to an earlier retained offset enables replay without modifying the topic.
• If retention removed a requested offset, the configured reset policy determines what happens.

Replication controls failure behaviour

Every partition has one leader and zero or more follower replicas. The in-sync replica set, or ISR, contains replicas Kafka currently regards as sufficiently caught up. ISR membership changes as replicas fall behind or recover.

Durable writes depend on replication factor, minimum in-sync replicas and producer acknowledgements together. A common production profile is replication factor 3, minimum ISR 2 and producer acks=all. If the ISR falls to one, Kafka rejects these writes rather than acknowledging below the chosen durability threshold.

Producers write; consumer groups divide work

A producer serializes a key and value, selects a partition, batches records and sends them to the leader. Batching and compression improve throughput. Idempotent production prevents retry duplicates within its defined producer session, but it does not deduplicate repeated business requests.

Consumers sharing a group ID divide partitions among themselves. One partition can have at most one active owner in a conventional group, so adding consumers beyond the partition count does not increase that group's parallelism. Different groups can independently read every partition.

acks=all
enable.idempotence=true
compression.type=zstd
delivery.timeout.ms=120000

Design delivery semantics end to end

At-most-once processing risks loss by advancing position before work completes. At-least-once processing performs the work first and commits afterwards, accepting that a crash can repeat completed work. This is the most common baseline and requires idempotent side effects or deduplication.

Kafka transactions can atomically publish Kafka output and consumed offsets. They do not automatically include an external database, email system or payment API in the transaction.

Explore the model in dFlow~IQ

1. Open a controlled workspace and select a topic with several partitions.
2. Inspect partition count, replication factor, cleanup policy and minimum ISR.
3. Browse records from a precise offset and view key, partition, offset, timestamp and headers together.
4. Filter by one stable key and verify that its offsets increase within one partition.
5. Produce test records only to an approved non-production topic, then read them back from the acknowledged positions.
6. Use a bounded integration test when downstream behaviour must be verified.