Operating Kafka topics deliberately

Start with the workload contract

Document event meaning, owners, key, schema, ordering requirement, expected rate, record-size distribution, replay window, failure tolerance and data classification before creating a topic.

Decide whether the stream represents immutable history, current keyed state, commands or transient transport. That decision drives cleanup policy and key design.

Size partitions from measured constraints

Partition count limits conventional group parallelism and divides ordering domains. Estimate the count from tested per-partition throughput and required active consumers, then check key skew and broker capacity.

Kafka can increase but not reduce partition count in place. Existing records are not redistributed, and default key mapping may change for future records. Treat expansion as an ordering-affecting operation.

throughput partitions = ceil(peak throughput / tested throughput per partition)
required partitions   = max(throughput partitions, required active consumers)

Design replication and minimum ISR together

Replication factor controls assigned copies. Minimum ISR sets the healthy-replica threshold for producers using acks=all. With replication factor 3 and minimum ISR 2, one replica may leave ISR while durable writes continue; another loss stops those writes.

Lowering minimum ISR during an incident restores availability by accepting weaker durability. Make that trade-off explicit rather than treating it as a harmless fix.

Set retention from replay and capacity

Delete retention removes eligible log segments by age or per-partition size. It is asynchronous and segment based, so it is not a precise deletion timer. Set it from the longest acceptable consumer outage, recovery needs and storage budget.

Increasing retention preserves future data for longer but cannot restore data already removed. Reducing it can make old segments eligible for irreversible deletion shortly after the change.

retained broker bytes × replication factor
 partition imbalance and operational headroom

Choose delete, compact or both

Compaction is eventual. Several values for one key can coexist until cleaning occurs. A keyed null value is a tombstone that announces deletion; it must remain long enough for rebuilding consumers to observe it.

Align message-size limits end to end

Topic max.message.bytes limits accepted record batches after compression, not simply a JSON document's character count. Producer request, consumer fetch and replica-fetch limits must all support the intended maximum.

Large records increase memory pressure, latency and retry cost. Prefer object storage plus a validated reference for large binary objects when the business consistency model permits it.

Use a safe change checklist

1. Confirm cluster, topic and environment; capture current metadata and overrides.
2. Identify producer and consumer owners and record peak rate, size percentiles, lag and partition skew.
3. State the intended behaviour, risk and measurable success criteria.
4. Test on a representative non-production topic when feasible.
5. Apply one behavioural change at a time through the approved admin path.
6. Monitor ISR, produce errors, latency, disk use and consumer lag.
7. Re-describe effective configuration and verify all replicas return to ISR.
8. Produce and consume an approved canary, then retain the before-and-after evidence.

Investigate and verify with dFlow~IQ

1. Open the exact workspace and inspect the topic's partition and configuration details.
2. Browse a bounded sample to assess key distribution, payload size and partition placement.
3. Use precise offsets around the incident or deployment instead of Latest alone.
4. Correlate record evidence with broker and consumer metrics from your monitoring platform.
5. Apply approved administrative changes outside dFlow~IQ where required.
6. Refresh the topic and verify the effective settings.
7. Use a uniquely keyed canary or bounded test to verify resulting behaviour.