Step 9 — Kafka Topic Design

Step 6 decided whether to use Kafka. If you chose to, the next question is "how." Topic names, partitions, keys, Producer/Consumer settings, and idempotent consumers — design one topic from start to finish. Code can be fixed by refactoring, but topic design needs a migration, so the start matters.

1. Decide the topic name first

A topic name is an API. Once consumers depend on it, it is hard to change. The <domain>.<entity>.<event> pattern is recommended.

orders.order.created
orders.order.cancelled
billing.invoice.paid

• Pick one of dot or kebab as the separator (mixing breaks sorting and filtering)
• Separate environments by prefix (dev.orders.order.created) or by separate cluster — separate clusters are safer
• Do not hardcode the name in code — inject it as a config value

// config.ts — topic names as config
export const TOPICS = {
  orderCreated: process.env.ORDER_CREATED_TOPIC ?? "orders.order.created",
};

2. Decide the partition count and key

Partitions set two things at once — the throughput ceiling and the consumer parallelism ceiling.

Topic: orders.order.created  (partitions = 6)
├── Partition 0  ◀── key hash
├── Partition 1
├── ...
└── Partition 5

Consumers in a group ≤ 6

• Same key → same partition → ordering guaranteed within a partition. Use one order id as the key and that order's events land in one partition.
• No key means round-robin — the ordering guarantee disappears.
• Increasing the partition count changes hash(key) % count, so the same key goes to a different partition. The ordering assumption breaks, so start with roughly 2–3x the expected throughput for a margin.

await producer.send({
  topic: TOPICS.orderCreated,
  messages: [
    { key: order.id, value: JSON.stringify(event) },  // key = unit of ordering
  ],
});

3. Producer settings — send without loss

Confirm not "I sent it" but "the broker received it."

import { Kafka } from "kafkajs";

const kafka = new Kafka({ clientId: "my-service", brokers: ["kafka:9092"] });
const producer = kafka.producer({
  idempotent: true,        // prevents duplicate publishes from retries
});

await producer.connect();
await producer.send({
  topic: TOPICS.orderCreated,
  acks: -1,                // -1 = all (full ISR) · prevents loss
  messages: [{ key: order.id, value: JSON.stringify(event) }],
});

• acks: -1 (= all) — confirm until the full ISR has received it. Essential when data matters
• idempotent: true — the broker prevents Producer retries from creating duplicates
• Waiting on every message one by one with await collapses throughput — send in batches and check the result at once

No publishing before the DB commit — if you send the message first inside a transaction, the message goes out even on rollback. INSERT into an outbox table in the same transaction, and a separate publisher reads only committed rows and publishes them.

4. Consumer groups — manage the offset yourself

const consumer = kafka.consumer({
  groupId: "order-notifier",          // same group = partition split
});

await consumer.connect();
await consumer.subscribe({
  topic: TOPICS.orderCreated,
  fromBeginning: true,                // a new group starts from the beginning — prevents loss
});

await consumer.run({
  autoCommit: false,                  // commit yourself after processing
  eachMessage: async ({ message, partition, topic }) => {
    await handle(message);
    await consumer.commitOffsets([
      { topic, partition, offset: (Number(message.offset) + 1).toString() },
    ]);
  },
});

• groupId — consumers in the same group split the partitions. A different group reads the same topic independently
• fromBeginning: true — starts from the beginning when no stored offset exists (kafkajs's equivalent of auto.offset.reset=earliest). false silently skips messages in the meantime
• autoCommit: false plus manual commit after processing — pairs with at-least-once
• Blocking too long inside the consumer makes the group treat it as dead and rebalancing repeats

5. Idempotent consumer + DLQ

Kafka's default delivery guarantee is at-least-once — the same message may arrive twice. Deduplication is the consumer's responsibility.

async function handle(message: KafkaMessage) {
  const payload = JSON.parse(message.value!.toString());
  const messageId = payload.id;                 // a uuid assigned at publish time

  if (await alreadyProcessed(messageId)) return; // idempotent — skip duplicate

  try {
    await applyBusinessLogic(payload);
    await markProcessed(messageId);              // record as processed
  } catch (e) {
    if (isTransient(e)) throw e;                 // transient → retry
    await sendToDlq(message, e);                 // permanent → DLQ
  }
}

-- Idempotency table — the UNIQUE constraint blocks duplicate INSERTs
CREATE TABLE IF NOT EXISTS processed_messages (
  message_id  TEXT PRIMARY KEY,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

• Put a uuid on each message at publish time, and the consumer checks before processing and skips a duplicate
• DLQ (Dead Letter Queue) — sends permanent-failure messages (bad format, etc.) to a separate topic. The main flow is not blocked
• For transient errors, retry not immediately but with a progressively widening interval (backoff); send to the DLQ beyond the limit

6. Schema evolution (briefly)

Because Producer and Consumer deploy separately, message format changes break easily.

• Adding an optional field with a default — usually safe. Old consumers ignore it
• Removing, renaming, or retyping a field — breaking changes. Proceed in stages
• As scale grows, a Schema Registry (Avro/Protobuf/JSON Schema) automatically verifies compatibility (backward recommended)

Starting an MVP with JSON and adopting a Schema Registry after stabilizing is the natural order.

Common pitfalls

• Synchronous wait per message — send in batches. Awaiting one by one drops throughput sharply
• Under-sized partition count — starting with 4 and unable to grow to 8 consumers. Increasing breaks ordering
• Long blocking inside the consumer — exceeds max.poll.interval.ms → repeated rebalancing
• A new group with fromBeginning: false — loses messages in the meantime. A new group uses true
• Propagating exceptions on bad messages — offset uncommitted → infinite reprocessing. Send permanent failures to the DLQ
• Publishing before the DB commit — the message goes out even on rollback. Separate with outbox or after-commit

Closing thoughts

Topic names, partition counts, and key strategies follow you through the whole operational life once decided. Agree only on the domain naming convention and the key strategy at the start, and idempotent consumers, DLQ, and schemas can be added gradually.

Next

• data/13-kafka-topics
• data/05-data-pipeline