Idempotent Kafka Consumers: Exactly-Once in Practice

A charge went out twice. The logs showed one payment event published, and two PaymentCaptured records written. Nothing had crashed, nothing had been retried by a human, and the code had no loop in it.

What happened was a consumer group rebalance. The consumer processed the record, then took slightly too long to commit its offset, the broker decided it was dead and reassigned the partition — and the new owner started from the last committed offset, which was before that record.

This is not a bug in Kafka. It is Kafka behaving exactly as documented. At-least-once delivery means duplicates are part of the contract, and the consumer is where that gets handled.

Why at-least-once is the default

Processing a record is two separate actions, and they cannot be made atomic for free:

1. Do the work — write a row, call a service, send an email.
2. Commit the offset saying "I am done with this record."

Whichever order you pick, a crash in between costs you something:

Given the choice between duplicates and silent data loss, duplicates win every time — a duplicate is recoverable, a lost payment is not. So the default is at-least-once, and your handler has to be safe to run twice.

Idempotency beats coordination

The instinct is to reach for exactly-once semantics and transactions. Usually you do not need them. If processing the same record twice produces the same end state, duplicates stop mattering and the whole problem dissolves.

The cheapest version: give every event a stable ID and record the ones you have seen. The unique constraint does the work — no locks, no read-then-write race.

@Entity
@Table(name = "processed_event")
public class ProcessedEvent {

  @Id
  @Column(name = "event_id", length = 64)
  private String eventId;          // from the producer, not generated here

  @Column(nullable = false)
  private String consumerGroup;    // same event, different consumers

  @Column(nullable = false)
  private Instant processedAt;
}

@Component
@RequiredArgsConstructor
public class PaymentConsumer {

  private final ProcessedEventRepository processed;
  private final PaymentService payments;

  @KafkaListener(topics = "payments.captured", groupId = "billing")
  @Transactional
  public void onPaymentCaptured(
      @Payload PaymentCaptured event,
      @Header(KafkaHeaders.RECEIVED_KEY) String key) {

    // Claim the event first. The PK constraint is the concurrency control:
    // two consumers racing on the same event, exactly one wins.
    try {
      processed.saveAndFlush(ProcessedEvent.of(event.eventId(), "billing"));
    } catch (DataIntegrityViolationException duplicate) {
      log.debug("event {} already processed, skipping", event.eventId());
      return;
    }

    // Same transaction: either both the claim and the work commit, or neither.
    payments.capture(event.orderId(), event.amount());
  }
}

The claim and the work share one database transaction. If capture throws, the claim rolls back too, so redelivery genuinely retries rather than being swallowed as a duplicate. Getting that ordering wrong is how events get silently dropped.

Better still: make the write itself idempotent

When the domain allows it, skip the bookkeeping entirely. An upsert keyed on something stable is naturally safe to repeat:

@Modifying
@Query("""
    INSERT INTO order_status (order_id, status, updated_at, version)
    VALUES (:orderId, :status, :at, :version)
    ON CONFLICT (order_id) DO UPDATE
      SET status = EXCLUDED.status,
          updated_at = EXCLUDED.updated_at,
          version = EXCLUDED.version
      WHERE order_status.version < EXCLUDED.version
    """)
void applyStatus(UUID orderId, String status, Instant at, long version);

The version guard handles the harder half of the problem: out-of-order delivery. Kafka only orders records within a partition, so if related events land on different partitions, a stale update can arrive after a newer one. Comparing versions makes a late event a no-op instead of a regression.

Which is also the argument for choosing partition keys deliberately. Key by orderId and every event for one order lands on one partition, in order.

Retries and the dead letter topic

Not every failure deserves a retry. A malformed payload will fail identically forever, and retrying it blocks the partition behind it — one bad record stalls every consumer on that partition. Separate the two cases explicitly.

@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<String, Object> template) {
  // exhausted retries -> payments.captured.DLT
  var recoverer = new DeadLetterPublishingRecoverer(template,
      (record, ex) -> new TopicPartition(record.topic() + ".DLT", record.partition()));

  // 1s, 2s, 4s, 8s, 16s — capped, jittered by the framework
  var backoff = new ExponentialBackOffWithMaxRetries(5);
  backoff.setInitialInterval(1_000L);
  backoff.setMultiplier(2.0);
  backoff.setMaxInterval(16_000L);

  var handler = new DefaultErrorHandler(recoverer, backoff);

  // These will never succeed on retry. Straight to the DLT.
  handler.addNotRetryableExceptions(
      DeserializationException.class,
      MethodArgumentNotValidException.class,
      IllegalArgumentException.class);

  return handler;
}

Blocking retries hold the partition. For long backoffs, use @RetryableTopic instead — failed records go to a delayed retry topic and the main partition keeps moving. The trade-off is that you lose ordering for retried records, which is fine if your consumer is idempotent, and not fine if it is not.

When you actually do need transactions

There is one case idempotency does not cover: consume-transform-produce, where you read from one topic and write to another. Without a transaction, you can publish downstream and then fail before committing the offset, so redelivery publishes again — and this time the duplicate is in a topic other services already consumed.

spring:
  kafka:
    producer:
      transaction-id-prefix: billing-tx-
      acks: all
      enable-idempotence: true
    consumer:
      isolation-level: read_committed   # never see uncommitted records

@KafkaListener(topics = "orders.placed", groupId = "billing")
@Transactional("kafkaTransactionManager")
public void onOrderPlaced(OrderPlaced event) {
  Invoice invoice = invoices.create(event);
  // the send and the offset commit land in one Kafka transaction
  template.send("invoices.created", invoice.orderId().toString(), InvoiceCreated.from(invoice));
}

Note what this does not cover: your database. A Kafka transaction is atomic across Kafka only. Mixing a database write and a Kafka publish in one unit of work is the classic dual-write problem, and the real answer there is the transactional outbox pattern — write the event to an outbox table in the same database transaction, and publish it from there.

Exactly-once also costs throughput and adds coordinator failure modes. Reach for it when the semantics require it, not by default.