Part 073 — Kafka Retry, DLQ, Idempotency, Duplicates, and Ordering

Fokus part ini: memahami bagaimana Kafka consumer/producer gagal di production dan bagaimana mendesain retry, DLQ, idempotency, duplicate handling, ordering, dan replay agar sistem tetap benar.

Di sistem enterprise, Kafka jarang gagal dalam bentuk sederhana seperti "broker mati". Yang lebih sering terjadi:

• satu message valid secara schema tetapi gagal secara business rule
• downstream database lambat sehingga consumer timeout
• consumer memproses event sukses tetapi gagal commit offset
• event terkirim dua kali karena retry producer
• event diproses out-of-order karena partitioning salah
• DLQ penuh dengan message tanpa ownership
• replay event memperbaiki satu bug tetapi menciptakan duplicate side effect baru

Kafka memberikan log dan offset. Kafka tidak otomatis memberikan business correctness.

Correctness harus dibangun di consumer, producer, data model, retry policy, dan operational runbook.

1. Core Mental Model

Kafka failure handling harus dilihat sebagai state machine.

record received
  -> validate envelope
  -> validate schema
  -> validate business precondition
  -> process side effect
  -> persist result
  -> publish next event if needed
  -> commit offset

Setiap step bisa gagal.

Failure tidak boleh diperlakukan sama.

bad payload              -> non-retryable, DLQ/quarantine
schema incompatible      -> contract failure, stop/alert
DB transient timeout     -> retryable, bounded retry
duplicate event          -> idempotent no-op
out-of-order event       -> buffer/reject/reconcile depending on model
unknown tenant           -> security/config/data governance failure
poison message           -> isolate, do not block partition forever

Senior engineer harus bertanya:

1. Failure ini transient atau permanent?
2. Apakah retry akan memperbaiki keadaan atau memperbesar kerusakan?
3. Apakah side effect sudah terjadi sebelum failure?
4. Apakah offset sudah di-commit?
5. Apakah consumer bisa dipanggil ulang dengan event yang sama?
6. Apakah event ini boleh diproses out-of-order?
7. Siapa owner message jika masuk DLQ?
8. Bagaimana replay dilakukan dengan aman?

2. Kafka Delivery Reality

Kafka sering disebut at-least-once by default dalam consumer processing.

Makna praktisnya:

consumer may receive the same logical event more than once

Penyebab duplicate:

• producer retry after broker ack lost
• consumer processing succeeds but offset commit fails
• rebalance terjadi saat processing belum selesai
• application crash setelah side effect tetapi sebelum commit
• manual replay
• topic reprocessing dari offset lama
• outbox publisher restart

Karena itu, consumer yang menulis database, memanggil downstream service, atau mem-publish event lanjutan harus idempotent.

At-most-once biasanya mengurangi duplicate tetapi membuka risiko data loss.

Exactly-once semantics Kafka membantu dalam skenario tertentu, terutama Kafka-to-Kafka transactional pipeline, tetapi tidak otomatis membuat external side effect seperti database update, HTTP call, email, atau payment menjadi exactly-once.

Prinsip aman:

Assume at-least-once delivery.
Design idempotent processing.
Make duplicate visible but harmless.

3. Retry Taxonomy

Tidak semua error layak di-retry.

3.1 Retryable Errors

Contoh:

• database connection timeout
• temporary network failure
• downstream 503
• broker temporary unavailable
• optimistic lock conflict yang bisa diulang
• rate limit dengan Retry-After

Karakteristik:

same input + later time may succeed

3.2 Non-Retryable Errors

Contoh:

• invalid JSON/schema
• required field missing
• unknown enum jika compatibility rusak
• impossible business state
• unknown tenant tanpa provisioning
• unauthorized source service
• corrupted payload

Karakteristik:

same input + later time will still fail

3.3 Ambiguous Errors

Contoh:

• downstream timeout setelah request dikirim
• DB statement timeout setelah partial work tidak jelas
• HTTP 500 dari service yang mungkin sudah melakukan side effect
• consumer crash di tengah transaction

Karakteristik:

side effect may or may not have happened

Ambiguous error membutuhkan idempotency key, reconciliation, atau state check sebelum retry.

4. Retry Placement

Retry bisa ditempatkan di beberapa layer.

producer retry
consumer local retry
retry topic
DLQ replay
manual operational retry
job-based reconciliation

4.1 Producer Retry

Producer retry biasanya untuk broker/network transient failure.

Risiko:

• duplicate publish jika idempotent producer tidak dikonfigurasi
• ordering berubah jika multiple in-flight requests
• latency meningkat di API path

Checklist:

• enable.idempotence=true jika sesuai
• acks=all untuk durability lebih kuat
• bounded delivery.timeout.ms
• max.in.flight.requests.per.connection dipahami dampaknya ke ordering
• error publish tidak disembunyikan dari command path tanpa outbox

4.2 Consumer Local Retry

Local retry terjadi dalam satu poll cycle.

poll -> process -> fail -> sleep/backoff -> retry -> success/fail

Cocok untuk transient error singkat.

Risiko:

• menahan partition terlalu lama
• melanggar max.poll.interval.ms
• blocking message setelahnya pada partition yang sama
• retry storm ke dependency

Gunakan bounded retry.

for (int attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
        handle(record);
        return;
    } catch (TransientDependencyException e) {
        backoff(attempt);
    }
}
throw new RetryExhaustedException(record);

4.3 Retry Topic

Retry topic memindahkan message gagal ke topic lain dengan delay/backoff.

main-topic
  -> fail transient
  -> retry-topic-1m
  -> retry-topic-5m
  -> retry-topic-30m
  -> DLQ

Keuntungan:

• main consumer tidak stuck lama
• retry delay tidak memblokir poll loop
• retry dapat diamati sebagai traffic terpisah

Risiko:

• ordering melemah
• topic sprawl
• metadata retry harus jelas
• event lama bisa bertabrakan dengan state baru

4.4 DLQ

DLQ bukan tempat sampah. DLQ adalah quarantine dengan ownership.

DLQ harus menyimpan:

• original topic
• original partition
• original offset
• original key
• original payload
• headers
• failure reason
• exception type
• first failure time
• last failure time
• attempt count
• consumer/service version
• trace/correlation ID
• tenant ID jika aman untuk disimpan

Tanpa metadata ini, DLQ sulit dioperasikan.

5. Poison Message

Poison message adalah message yang selalu membuat consumer gagal.

Contoh:

• payload invalid
• schema incompatible
• value tertentu men-trigger bug
• business state tidak mungkin
• tenant config hilang

Jika poison message tidak diisolasi, satu partition bisa berhenti.

partition 3:
  offset 100 ok
  offset 101 poison
  offset 102 blocked
  offset 103 blocked

Strategi:

1. Detect non-retryable failure.
2. Publish ke DLQ/quarantine.
3. Commit offset original hanya jika policy mengizinkan skip.
4. Alert owner.
5. Sediakan replay/remediation procedure.

Trade-off:

• Skip poison message menjaga availability tetapi bisa melewatkan business event penting.
• Stop consumer menjaga strictness tetapi bisa menyebabkan backlog besar.

Untuk enterprise system, policy harus eksplisit per event type.

6. Idempotent Consumer

Idempotent consumer berarti event yang sama diproses lebih dari sekali tetapi final state tetap benar.

process(event)
process(event)
process(event)
=> same final state

Bukan berarti tidak ada duplicate. Artinya duplicate tidak merusak state.

6.1 Idempotency Key

Key harus stabil untuk logical event.

Contoh:

eventId
sourceSystem + sourceEventId
aggregateId + eventVersion
commandId
requestId + operationType

Bad key:

random UUID generated by consumer per processing attempt
current timestamp
partition + offset only, if replay from transformed topic changes identity

6.2 Processed Event Table

Pattern umum:

CREATE TABLE processed_event (
    consumer_name text NOT NULL,
    event_id text NOT NULL,
    processed_at timestamptz NOT NULL DEFAULT now(),
    source_topic text,
    source_partition int,
    source_offset bigint,
    PRIMARY KEY (consumer_name, event_id)
);

Processing:

begin transaction
  insert processed_event(consumer, eventId)
  if duplicate key -> already processed -> no-op
  apply business change
commit transaction
commit Kafka offset

Pseudo-code:

@Transactional
public void handle(Event event, RecordMetadata meta) {
    boolean firstTime = processedEventRepository.tryInsert(
        "quote-order-consumer",
        event.eventId(),
        meta.topic(),
        meta.partition(),
        meta.offset()
    );

    if (!firstTime) {
        log.info("duplicate_event_ignored eventId={}", event.eventId());
        return;
    }

    orderService.apply(event);
}

6.3 Natural Idempotency

Kadang operasi bisa dibuat idempotent melalui state transition.

APPROVED -> APPROVED = no-op
CANCELLED -> CANCELLED = no-op
APPROVED -> CANCELLED = valid/invalid depending on state machine

Untuk quote/order lifecycle, ini sangat penting.

Jangan hanya dedupe event. Validasi state transition juga harus deterministic.

7. Duplicate Event Policy

Duplicate policy harus tertulis.

Contoh policy:

Duplicate with same eventId and same payload:
  ignore and mark as duplicate

Duplicate with same eventId but different payload:
  alert as contract/data integrity violation

Duplicate with different eventId but same business command:
  resolve using commandId or aggregate version

Duplicate after successful side effect:
  no-op if state already reflects effect

Mengabaikan duplicate secara diam-diam berbahaya jika duplicate membawa payload berbeda.

Minimal audit:

• duplicate count metric
• duplicate with payload mismatch alert
• dedupe store size/TTL monitoring
• per-consumer duplicate dashboard

8. Ordering Model

Kafka hanya menjamin ordering di dalam satu partition.

same topic + same partition -> ordered by offset
across partitions           -> no total order
across topics               -> no total order
retry topics                -> ordering usually weakened
DLQ replay                  -> ordering must be explicit

Ordering strategy dimulai dari event key.

Jika semua event untuk satu aggregate harus ordered, gunakan aggregate ID sebagai key.

new ProducerRecord<>(
    "quote.events",
    quoteId,       // key controls partitioning
    event
);

Untuk Quote/Order-like systems:

• quote lifecycle events mungkin harus keyed by quoteId
• order lifecycle events mungkin harus keyed by orderId
• tenant-level ordering jarang feasible karena membuat hot partition
• customer-level ordering bisa menyebabkan skew jika customer besar

8.1 Out-of-Order Detection

Tambahkan version/sequence pada event aggregate.

{
  "eventId": "evt-123",
  "aggregateId": "quote-456",
  "aggregateVersion": 17,
  "eventType": "QuoteApproved",
  "occurredAt": "2026-07-10T04:00:00Z"
}

Consumer dapat memeriksa:

expectedVersion = currentVersion + 1
incomingVersion == expectedVersion -> apply
incomingVersion <= currentVersion   -> duplicate/old event
incomingVersion > expectedVersion   -> gap/out-of-order

8.2 Ordering vs Throughput

Semakin kuat ordering requirement, semakin kecil parallelism.

strong aggregate ordering -> key by aggregateId -> parallelism by aggregate distribution
strong tenant ordering    -> key by tenantId    -> possible hot partition
global ordering           -> one partition     -> poor scalability

Senior engineer harus menantang requirement "harus ordered".

Pertanyaan review:

• Ordered terhadap apa?
• Dalam scope aggregate, tenant, customer, atau global?
• Apa consequence jika out-of-order?
• Bisa diselesaikan dengan version check?
• Bisa direkonsiliasi melalui job?

9. Retry and Ordering Conflict

Retry topic sering merusak ordering.

Contoh:

main topic partition:
  offset 10 QuoteCreated(v1)
  offset 11 QuotePriced(v2)  -> fails, sent to retry-topic
  offset 12 QuoteApproved(v3) -> processed successfully

later retry-topic:
  QuotePriced(v2) reprocessed after QuoteApproved(v3)

Jika consumer tidak memeriksa version/state, state bisa mundur.

Strategi:

1. Gunakan aggregate version.
2. Jadikan event application monotonic.
3. Untuk event lama, no-op atau reconciliation.
4. Untuk strict ordering, jangan skip ke retry topic tanpa blocking/parking strategy.
5. Gunakan state machine guard.

10. DLQ Design

DLQ harus punya contract.

Topic naming example:

<domain>.<event>.dlq
<source-topic>.dlq
<service-name>.<source-topic>.dlq

DLQ payload bisa:

1. Original payload untouched + metadata headers
2. Wrapper envelope berisi original payload + failure metadata

Wrapper lebih operable tetapi bisa menyulitkan replay jika tooling tidak jelas.

Example DLQ wrapper:

{
  "failureId": "fail-001",
  "failedAt": "2026-07-10T04:00:00Z",
  "consumer": "quote-order-consumer",
  "source": {
    "topic": "quote.events",
    "partition": 3,
    "offset": 12345,
    "key": "quote-456"
  },
  "failure": {
    "classification": "NON_RETRYABLE_VALIDATION",
    "exceptionType": "InvalidEventException",
    "message": "missing required field: quoteId"
  },
  "payload": {
    "eventId": "evt-123"
  }
}

DLQ retention harus cukup untuk investigation, tetapi sesuai data retention/security policy.

11. Replay Safety

Replay bukan hanya "consume dari offset lama".

Replay adalah operasi production berisiko tinggi.

Replay bisa menyebabkan:

• duplicate database write
• duplicate notification
• duplicate external API call
• re-open closed order
• overwrite newer state
• trigger outdated business rule
• violate tenant/security boundary
• overload downstream

Replay contract harus menjawab:

Can this event be replayed?
Can it be replayed partially?
Is replay idempotent?
Which side effects are suppressed during replay?
Which version of business logic should apply?
How is replay rate limited?
How is replay audited?
Who approves replay?

11.1 Replay Modes

technical replay:
  same event, same handler, fix transient missing processing

semantic replay:
  re-derive state from event history

repair replay:
  after code/data fix, process DLQ again

backfill replay:
  produce new events for historical data

Masing-masing punya risk berbeda.

11.2 Replay Guard

Tambahkan header/context:

x-replay-id
x-replay-reason
x-replay-operator
x-replay-start-time
x-replay-mode

Consumer bisa menggunakan context ini untuk:

• audit
• rate limiting
• suppress non-idempotent side effect
• route to special metrics
• enforce approval policy

12. Commit Strategy and Side Effects

Offset commit harus terjadi setelah side effect aman.

Bad:

commit offset -> write DB -> crash

Risiko: message hilang dari perspektif consumer, tetapi state belum berubah.

Better:

write DB transaction -> commit DB -> commit offset

Risiko masih ada:

write DB succeeds -> crash before offset commit -> message redelivered

Karena itu idempotency tetap wajib.

Rule praktis:

Never rely only on offset commit for correctness.
Use idempotent business state or processed-event store.

13. Database Transaction + Kafka Consumer

Jika consumer menulis database, pattern aman:

poll record
begin DB transaction
  dedupe event
  apply state transition
  write audit/log/outbox if needed
commit DB transaction
commit Kafka offset

Jika DB commit gagal:

• jangan commit offset
• retry sesuai klasifikasi

Jika offset commit gagal setelah DB commit:

• event akan diterima ulang
• dedupe harus mencegah duplicate side effect

Jika event processing memanggil external HTTP:

• harus ada idempotency key
• atau external call dipindah ke outbox/job terpisah
• atau side effect harus bisa dicek/reconciled

14. Producer Duplicate and Idempotence

Producer dapat menghasilkan duplicate logical event jika:

• API request di-retry
• outbox row dipublish ulang
• producer timeout ambiguity
• service restart saat publish belum tercatat

Producer-side guard:

• stable event ID
• stable command ID
• outbox unique constraint
• idempotent producer config
• event publication status

Outbox example:

CREATE TABLE outbox_event (
    id uuid PRIMARY KEY,
    aggregate_id text NOT NULL,
    aggregate_version bigint NOT NULL,
    event_type text NOT NULL,
    payload jsonb NOT NULL,
    status text NOT NULL,
    created_at timestamptz NOT NULL,
    published_at timestamptz,
    UNIQUE (aggregate_id, aggregate_version, event_type)
);

Unique constraint mencegah duplicate event untuk version aggregate yang sama.

15. Event Envelope Fields for Failure Handling

Minimal event envelope untuk production:

{
  "eventId": "evt-123",
  "eventType": "QuoteApproved",
  "eventVersion": 1,
  "aggregateType": "Quote",
  "aggregateId": "quote-456",
  "aggregateVersion": 17,
  "tenantId": "tenant-a",
  "occurredAt": "2026-07-10T04:00:00Z",
  "producedAt": "2026-07-10T04:00:01Z",
  "producer": "quote-service",
  "correlationId": "corr-789",
  "causationId": "cmd-555"
}

Fields membantu:

• idempotency: eventId
• ordering: aggregateVersion
• routing: tenantId, aggregateId
• debugging: producer, correlationId, causationId
• compatibility: eventVersion
• replay: occurredAt, metadata headers

16. Metrics and Alerts

Kafka failure handling harus observable.

Metrics minimum:

consumer_lag
records_processed_total
records_failed_total
records_retried_total
records_dlq_total
duplicate_events_total
out_of_order_events_total
poison_messages_total
processing_duration_seconds
retry_attempts_histogram
commit_failures_total
rebalance_total

Alert candidate:

• DLQ > 0 untuk event critical
• DLQ rate naik tajam
• consumer lag melebihi SLO
• duplicate payload mismatch
• retry exhausted naik
• out-of-order event naik
• poison message blocking partition
• commit failure berulang

Jangan alert semua duplicate jika duplicate expected. Alert mismatch dan rate anomaly.

17. Logging and Trace Context

Setiap failure log harus memiliki:

eventId
aggregateId
aggregateVersion
tenantId jika aman
consumerName
topic
partition
offset
attempt
failureClassification
correlationId
causationId
traceId

Bad log:

Failed to process message

Useful log:

failed_event_processing consumer=quote-order-consumer topic=quote.events partition=3 offset=12345 eventId=evt-123 aggregateId=quote-456 aggregateVersion=17 classification=TRANSIENT_DB_TIMEOUT attempt=2 correlationId=corr-789

PII dan sensitive data tidak boleh masuk log.

18. JAX-RS API + Kafka Boundary

Dalam JAX-RS command endpoint, jangan mencampur HTTP success dengan Kafka publish success secara naif.

Pattern risk:

HTTP POST approve quote
  -> update DB
  -> publish Kafka event
  -> return 200

Jika DB update sukses tetapi Kafka publish gagal, state dan event tidak konsisten.

Pilihan:

1. Transactional outbox

   • endpoint update DB + insert outbox dalam satu transaction
   • publisher mengirim event async

2. Synchronous publish before response

   • lebih sederhana tetapi latency/failure coupling tinggi
   • masih butuh ambiguity handling

3. Command accepted + async processing

   • endpoint return 202 Accepted
   • command diproses async
   • client check status

Untuk enterprise quote/order system, outbox sering lebih defensible karena ada audit trail dan replay path.

19. Failure Scenarios

Scenario A — Consumer crashes after DB commit before offset commit

Expected:

• record redelivered
• processed_event unique key detects duplicate
• no duplicate state change
• offset eventually committed

Scenario B — Retry topic reorders event

Expected:

• aggregateVersion check detects old event
• old event no-op or reconciliation
• metric increments out-of-order/old-event count

Scenario C — DLQ receives schema incompatible event

Expected:

• alert contract owner
• preserve original payload and schema metadata
• no blind replay until schema compatibility fixed

Scenario D — External HTTP call times out

Expected:

• classify as ambiguous
• use idempotency key
• check downstream status before retry if possible
• avoid unbounded retry

Scenario E — Tenant config missing

Expected:

• classify as configuration/tenant provisioning failure
• DLQ or park event depending on policy
• alert tenant/config owner
• replay after config fix

20. PR Review Checklist

Review Kafka retry/DLQ/idempotency changes with these questions:

Event Identity

• Does every event have stable eventId?
• Is there a command/request/causation ID?
• Is aggregate ID and aggregate version present where ordering matters?
• Is tenant ID present and safely handled?

Retry

• Are retryable and non-retryable errors separated?
• Is retry bounded?
• Is there backoff?
• Is retry budget considered?
• Could retry create storm against DB/downstream?

DLQ

• Is DLQ topic defined?
• Is failure metadata preserved?
• Is DLQ alerting defined?
• Who owns DLQ remediation?
• Is replay documented?

Idempotency

• Can the same event be processed twice safely?
• Is dedupe atomic with business state change?
• Does duplicate with different payload trigger alert?
• Is dedupe retention/TTL appropriate?

Ordering

• What ordering is required?
• Is key selection aligned with ordering requirement?
• What happens with out-of-order event?
• Does retry topic weaken ordering?

Offset and Transaction

• Is offset committed only after processing succeeds?
• What happens if DB commit succeeds but offset commit fails?
• Is external side effect idempotent?

Observability

• Are lag, retry, DLQ, duplicate, out-of-order metrics present?
• Are event metadata and trace context logged safely?
• Are dashboards/alerts defined?

21. Internal Verification Checklist

Untuk codebase CSG/internal environment, verifikasi:

Kafka Library and Framework

• Kafka client library yang digunakan
• Apakah ada framework wrapper internal
• Apakah consumer manual commit atau auto commit
• Apakah retry/DLQ dikelola framework atau custom
• Apakah Resilience4j/equivalent dipakai untuk dependency calls dalam consumer

Topic and Event Policy

• Topic naming convention
• Event naming convention
• Partition key policy
• Event envelope standard
• Required headers
• Tenant propagation standard

Retry and DLQ

• Retry topic pattern
• DLQ naming pattern
• DLQ retention
• DLQ ownership
• Replay procedure
• Alerting rule untuk DLQ

Idempotency

• Processed event table/cache
• Idempotency key source
• Unique constraint
• Dedupe retention
• Duplicate payload mismatch handling

Ordering

• Aggregate version usage
• Event key strategy
• Out-of-order handling
• Reconciliation job
• Saga state table jika ada

Operations

• Consumer lag dashboard
• Retry/DLQ dashboard
• Runbook for poison message
• Runbook for replay
• Incident examples from previous PR/issues

22. Anti-Patterns

Anti-pattern 1 — Infinite Retry in Consumer

while true:
  try process
  catch -> sleep 1s

Dampak:

• partition stuck
• no visibility
• dependency hammered forever
• no DLQ ownership

Anti-pattern 2 — DLQ Without Metadata

DLQ payload hanya berisi original event tanpa reason.

Dampak:

• operator tidak tahu kenapa gagal
• replay menjadi trial-and-error
• RCA sulit

Anti-pattern 3 — Commit Offset Before Side Effect

Dampak:

• data loss jika crash setelah commit sebelum processing

Anti-pattern 4 — No Idempotency Because "Kafka Is Ordered"

Ordering bukan duplicate prevention.

Anti-pattern 5 — Retry All Exceptions

Validation error tidak akan sembuh dengan retry.

Anti-pattern 6 — Key by Tenant for All Events

Bisa membuat hot partition untuk tenant besar.

Anti-pattern 7 — Replay Without Suppressing Side Effects

Bisa mengirim ulang notification, billing call, atau external command.

23. Senior Mental Model

Kafka failure handling harus menggabungkan tiga model:

delivery model:
  at-least-once, offset, partition, rebalance

state model:
  aggregate version, idempotency, transaction, saga state

operations model:
  retry, DLQ, replay, alert, runbook, ownership

Jika salah satu hilang, sistem terlihat berjalan tetapi rapuh saat incident.

Prinsip utama:

Retry is not correctness.
DLQ is not ownership.
Ordering is not idempotency.
Offset commit is not business state.
Replay is not free.

24. Summary

Di part ini kita membahas:

• retryable vs non-retryable vs ambiguous failure
• local retry, retry topic, DLQ
• poison message isolation
• idempotent consumer dan dedupe store
• duplicate event policy
• Kafka ordering dan partition key
• retry vs ordering conflict
• replay safety
• offset commit dan side effect ordering
• JAX-RS API + Kafka consistency boundary
• observability dan PR review checklist

Part berikutnya membahas schema governance: Schema Registry, Avro, JSON Schema, Protobuf, compatibility mode, dan cara memastikan event contract bisa berevolusi tanpa merusak producer/consumer.