Kafka Data Enrichment with LLM: Idempotent Consumers, DLQ, and Tracing

Problem

Event-driven systems often need enrichment (classification, normalization, tagging, summarization) before downstream consumers can act. Doing enrichment with LLM calls introduces production risks: duplicate processing, partial failures, uncontrolled latency, and cost spikes.

This solution provides a production-grade pattern to enrich Kafka messages with an LLM while guaranteeing idempotency, safe retries, dead-letter handling, and end-to-end tracing.

What You Get

• Idempotent Kafka consumer enrichment pipeline
• Retry + DLQ strategy that avoids poison-message loops
• Trace propagation across consumer → enrichment → producer
• Audit-ready run history + step outcomes
• Runnable Docker Compose environment

Who This Is For

Backend engineers operating Kafka-based pipelines who need:

• reliable enrichment
• predictable retry behavior
• strong observability
• operational safety controls

Key Constraints

• Designed for “at-least-once” Kafka consumption; idempotency handles duplicates
• LLM calls can be slow/variable; system includes timeouts and backpressure options
• Requires a persistent store for deduplication/audit (PostgreSQL)

Architecture Overview

Consumer → Dedup Check → Enrichment Call → Publish Result → Audit/Trace

Core components:

• Kafka consumer with idempotency key strategy
• PostgreSQL tables for dedup + run history
• DLQ topic for failures exceeding retry policy
• OpenTelemetry tracing to correlate message lifecycle

When NOT to Use This

• If enrichment must be strictly synchronous and low-latency (<50ms)
• If you cannot tolerate LLM variability (you may need deterministic rules/ML instead)
• If your event volume is extremely high and enrichment is not worth the cost per event

Upgrade to Pro

Pro includes:

• full implementation details
• runnable code bundle downloads
• configuration matrix and failure-mode playbooks
• evidence artifacts (logs/traces/screenshots where published)