Problem
Event-driven systems often need enrichment (classification, normalization, tagging, summarization) before downstream consumers can act. Doing enrichment with LLM calls introduces production risks: duplicate processing, partial failures, uncontrolled latency, and cost spikes.
This solution provides a production-grade pattern to enrich Kafka messages with an LLM while guaranteeing idempotency, safe retries, dead-letter handling, and end-to-end tracing.
What You Get
- Idempotent Kafka consumer enrichment pipeline
- Retry + DLQ strategy that avoids poison-message loops
- Trace propagation across consumer → enrichment → producer
- Audit-ready run history + step outcomes
- Runnable Docker Compose environment
Who This Is For
Backend engineers operating Kafka-based pipelines who need:
- reliable enrichment
- predictable retry behavior
- strong observability
- operational safety controls
Key Constraints
- Designed for “at-least-once” Kafka consumption; idempotency handles duplicates
- LLM calls can be slow/variable; system includes timeouts and backpressure options
- Requires a persistent store for deduplication/audit (PostgreSQL)
Architecture Overview
Consumer → Dedup Check → Enrichment Call → Publish Result → Audit/Trace
Core components:
- Kafka consumer with idempotency key strategy
- PostgreSQL tables for dedup + run history
- DLQ topic for failures exceeding retry policy
- OpenTelemetry tracing to correlate message lifecycle
When NOT to Use This
- If enrichment must be strictly synchronous and low-latency (<50ms)
- If you cannot tolerate LLM variability (you may need deterministic rules/ML instead)
- If your event volume is extremely high and enrichment is not worth the cost per event
Upgrade to Pro
Pro includes:
- full implementation details
- runnable code bundle downloads
- configuration matrix and failure-mode playbooks
- evidence artifacts (logs/traces/screenshots where published)