The Outbox Pattern in Spring Boot: Guaranteed Kafka Delivery
How to guarantee at-least-once Kafka message delivery without distributed transactions: using a pattern I applied in a production ecommerce microservices platform.
The problem: you can't atomically write to a database and publish to Kafka
Imagine your Order Service does two things when an order is placed: it saves the order to PostgreSQL and publishes an order-created event to Kafka. These are two separate systems. If the database write succeeds but Kafka is temporarily down: or the application crashes between the two calls: the Payment Service never receives the event. The order exists in the database but payment is never triggered. Silent data corruption.
The naive approach is a simple try-catch, but that doesn't help: the failure could be a network partition, a JVM crash, or a Kafka broker restart. You can't reliably catch those.
What you need is atomicity: either both things happen, or neither does. The Transactional Outbox Pattern gives you this without requiring a distributed transaction or two-phase commit.
How the pattern works
The core idea is simple: instead of publishing directly to Kafka, you write the event into a dedicated outbox table in the same database transaction as your business write. A separate process (a poller or a change-data-capture relay) reads the outbox table and publishes to Kafka, then marks the record as sent.
Because the business write and the outbox write happen in a single database transaction, they either both succeed or both roll back. Atomicity is guaranteed by the database, not by any distributed protocol.
┌─────────────────────────────────────────────────────┐
│ Order Service │
│ │
│ BEGIN TRANSACTION │
│ INSERT INTO orders (...) ← business │
│ INSERT INTO outbox_events (...) ← event record │
│ COMMIT │
│ │
│ [Outbox Relay] │
│ SELECT * FROM outbox_events WHERE sent = false │
│ kafkaTemplate.send(topic, event) │
│ UPDATE outbox_events SET sent = true │
└─────────────────────────────────────────────────────┘Implementation in Spring Boot
Here's how I implemented this in the Order Service of my ecommerce microservices platform.
Step 1: The outbox table entity
@Entity
@Table(name = "outbox_events")
public class OutboxEvent {
@Id
@GeneratedValue(strategy = GenerationType.UUID)
private String id;
private String topic; // e.g. "order-created"
private String aggregateId; // e.g. order UUID
@Column(columnDefinition = "TEXT")
private String payload; // JSON-serialized event
private boolean sent = false;
@CreationTimestamp
private LocalDateTime createdAt;
}Step 2: Write business data + outbox record in one transaction
@Service
@Transactional
public class OrderService {
private final OrderRepository orderRepository;
private final OutboxEventRepository outboxRepository;
private final ObjectMapper objectMapper;
public Order placeOrder(CreateOrderRequest request) {
// 1. Business write
Order order = new Order(request);
orderRepository.save(order);
// 2. Outbox write: same transaction
OrderCreatedEvent event = new OrderCreatedEvent(
order.getId(),
order.getUserId(),
order.getTotalAmount()
);
OutboxEvent outbox = new OutboxEvent();
outbox.setTopic("order-created");
outbox.setAggregateId(order.getId().toString());
outbox.setPayload(objectMapper.writeValueAsString(event));
outboxRepository.save(outbox);
return order;
// Both saves committed together: or both rolled back
}
}Step 3: The relay: a scheduled poller that publishes and marks as sent
@Component
public class OutboxRelay {
private final OutboxEventRepository outboxRepository;
private final KafkaTemplate<String, String> kafkaTemplate;
@Scheduled(fixedDelay = 1000) // every second
@Transactional
public void relay() {
List<OutboxEvent> pending = outboxRepository
.findBySentFalseOrderByCreatedAtAsc();
for (OutboxEvent event : pending) {
kafkaTemplate.send(event.getTopic(),
event.getAggregateId(),
event.getPayload());
event.setSent(true);
// Marked sent in the same transaction as the read
}
}
}Why not just use @TransactionalEventListener?
Spring's @TransactionalEventListener(phase = AFTER_COMMIT) fires after the transaction commits and publishes to Kafka directly. This looks clean but it breaks the guarantee: if the JVM crashes after commit but before the listener fires, the event is lost. The outbox survives a crash because the record is already in the database.
Handling duplicates on the consumer side
Because the relay uses at-least-once delivery, a consumer might receive the same event twice (e.g. if the relay crashes after publishing but before marking sent). Your consumers need to be idempotent. A simple approach: store processed event IDs in a processed_events table and skip any event whose ID you've already seen.
@KafkaListener(topics = "order-created")
@Transactional
public void handleOrderCreated(OrderCreatedEvent event) {
if (processedEventRepository.existsById(event.getEventId())) {
return; // already processed: skip
}
// ... business logic ...
processedEventRepository.save(new ProcessedEvent(event.getEventId()));
}Key takeaways
- Never publish to Kafka directly inside a
@Transactionalmethod: Kafka can't participate in a database transaction. - Write your event to an outbox table in the same transaction as your business data.
- A separate relay reads and publishes pending events, then marks them sent.
- Consumers must be idempotent because at-least-once delivery means duplicates can arrive.
- This pattern requires no additional infrastructure: just a table, a scheduled job, and your existing Kafka setup.
Want to see the full implementation?
The complete ecommerce microservices platform: including all Kafka producers, consumers, and resilience patterns: is on GitHub.