Event-Driven Systems with Kafka

Understanding Event-Driven Architecture

Event-driven architecture (EDA) is a paradigm where services communicate through events. This approach enables loose coupling, scalability, and real-time processing capabilities that are essential for modern distributed systems.

What is Apache Kafka?

Apache Kafka is a distributed event streaming platform capable of handling trillions of events per day. It's designed for high-throughput, low-latency, and fault-tolerant event streaming.

Key Concepts

1. Topics: Categories to which records are published
2. Producers: Systems that publish events to topics
3. Consumers: Systems that subscribe to topics and process events
4. Partitions: Divisions within topics for parallel processing
5. Consumer Groups: Groups of consumers that share the workload

Setting Up Kafka

Basic Configuration

``typescript
// Kafka producer configuration
const producer = kafka.producer({
'metadata.broker.list': 'localhost:9092',
'acks': 'all',
'retries': 3
})

// Kafka consumer configuration
const consumer = kafka.consumer({
'group.id': 'order-processing-group',
'metadata.broker.list': 'localhost:9092'
})
`

Event Design Patterns

1. Event Notification

Notify that something happened:

`typescript
interface OrderCreatedEvent {
type: 'order.created'
orderId: string
userId: string
timestamp: Date
}
`

2. Event Carried State Transfer

Include the full state in the event:

`typescript
interface OrderUpdatedEvent {
type: 'order.updated'
orderId: string
order: {
id: string
status: 'paid' | 'shipped' | 'delivered'
items: OrderItem[]
total: number
}
}
`

3. Event Sourcing

Store all state changes as events:

`typescript
interface OrderEvent {
type: 'order.created' | 'order.paid' | 'order.shipped'
orderId: string
data: any
timestamp: Date
}
`

Real-World Implementation

Order Processing System

`typescript
// Producer: Order Service
class OrderService {
async createOrder(order: CreateOrderDTO): Promise {
const newOrder = await this.repository.create(order)

// Publish event
await producer.send({
topic: 'orders',
messages: [{
key: newOrder.id,
value: JSON.stringify({
type: 'order.created',
orderId: newOrder.id,
userId: newOrder.userId,
items: newOrder.items,
timestamp: new Date()
})
}]
})

return newOrder
}
}

// Consumer: Inventory Service
class InventoryService {
async start() {
await consumer.subscribe({ topic: 'orders' })

await consumer.run({
eachMessage: async ({ message }) => {
const event = JSON.parse(message.value.toString())

if (event.type === 'order.created') {
await this.reserveItems(event.items)
}
}
})
}

private async reserveItems(items: OrderItem[]) {
for (const item of items) {
await this.inventoryRepository.decrement(
item.productId,
item.quantity
)
}
}
}
`

Real-Time Analytics

`typescript
// Analytics consumer
class AnalyticsService {
async processEvents() {
await consumer.subscribe({ topic: 'orders' })

await consumer.run({
eachMessage: async ({ message }) => {
const event = JSON.parse(message.value.toString())

// Update real-time metrics
await this.metrics.increment('orders.created')
await this.metrics.track('revenue', event.total)

// Update dashboards
await this.websocket.broadcast({
type: 'order.created',
data: event
})
}
})
}
}
`

Advanced Patterns

Exactly-Once Semantics

Ensure events are processed exactly once:

`typescript
// Enable idempotent writes
class OrderService {
async handleEvent(event: OrderEvent) {
// Check if already processed
const processed = await this.checkProcessed(event.eventId)
if (processed) return

// Process event
await this.processOrder(event)

// Mark as processed
await this.markProcessed(event.eventId)
}
}
`

Dead Letter Queues

Handle failed events:

`typescript
// Configure DLQ
await consumer.subscribe({
topic: 'orders',
dlqTopic: 'orders-dlq'
})

// Process failed events
async processDLQ() {
await consumer.subscribe({ topic: 'orders-dlq' })

await consumer.run({
eachMessage: async ({ message }) => {
const event = JSON.parse(message.value.toString())

// Analyze failure
await this.analyzeFailure(event)

// Retry or alert
if (this.canRetry(event)) {
await this.retryEvent(event)
} else {
await this.alertTeam(event)
}
}
})
}
`

Performance Optimization

Partitioning Strategy

`typescript
// Key-based partitioning
await producer.send({
topic: 'orders',
messages: [{
key: order.userId, // Ensures all user orders go to same partition
value: JSON.stringify(event)
}]
})
`

Batching

`typescript
// Batch messages for efficiency
await producer.sendBatch({
topic: 'orders',
messages: events.map(event => ({
key: event.orderId,
value: JSON.stringify(event)
}))
})
`

Monitoring and Observability

Consumer Lag Monitoring

`typescript
// Monitor consumer lag
async getConsumerLag(groupId: string, topic: string) {
const admin = kafka.admin()
const offsets = await admin.fetchOffsets({
groupId,
topics: [topic]
})

// Calculate lag
return offsets.map(offset => ({
partition: offset.partition,
lag: offset.high - offset.offset
}))
}
`

Event Tracking

`typescript
// Track event flow
class EventTracker {
async track(event: Event) {
await this.repository.save({
eventId: event.id,
eventType: event.type,
timestamp: event.timestamp,
producers: ['order-service'],
consumers: ['inventory', 'analytics'],
status: 'completed'
})
}
}
`

Common Challenges

1. Schema Evolution

Handling changing event schemas over time:

`typescript
// Use schema registry
const schema = await schemaRegistry.getSchema('orders.v2')
const event = validate(event, schema)
`

2. Ordering Guarantees

Maintaining event order within partitions:

`typescript
// Use message keys
await producer.send({
topic: 'orders',
messages: [{
key: order.id, // Orders with same key maintain order
value: JSON.stringify(event)
}]
})
`

3. Backpressure

Handling slow consumers:

`typescript
// Configure max poll records
const consumer = kafka.consumer({
'max.poll.records': 100,
'max.poll.interval.ms': 300000
})
``

Best Practices

1. Schema First: Define event schemas upfront
2. Immutable Events: Never modify events after publishing
3. Versioning: Use versioning for event schemas
4. Monitoring: Implement comprehensive monitoring
5. Testing: Test event flows end-to-end
6. Documentation: Document all event types
7. Security: Secure your Kafka cluster
8. Backup: Implement backup and recovery

Conclusion

Event-driven architecture with Kafka provides a powerful foundation for building scalable, real-time systems. By understanding the core concepts and following best practices, you can build systems that handle millions of events reliably and efficiently.

The key is to start simple, iterate based on your needs, and invest in monitoring and observability from the beginning.