Redis Streams as an Event Buffer in Datahub

Series: Designing a Microservice-Friendly Datahub
PART III — CASE STUDY: MY CSL DATAHUB IMPLEMENTATION
Previous: The Web App of my CSL Datahub implementation
Next: The .NET Processor: Orchestration and Translation in Datahub

Redis is famous for being fast. That reputation can make architects nervous: fast is often mistaken for fragile. In my CSL Datahub implementation, Redis Streams play a deceptively modest role—but that role is critical. They are the event buffer that absorbs pressure, decouples time, and protects the legacy core system from the chaos of the outside world.

This article explains why Redis Streams were chosen, how they work in practice, and why “fast” does not mean “reckless” when used deliberately.

Disclaimer (Context & NDA)
The CSL Datahub implementation discussed here was designed and built in 2021. While the architectural principles remain valid, some tooling choices could be updated today. To comply with NDA requirements, domain-specific logic, schemas, and sensitive details are intentionally generalized.

Why an Event Buffer Exists at All

Before naming any technology, let’s restate the need.

The CSL Web App must be able to:

• Commit state quickly

• Remain responsive under load

• Avoid depending on downstream systems

• Emit events without orchestrating outcomes

Downstream systems (processors, brokers, modules) must be able to:

• Fall behind temporarily

• Restart without data loss

• Catch up deterministically

Those needs are in tension. An event buffer resolves that tension by inserting time and elasticity between producers and consumers.

Redis Streams fill that role.

Why Redis Streams Was Chosen

The choice was not “Redis vs Kafka” or “Redis vs RabbitMQ.” It was Redis Streams vs the actual constraints.

Redis Streams were chosen because they:

• Were already available operationally

• Offered append-only semantics

• Supported consumer groups

• Had predictable failure modes

• Required minimal changes to the legacy PHP app

Most importantly, Redis behaves like local infrastructure from the perspective of the Web App. It’s fast, nearby, and familiar—qualities that matter when protecting a core system.

Streams, Not Queues: A Subtle but Important Distinction

Traditional queues:

• Deliver messages

• Remove them once consumed

• Prioritize throughput over history

Streams:

• Append entries immutably

• Track consumption via offsets

• Allow multiple consumers to read independently

In Redis Streams, data is not destroyed by reading it. Instead, consumers acknowledge progress.

That difference enables:

• Replay

• Debugging

• Recovery after crashes

• Clear visibility into lag

This makes Streams ideal as a buffer between a stateful producer and asynchronous consumers.

Producing Events from the Web App

From the CSL Web App’s point of view, emitting an event is deliberately boring:

$redis->xAdd(
    'csl:events',
    '*',
    [
        'type' => 'user.updated',
        'user_id' => $userId,
        'occurred_at' => time()
    ]
);

There is no routing.
There is no retry logic.
There is no consumer awareness.

The Web App’s only responsibility is to record the fact that something happened—quickly and safely.

Consumer Groups: Coordinated, Not Centralized

Redis Streams support consumer groups, which allow multiple consumers to split work without duplicating it.

A group is created once:

XGROUP CREATE csl:events csl-group $ MKSTREAM

Each Processor instance joins the group with its own consumer name:

var entries = redis.StreamReadGroup(
    "csl-group",
    "processor-1",
    "csl:events",
    ">"
);

Key properties:

• Each event is delivered to one consumer in the group

• Progress is tracked per group, not per consumer

• Consumers can come and go

• Crashed consumers leave pending messages behind

This gives you horizontal scalability without central coordination.

Acknowledgement and Safety

Processing is explicit. After successful handling, the Processor acknowledges the message:

redis.StreamAcknowledge(
    "csl:events",
    "csl-group",
    entry.Id
);

If a Processor crashes before acknowledging:

• The message remains pending

• Another consumer can claim it

• No data is lost

This is the foundation of at-least-once delivery.

Ordering and Durability (In Context)

Redis Streams provide:

• Order within a stream

• Timestamp-based IDs

• Disk-backed persistence (via RDB/AOF)

What they do not promise:

• Infinite retention

• Global ordering across partitions

• Historical analytics

And that’s fine—because the CSL Datahub doesn’t need those things at this layer.

Ordering here means:

• “Changes are processed roughly in the order they occurred”

• Not “the entire system depends on strict sequencing”

Durability here means:

• “Events survive process restarts”

• Not “events are archived forever”

This is right-sized reliability.

Backpressure: Letting the System Breathe

One of Redis Streams’ most valuable traits is how visibly they handle backpressure.

When consumers slow down:

• Stream length grows

• Pending entries accumulate

• Lag becomes measurable

Nothing explodes. Nothing blocks the Web App. Pressure accumulates where it belongs—in the buffer.

This turns overload into latency instead of failure.

Operationally, this is gold:

• You can alert on lag

• Scale consumers horizontally

• Investigate without panic

Backpressure is not a bug. It’s a signal.

Fast Doesn’t Mean Reckless

Redis is fast because it does less—not because it does unsafe things.

Used correctly, Redis Streams:

• Avoid business logic

• Avoid orchestration

• Avoid cross-system coupling

They act as a shock absorber, not a coordinator.

The danger is not Redis itself—it’s overloading Redis with responsibilities it was never meant to carry.

Streams vs Queues: When to Use Which

Use streams when:

• You need buffering, not history

• You want replay and visibility

• You expect consumer lag

• You value simplicity

Use queues when:

• Messages are ephemeral

• You don’t care about replay

• Routing is the primary concern

In the CSL Datahub, Redis Streams buffer. RabbitMQ routes. Each tool does one job.

Redis Beyond Caching

This is the quiet triumph of Redis Streams.

They allow Redis to:

• Be part of an event-driven architecture

• Absorb pressure gracefully

• Enable decoupling without heavy infrastructure

Redis doesn’t replace the message broker. It protects the producer.

And that makes all the difference.

Where We Go Next

Now that events are safely buffered, the next step is transformation.

In the next article, The CSL Processor: Translation and Orchestration, we’ll look at how buffered events are consumed, enriched, retried, and forwarded—without turning the Processor into a “God service.”

Redis Streams don’t get credit for being exciting.
They get credit for keeping everything else calm.