The .NET Processor: Orchestration and Translation in Datahub

Series: Designing a Microservice-Friendly Datahub
PART III — CASE STUDY: MY CSL DATAHUB IMPLEMENTATION
Previous: Redis Streams as an Event Buffer in Datahub
Next: RabbitMQ as the Inter-Module Backbone in Datahub

If the CSL Web App is the source of truth, and Redis Streams are the buffer, then the .NET Processor is the bridge—the component that allows two very different worlds to communicate without colliding.

This article focuses on the most delicate part of my implemented CSL Datahub: the Processor service. It explains why this service exists, what it is allowed to do, what it must never do, and how it keeps the entire system decoupled instead of centralized.

Disclaimer (Context & NDA)
The CSL Processor described here was designed and implemented in 2021. While the architectural principles remain relevant, specific implementation details could be modernized today. To comply with NDA requirements, business rules, payload structures, and proprietary workflows are intentionally generalized.

Why the Processor Exists at All

A natural question is:
Why not let the CSL Web App talk directly to RabbitMQ?

Because doing so would:

• Introduce external dependencies into request flows

• Pull message-broker concerns into legacy code

• Expose the core system to new failure modes

• Make infrastructure changes risky and invasive

The Processor exists to absorb that complexity.

Its mission is simple:

Translate local facts into distributed events, and distributed signals into controlled actions—without leaking responsibility across boundaries.

It is not a domain service.
It is not a workflow engine.
It is an integration service.

Responsibilities (and Just as Important: Non-Responsibilities)

The Processor has a narrow, enforced scope.

Responsibilities

• Consume events from Redis Streams

• Normalize and enrich events if needed

• Publish events to RabbitMQ

• Consume RabbitMQ messages from other modules

• Call CSL Web App REST APIs when explicitly required

• Handle retries, failures, and DLQs

Non-Responsibilities

• No authoritative data ownership

• No business decision-making

• No cross-domain orchestration

• No long-lived state

This line is critical. Once crossed, the Processor turns into a “God service.”

Consuming Redis Streams: The First Translation

The Processor’s first job is to consume buffered events from Redis Streams.

A simplified consumption loop looks like this:

var entries = redis.StreamReadGroup(
    "csl-group",
    "processor-1",
    "csl:events",
    ">"
);

foreach (var entry in entries)
{
    try
    {
        var eventData = Parse(entry);
        PublishToRabbitMq(eventData);

        redis.StreamAcknowledge(
            "csl:events",
            "csl-group",
            entry.Id
        );
    }
    catch (Exception ex)
    {
        // leave unacked for retry or inspection
        LogError(ex, entry);
    }
}

Key characteristics:

• At-least-once delivery

• Explicit acknowledgements

• Failures don’t block the stream

• Crashes don’t lose data

The Processor does not assume success. It earns it.

Translating Events, Not Owning Meaning

The Processor may:

• Rename fields

• Normalize formats

• Add metadata

• Split one event into multiple routing messages

But it does not reinterpret business meaning.

For example:

var message = new {
    EventType = "user.updated",
    UserId = entry["user_id"],
    OccurredAt = entry["occurred_at"],
    Source = "csl"
};

This is translation, not transformation.

If the Processor starts deciding what the update means, it becomes a domain service—and coupling creeps in.

Publishing to RabbitMQ: Crossing the Boundary

Once translated, events are published into RabbitMQ:

channel.BasicPublish(
    exchange: "csl.events",
    routingKey: "user.updated",
    body: Serialize(message)
);

At this point:

• The CSL Web App is no longer involved

• Consumers are completely decoupled

• Fan-out happens naturally

• Backpressure shifts downstream

This is the moment where local state becomes distributed signal.

Consuming RabbitMQ: The Reverse Path

The Processor also listens to events from other modules.

For example:

channel.BasicConsume(
    queue: "csl.commands",
    autoAck: false,
    consumer: consumer
);

consumer.Received += (sender, ea) =>
{
    try
    {
        HandleIncomingEvent(ea.Body);
        channel.BasicAck(ea.DeliveryTag, false);
    }
    catch
    {
        channel.BasicNack(ea.DeliveryTag, false, false);
    }
};

Incoming messages typically represent:

• Completion notifications

• External state changes

• Requests for controlled updates

The Processor interprets these as signals, not commands.

Calling the CSL Web App API (Carefully)

When the Processor needs to affect CSL state, it does so via explicit REST APIs.

await httpClient.PostAsync(
    "/api/csl/update-status",
    new StringContent(payload)
);

Important constraints:

• Calls are explicit and intentional

• Failures are retried or DLQ’d

• No synchronous chaining across services

• No assumption of immediate success

The API boundary protects the Web App from accidental orchestration.

How the Processor Prevents Coupling

The Processor decouples the system by:

• Isolating infrastructure complexity

• Translating protocols and semantics

• Acting as a shock absorber for failure

• Preventing direct dependencies between services

No module:

• Knows Redis internals

• Knows CSL database structure

• Knows other modules exist

Everything speaks in events and APIs.

The Primary Risk: Overloading the Processor

The Processor’s power is also its danger.

Warning signs of overload:

• Too many event types handled

• Business logic creeping in

• Long synchronous workflows

• Tight coupling to multiple domains

If you hear:

“Let’s just add it to the Processor…”

…pause.

A good Processor is boring.
A clever Processor is a liability.

Guardrails That Keep It Healthy

To keep the Processor sane:

• Keep handlers small and focused

• Split responsibilities early

• Scale horizontally

• Make idempotency mandatory

• Monitor lag and failure rates

If it grows, split it by responsibility—not by convenience.

Why This Is the Heart of the System

The CSL Datahub works because:

• The Web App owns truth

• Redis absorbs pressure

• The Processor translates safely

• RabbitMQ distributes freely

The Processor sits at the only place where both worlds touch. That makes it critical—and deserving of restraint.

Mental Model Recap

Think of the Processor as:

• A translator, not an author

• A bridge, not a capital city

• A facilitator, not a coordinator

When it stays in that role, the entire system remains flexible.

Where We Go Next

Now that events are translated and published, the final piece of the communication story comes into focus.

In the next article, RabbitMQ as the Inter-Module Backbone, we’ll explore how exchanges, routing keys, and queues allow independent modules to react at scale—without ever knowing who else is listening.

The Processor speaks both languages so no one else has to.