Chatter.IntegrationEventRelay

1

Chapter 1 of 9

The island

Your legacy system writes to a SQL Server database. It can’t create or publish integration events: its as-is architecture, its technical debt, or an inability to integrate with message broker infrastructure keeps it out of an event-driven architecture.

Think of a monolithic desktop app (WinForms or Swing), a VBA Excel app, or people editing tables directly. The data changes. Nobody downstream hears about it.

README: How it works

2

Chapter 2 of 9

The event-driven mainland

On the other side, systems publish and subscribe to events through message broker infrastructure. The relay publishes integration events through any Chatter message broker infrastructure implementation, such as Azure Service Bus.

An integration event is the message the relay publishes. An attribute names the topic it goes to, and consumers subscribe to that topic.

[BrokeredMessage("order-shipped-event")]
public class OrderShippedEvent : IEvent

README: Features

3

Chapter 3 of 9

Watch the table

The bridge is a change source. It watches your tables for row changes. Each row change becomes a source event: the row’s values, typed by the mapping’s source event type.

Two change sources are available, and the stage draws them differently on purpose: SQL Service Broker is the dashed teal bridge with square packets, and SQL Server CDC is the solid rust bridge with round packets. One relay can use both, but each table uses only one.

Our running example is an Orders table. When OrderStatus changes to Shipped, the relay should publish an OrderShippedEvent. The relay runs in a .NET Worker Service, and it checks its configuration when the host starts and stops with a list of problems.

Row change

An insert, update or delete of one row.

Source event

The row’s values, typed by the mapping’s source event type.

Change source

What watches the tables: SQL Service Broker or CDC.

Source

A named, configured instance of a change source. Each mapping names one.

README: How it works

4

Chapter 4 of 9

The mapper decides

A mapping in configuration pairs a table and a change type with a source event type and an integration event type. Each mapping has exactly one mapper.

The mapper is your code. It decides whether the source event becomes an integration event and builds it. Return null and nothing is published. For updates the mapper sees both OldValue and NewValue.

// mappingData.OldValue only has a value for an update
if (mappingData.OldValue?.OrderStatus != "Shipped" && mappingData.NewValue?.OrderStatus == "Shipped")

In the example, OrderStatus changing to Shipped makes the mapper publish OrderShippedEvent to the order-shipped-event topic.

Mapping

Pairs a table and change type with a source event type and an integration event type.

Mapper

Your code: decides whether the source event becomes an integration event, and builds it.

Integration event

The message the relay publishes.

5

Chapter 5 of 9

Publish

The relay hands the integration event to the message broker infrastructure you configured. The topic named by [BrokeredMessage("order-shipped-event")] receives it, and consumers subscribe to that topic.

The legacy app never changed. It wrote one row, and the event-driven side heard an event.

Processing Update of 'OrderChangedEvent' from source 'orders-cdc' with 1 mapping(s)
Published integration event 'OrderShippedEvent' to messaging infrastructure

README: Quick start

6

Chapter 6 of 9

Two ways across

Both bridges end at the same relay and the same mappers. They differ in how they see a change, and in what they promise. Pick the one that fits the table.

A mapping can require capabilities with RequiredCapabilities. The relay stops at startup if the mapping’s source doesn’t provide them.

README: Choosing a change source README: Using the CDC change source

7

Chapter 7 of 9

Keep your place

Both change sources deliver at least once. That is a promise to plan around, so here is what keeps your place and what to do about repeats.

Checkpoint and same-id replay

With CDC, a batch is checkpointed only after every change in it is published. After a crash the batch is republished with the same message ids, derived from the source name, the change’s position and the integration event type. Enable duplicate detection on the topics (Azure Service Bus) or make consumers idempotent.

With SQL Service Broker, Chatter acknowledges by committing its receive transaction once processing succeeds, and Chatter’s recovery options retry failures.

Lease

Several relay instances can run with the same configuration. A lease (a SQL Server application lock) lets one of them poll each capture instance while the others stand by.

When a change keeps failing

Transient failures are retried with exponential backoff and never parked. For anything else you choose an error policy:

• Halt (default) stops at the change and keeps retrying. Nothing after it is published, so order holds.
• ParkRow parks the change in <Schema>.Chatter_RelayParkedChanges after MaxAttemptsBeforePark attempts. Later changes to the same row park behind it, and other rows carry on. On request, a replay republishes the row’s parked changes in order.

UPDATE Chatter.Chatter_RelayParkedChanges SET ReplayRequestedAt = SYSUTCDATETIME()
WHERE SourceName = N'orders-cdc' AND CaptureInstance = N'dbo_Orders' AND RowKey = N'<row key, e.g. [42]>';

Watching it

Metrics are published on the meter Chatter.IntegrationEventRelay.Cdc: chatter.relay.cdc.changes, chatter.relay.cdc.parked, chatter.relay.cdc.batch.failures, chatter.relay.cdc.batch.duration, chatter.relay.cdc.lag and chatter.relay.cdc.parked.current.

services.AddHealthChecks().AddCdcChangeSource(maxLag: TimeSpan.FromMinutes(5));

Tested SQL scripts enable CDC and grant the relay’s login least-privilege permissions. The integration tests run the relay under exactly those grants.

README: Delivery semantics README: Error policy and parked changes README: Metrics and health check

8

Chapter 8 of 9

Change bridges safely

You can move a table from one bridge to the other. Two scripts do it: .scripts/cdc/switch-to-cdc.sql and .scripts/cdc/switch-to-service-broker.sql.

Each script picks a switch point. The source you switch away from relays every change committed before it, and the source you switch to relays every change committed after it. Every change is relayed by only one of the two sources, and each source is still at least once.

• The switch commits in one transaction, or the script rolls all of it back and changes nothing.
• The script locks the table, so writes to it pause until the script ends, normally a few seconds. Its waits for the relay, the capture job and the CDC sources end within about @timeoutSeconds (default 300); past it, the script rolls back and changes nothing. Some reads, the checks before the lock and, switching back, SqlChangeFeed’s install procedures and the table’s triggers aren’t bounded by @timeoutSeconds. If the script doesn’t finish, cancel it: that rolls it back too.
• Mappers don’t change.
• Integration tests run both scripts in both directions.

README: Switching a table from SQL Service Broker to CDC

9

Chapter 9 of 9

Start

Three NuGet packages. The core maps row changes to integration events and publishes them, and a change source package captures the changes.

• Chatter.IntegrationEventRelay

  Core: mapping configuration, mapper interfaces, and the processor that maps row changes to integration events and publishes them. It doesn’t capture changes itself.

  Chatter.IntegrationEventRelay on NuGet

• Chatter.IntegrationEventRelay.ServiceBroker

  The SQL Service Broker change source. It uses Chatter.SqlChangeFeed, which installs a trigger on each watched table that sends changed rows to a SQL Service Broker queue.

  Chatter.IntegrationEventRelay.ServiceBroker on NuGet

• Chatter.IntegrationEventRelay.Cdc

  The CDC change source. It polls each capture instance’s change table, hands changes to the processor in commit order, and saves a checkpoint in the source database once a batch is published.

  Chatter.IntegrationEventRelay.Cdc on NuGet

It targets .NET 10 (net10.0) and runs in a .NET Worker Service.

dotnet add package Chatter.IntegrationEventRelay
dotnet add package Chatter.IntegrationEventRelay.ServiceBroker
dotnet add package Chatter.MessageBrokers.AzureServiceBus
dotnet add package Chatter.MessageBrokers.AzureServiceBus.Auth

// Wires up the mapper of each mapping in configuration
.AddIntegrationEventRelay()
    // Captures changes to the mapped tables with triggers and SQL Service Broker
    .UseServiceBrokerChangeSource();

.AddIntegrationEventRelay()
    .UseCdcChangeSource()
    .UseServiceBrokerChangeSource();

dotnet add package Chatter.IntegrationEventRelay.Cdc

public class OrderShippedMapper : IMapSourceUpdateToIntegrationEvent<OrderChangedEvent, OrderShippedEvent>
{
    public Task<OrderShippedEvent?> MapAsync(MappingData<OrderChangedEvent> mappingData, RelayContext context, EventMappingConfigurationItem? mappingConfig)
    {
        OrderShippedEvent? orderShippedEvent = null;

        // mappingData.OldValue only has a value for an update
        if (mappingData.OldValue?.OrderStatus != "Shipped" && mappingData.NewValue?.OrderStatus == "Shipped")
        {
            orderShippedEvent = new OrderShippedEvent
            {
                Id = mappingData.NewValue.Id,
                CustomerId = mappingData.NewValue.CustomerId,
                ProductId = mappingData.NewValue.ProductId,
                OrderPlacedOn = mappingData.NewValue.OrderPlacedOn,
                OccurredAt = DateTime.UtcNow
            };
        }

        return Task.FromResult(orderShippedEvent);
    }
}

"IntegrationEventRelay": {
  "ConnectionString": "...",
  "Schema": "Chatter",
  "Sources": {
    "orders-cdc": { "Type": "Cdc", "PollInterval": "00:00:01" }
  },
  "Mappings": [
    {
      "Source": "orders-cdc",
      "SourceEventTypeName": "MyNewIntegrationEventRelay.OrderChangedEvent",
      "IntegrationEventTypeName": "MyNewIntegrationEventRelay.OrderShippedEvent",
      "SourceChangeType": "Update",
      "TableName": "Orders",
      "TableSchema": "dbo",
      "CaptureInstance": "dbo_Orders"
    }
  ]
}

Then ship the seeded order and watch the event arrive:

UPDATE [dbo].[Orders]
   SET [OrderStatus] = 'Shipped'
 WHERE Id = '52CBE9FF-2496-4F46-9331-7F122AA3D973'
GO

Or run the demo

No Azure subscription needed. This starts SQL Server, the Azure Service Bus emulator, two Worker Services and two Consumer Services.

docker compose up --build

The demo uses both change sources. 100 database operations produce:

• 50Event1DeletedEvent via CDC
• 25Event1RestoredEvent via CDC
• 25Event2CreatedEvent via SQL Service Broker

README: Quick start README: Packages README: Running this repo locally