Clictic Field notes on running Endur

A note on prices go stale and nobody notices

Dynamic RabbitMQ shovels for resilient market-data distribution

How a chain of dynamic shovels gives an ETRM platform a market-data fabric that survives broker outages and reconfigures without a restart.

You can add, retarget or tear down a route without touching the producers or consumers, and without a restart.

By Harpreet. Connectivity, May 2026, 4 min read

Most ETRM platforms treat market data as a solved problem until the morning a feed goes quiet and half the curve is stale. The usual culprit isn’t the upstream vendor — it’s the brittle, hard-wired plumbing between the feed handler and the platform. RabbitMQ’s dynamic shovel is a quietly excellent tool for fixing that, and it deserves more attention than it gets in trading-tech circles.

The problem with point-to-point feeds

The naive design wires a market-data adapter straight into the platform’s import queue. It works on day one. Then you add a second venue, a UAT environment that needs a copy of the same prices, and a downstream analytics consumer — and suddenly the adapter is publishing to four places, each with its own retry logic and its own way of falling over. When one consumer is slow, back-pressure propagates to all of them. When you need to reroute traffic during an incident, you’re editing config and restarting processes during the one window where you least want to.

Shovels as a routing fabric

A shovel is just a reliable, asynchronous link that consumes from one queue and republishes to an exchange — possibly on a different broker entirely. A dynamic shovel is one you declare at runtime through the management API or a policy, rather than baking into the broker config file. That distinction matters: it means you can add, retarget or tear down a route without touching the producers or consumers, and without a restart.

The pattern I keep coming back to is a short chain rather than a single hop:

feed handler ──▶ ingress exchange
                     │  (dynamic shovel)
                     ▼
              mapping / enrichment
                     │  (dynamic shovel)
                     ▼
              platform import exchange

Each stage owns one responsibility. The ingress stage normalises and timestamps whatever the venue sends. The mapping stage translates external instrument identifiers to the platform’s internal symbology and drops anything the platform doesn’t care about. The final shovel delivers into the import exchange the platform already listens on. Because every hop is a shovel, every hop is independently observable, independently restartable, and independently rerouteable.

Why dynamic beats static here

Three properties make the dynamic variant worth the small extra setup:

  • Reconfigure live. Need to fail a venue over to a backup feed, or fan a copy of production prices into a test environment for a load test? Declare a new shovel, watch it drain, delete it. No producer ever knows.
  • Back-pressure stays local. A slow downstream consumer fills its own queue and trips its own alarm. It doesn’t starve the feed handler, because the shovel between them absorbs the lag up to the queue limit you set.
  • Failure is visible. A stalled shovel shows up immediately in the management API as an unacknowledged backlog on a named queue. That’s a far better alert than “prices look old” reported by a trader.

The details that bite

A few things are worth getting right before this goes near production:

  1. Acknowledgement mode. Use on-confirm so a message isn’t removed from the source until the destination has confirmed it. The throughput cost is real but small, and the alternative is silent data loss on a broker hiccup.
  2. Prefetch and queue limits. Set an explicit prefetch-count and a max-length policy on intermediate queues. Unbounded queues turn a slow consumer into an out-of-memory event at three in the morning.
  3. Idempotent consumers. on-confirm gives you at-least-once delivery, which means the platform import side must tolerate the occasional duplicate. Key on the venue’s own message id and dedupe.
  4. Reconnect behaviour. Dynamic shovels reconnect automatically, but tune reconnect-delay so a flapping link doesn’t hammer a recovering broker.

What you get

The payoff is a market-data layer you can reason about. Each link is a named, monitored thing with a queue you can watch and a route you can change on the fly. When a venue has a bad morning, the blast radius is one shovel and one queue — not the whole platform — and you can reroute around it while everyone’s still drinking their first coffee.

It isn’t glamorous infrastructure. But “boring and observable” is exactly what you want sitting between a volatile data feed and the system your traders price off.

Is this happening on your desk?This is the consulting work I do. Tell me what you're seeing and I'll say whether I can help.

Send a consulting enquiry

Read next

All notes

One Endur note a month, by email

Each one covers a single problem and how to fix it. Free to read, written by someone who still does the work.

Subscribe by email No sales emails. Unsubscribe any time.