Trading System Latency Measurement, Stage by Stage

Good trading system latency measurement works per stage, not only end to end. Split the path from market data to order into named stages, timestamp each with one monotonic clock, compute percentiles from raw samples, and watch the tail, because bursts are where latency hurts. Test with tape replay and alert when a stage's tail drifts.

TL;DR: Averages hide bursts, so measure p50, p99, and max per stage. On a private options platform we instrumented 12 stages; in tape replay, market-data decode ran at 2.5 µs p50 and 8 µs p99, and signal to order took about 0.5 ms. Use monotonic clocks, keep logging off the hot path, and alert on sustained tail drift.

Why do average latency numbers lie?

Averages lie because they smooth away the moments that matter. A strategy that decides in 20 microseconds most of the time and 5 milliseconds during a burst of ticks has a reassuring average and a bad day. Bursts are exactly when markets move, so the slow cases are the expensive ones.

The useful numbers are percentiles: p50 for the typical case, p99 for one event in a hundred, and the maximum over a window. How they're computed matters too. Prometheus's own documentation says that for a classic histogram, quantile error "is limited by the width of the bucket the quantile is located in" (Prometheus, histograms and summaries). If your buckets are 10 µs wide, you can't see a 3 µs regression in the tail.

We built latency instrumentation this way into a private options trading platform for a confidential client. It trades index options on an Indian exchange and covers tick capture, research and backtesting, paper and live execution, and a control dashboard. Client, broker, and strategy details are withheld, and every figure here was measured in tape replay, not live trading.

How do you split a trading path into stages?

Split it wherever work changes hands or changes kind: network to process, bytes to event, thread to thread, decision to order. End-to-end latency tells you there's a problem. Stage latency tells you where. The platform is instrumented across 12 stages, exported to Prometheus and Grafana with alert rules.

The table is a generic 12-stage breakdown at the same level of detail. The platform's own stage names aren't published, so treat this as a template rather than its exact list.

#	Stage	Question it answers
1	Network receive	How long until a tick reaches the process?
2	Socket read	How long to pull the bytes off the socket?
3	Decode	How long to parse the message into an event?
4	Normalize	How long to map it to the internal instrument and format?
5	Hand-off to engine	How long does the event wait to be picked up?
6	Engine dispatch	How long until the right handler runs?
7	State update	How long to update books, bars, and indicators?
8	Signal	How long does the strategy take to decide?
9	Risk check	How long do pre-trade checks take?
10	Order build	How long to create the order message?
11	Adapter submit	How long until the broker adapter accepts it?
12	Wire send	How long until it leaves the machine?

What did the platform measure?

In tape replay, market-data decode ran at 2.5 µs at p50 and 8 µs at p99, the risk check at 21 µs at p50, and signal to order created took about 0.5 ms. Those four figures are the ones we can publish. The hardware and sample size behind them aren't published alongside the client's other details, so read them as one setup's replay figures, not a benchmark.

Measurement	Value
Market-data decode, p50	2.5 µs
Market-data decode, p99	8 µs
Risk check, p50	21 µs
Signal to order created	about 0.5 ms

What does "exact percentiles" actually mean?

It means the percentile is computed from the recorded sample values themselves, not read off a bucket boundary. The platform records into what its code calls exact percentile reservoirs: per stage, a bounded window of raw samples, with p50, p99, and max computed from those values and then exported.

"Reservoir" can mislead, because in statistics reservoir sampling keeps a random subset. Here the point is the opposite: inside the window, nothing is sampled away, so a rare slow event can't vanish. The trade-off is memory, and a window has to be sized to hold every sample from its period at peak rate.

The main alternative is a high-dynamic-range histogram. HdrHistogram expresses precision as a number of significant digits, keeps a memory footprint that's "fixed regardless of the number of data value samples recorded," and records values in a few nanoseconds (HdrHistogram). Prometheus summaries are another option, but their quantiles can't be aggregated across instances. Pick one method, write down its error, and quote it with every number.

How do you record latency without adding latency?

Keep recording cheap and move everything else off the hot path. Measurement that slows the system measures itself.

1. Take timestamps from a monotonic, high-resolution clock. In Python, time.perf_counter_ns() gives nanoseconds from a counter with "the highest available resolution to measure a short duration," and only differences between two calls are meaningful (Python, time module).
2. Write samples into a fixed-size structure in memory.
3. Compute percentiles in a separate step, not on every event.
4. Export the results to your monitoring stack, such as Prometheus, and chart them in Grafana.

Which clock and environment mistakes skew the numbers?

Mixing clocks, noisy machines, warm-up, and hot-path logging cause most bad latency numbers.

1. Mixing clocks. Comparing an exchange timestamp with one from your own machine measures clock skew as much as latency. Measure in-process stages with one monotonic clock, and report cross-machine numbers separately.
2. Measuring on a laptop. Power saving, background apps, and thermal throttling add noise. Measure on hardware that matches production, and say what you measured on.
3. Warm-up. The first seconds after start include loading and allocation. Exclude them from reported figures.
4. Logging on the hot path. A log line per tick can cost more than the work being measured. Buffer it and write elsewhere.

Keep the hot path free

Design matters as much as measurement: keep anything that can block or allocate away from the engine loop, and let slow consumers fall behind without dragging trading with them. On the platform, market-data decode runs on its own thread and is handed to the engine loop thread-safely; nothing blocking ever sits on that loop.

Market state reaches the dashboard through Redis and a WebSocket gateway, throttled per instrument. The publisher is bounded and drops when full, so it never blocks the trading path, and if the gateway dies, trading doesn't notice. Instrument identity comes from a snapshot map, never derived from strings on the hot path.

Why use Python at all? Because research and data tools live there. The engine is built on a Rust-core event-driven engine, Python carries strategy logic and tooling, and order logic is kept free of engine imports so it can be ported to Rust later. Research runs on a Parquet tick archive queried with DuckDB and Polars.

Test with tape replay

Live markets aren't a test environment; replay gives identical input twice. Record market data, then replay it through the system at real speed or faster.

• Compare latency before and after a code change on the same ticks.
• Reproduce a burst that caused trouble.
• Run one strategy in research, paper, and live modes and confirm the code path is the same.

Replay numbers aren't live numbers. Network, broker, and exchange behaviour differ in production. Say which one you're quoting, every time. That habit predates trading for me: my first degree was in instrumentation and control engineering, where you name every stage, measure it, and alarm on the tail, not the mean.

When should latency alerts fire?

They should fire on a sustained shift in the tail, not on a single slow event. Prometheus alert rules support this with a for clause, which makes Prometheus wait a set duration before an alert counts as firing; until then it stays pending (Prometheus, alerting rules).

1. A stage's p99 stays above its usual band for several minutes.
2. The hand-off queue between threads grows, meaning a consumer is falling behind.
3. The publisher drops constantly rather than occasionally.
4. Any stage's maximum jumps after a deploy.

Risk checks belong in this measurement like any other stage: measure them and keep them fast, but never skip them. How the platform's position caps, per-bar guard, and kill switch work is the subject of algo trading risk controls, and our algo trading software page covers the service.

FAQ

What is a good p99 latency for a retail algo trading system?

There's no universal number; it depends on the strategy, broker, and exchange. A strategy reacting over seconds gains little from shaving microseconds, while one reacting to each tick does. Measure your stages first, find the largest one, and decide whether improving it changes outcomes before spending effort on it.

Should latency be measured in production or only in replay?

Both, but report them separately. Replay gives repeatable input for comparing code changes. Production shows the real network, broker, and market behaviour. Keep in-process stage metrics running live, because they cost little, and use replay to explain any change you see there.

How many latency stages are too many?

Too many is when nobody looks at them. Each stage should answer a question you'd act on, such as "is decode slower after this release?" The platform uses twelve. Merge stages that always move together, and split one that keeps surprising you.

Can the client behind the trading platform be named?

No. The client is confidential, and broker and strategy details are withheld. The published figures come from tape replay with the conditions stated here. We're happy to walk through the measurement approach in detail, under NDA, with anyone evaluating us for similar work.

Building or speeding up a trading system? See our trading platform development work, or our general performance optimization service, and we'll sign an NDA before the first technical call.