photon

A nanosecond-scale NASDAQ ITCH 5.0 feed handler and matching engine. photon replays a real trading day over multicast UDP, parses it off the wire with zero copies, and crosses the historical order flow in a price-time priority book, one book per symbol, while measuring every nanosecond with the CPU's cycle counter.

Median parse latency is 38ns. Socket to fully consumed is 452ns. No dependencies, no locks, no heap allocation on the hot path, and every number is reproducible on your own machine.

• Architecture and design rationale
• Benchmarks and methodology
• Contributing

What it looks like

One second of a real NASDAQ trading day flowing through the pipeline:

=== PHOTON LATENCY REPORT (1s window) ===
Events processed : 89317
Parse latency    : p50=  38ns  p95=  48ns  p99=  56ns  p99.9= 1616ns
Total latency    : p50= 452ns  p95= 471ns  p99= 490ns  p99.9=10001ns
Ring drops       : 0
Matcher (Lithium): symbols=826  trades=20352  shares_filled=6474004  notional=$316986693.20  cancels=18865  dropped=0
Tightest book    : best_bid=$20.95  best_ask=$20.96  spread=$0.01

That's 89k real exchange messages received, decoded, queued, and matched across 826 simultaneous order books in one second, with the end-to-end p99 still under half a microsecond.

Contents

• Features
• Benchmarks
• How it works
• Why photon, and why not
• Getting started
• Reading the output
• Configuration
• Project layout
• Tests
• Troubleshooting
• License

Features

• Real data, not synthetic. Replays full-day NASDAQ TotalView-ITCH 5.0 files (public, ~8 GB, tens of millions of messages), paced by the nanosecond timestamps embedded in the feed. --speed 1 reproduces the original day's microstructure in real time; --speed 0 runs flat out.
• Zero-copy parsing. ITCH messages are decoded by casting packed structs onto the receive buffer. Struct layouts are static_asserted against the wire sizes from the spec, so a packing mistake fails the build instead of corrupting data.
• Busy-polled receive path. SO_BUSY_POLL plus non-blocking recvmmsg in batches of 256 datagrams per syscall. The receiver never sleeps and never eats an interrupt wakeup on the critical path.
• Lock-free SPSC ring buffer. Head and tail on separate cache lines, each side caching the other's index, acquire/release ordering and nothing else. Steady-state pushes and pops generate zero cross-core cache traffic.
• A real matching engine. Lithium keeps one price-time priority book per symbol: array-indexed price levels (O(1) insert), intrusive FIFO order queues, and a fixed memory pool, so matching allocates nothing.
• Honest measurement. Timestamps come from rdtscp, calibrated at startup. Latency lands in 1ns histogram buckets with an explicit overflow bucket, so a scheduling hiccup shows up as "off the chart" instead of silently skewing an average.
• Zero dependencies. The pipeline is plain C++20 and the Linux syscall interface. GoogleTest is fetched only if you build the tests.

Benchmarks

i7-14700HX, Linux 6.x, full-day ITCH file replayed unthrottled, ~88,000 events/sec sustained, 700+ symbols with live books, zero drops:

Metric	p50	p95	p99
Parse (socket to decoded)	38 ns	48 ns	57 ns
End-to-end (socket to consumed)	452 ns	471 ns	490 ns

Reproduce with:

cmake --preset release && cmake --build --preset release
./scripts/download_itch.sh
./scripts/run_demo.sh

docs/benchmarks.md documents the methodology, exactly what each number includes and what it doesn't (these are loopback numbers; there is no NIC or wire in them), and the machine settings that move the tail.

How it works

exchange_sim --(multicast UDP, real ITCH 5.0 bytes)--> photon --> Lithium
 replays a real                            receiver: zero-copy    per-symbol
 NASDAQ trading day,                       parse + SPSC ring      order books,
 paced by embedded                         consumer: latency      price-time
 exchange timestamps                       histograms             matching

Three pinned threads, one syscall boundary, one queue:

1. exchange_sim (core 0) streams the ITCH file, one message per datagram, paced against the feed's own timestamps.
2. photon's receiver (core 1) busy-polls the socket, decodes each message in place, stamps it with rdtscp, and pushes a 64-byte event (exactly one cache line) into the ring.
3. photon's consumer (core 2) pops events, records parse and end-to-end latency, and feeds adds/cancels/executions into the per-symbol books. Cancels are routed to the right book even though ITCH cancels don't carry a symbol.

Every design decision (why busy-polling, why the ring buffer looks the way it does, why the book is a flat array and not a map) is written up in docs/architecture.md.

Why photon, and why not

Use photon if you want a working, measured reference for the low-latency building blocks trading systems are made of: kernel busy-polling, zero-copy binary protocol handling, SPSC queues without false sharing, TSC-based timing, cache-line-conscious data layout, and order book design. The whole pipeline is small enough to read in an afternoon and instrumented enough to verify every claim.

Don't use photon if you need a production feed handler. There is no MoldUDP64 framing, no gap detection or retransmission, no kernel bypass, and the matcher deliberately caps prices at $1,000/share to keep its flat-array price levels affordable per symbol. The limits section of the architecture doc lists every such trade and why it was taken.

Getting started

Requirements

• Linux, x86-64 (the pipeline is built on rdtscp, recvmmsg, and core pinning)
• CMake 3.20+, Ninja, and a C++20 compiler (clang or gcc)
• ~10 GB of disk for a full-day ITCH file

On macOS or Windows, use a Linux VM or WSL2.

Build

git clone https://github.com/xevrion/photon.git
cd photon
cmake --preset release
cmake --build --preset release

Get data

NASDAQ publishes historical TotalView-ITCH sample days for free:

./scripts/download_itch.sh    # ~1.8 GB compressed, unpacks to ~8 GB

Run

./scripts/run_demo.sh         # unthrottled (sudo: multicast route + socket opts)
./scripts/run_demo.sh 1       # real time, the actual pace of the trading day

Or drive the two sides yourself:

# terminal 1: feed handler + matcher
sudo ./build/release/photon

# terminal 2: exchange feed
./build/release/exchange_sim --speed 0 data/*.NASDAQ_ITCH50

sudo is optional; without it photon falls back gracefully (you lose SO_BUSY_POLL and the large kernel receive buffer, and the tail gets wider).

Reading the output

• Parse / Total latency: socket-to-decoded and socket-to-consumed percentiles for the last second. 10001ns is the histogram's overflow bucket ("past the 10us we track"), not a measurement.
• Ring drops: events dropped because the consumer fell behind. Should be 0 in steady state; if it climbs, the consumer is the bottleneck.
• Matcher: fills from crossing the historical flow. notional is real dollar volume (price x quantity, summed over every fill). Volume, not P&L: the replayed flow is anonymous market data, there is no "our side". dropped counts orders rejected by the price cap or a full order pool.
• Tightest book: best bid/ask of whichever symbol currently has the tightest spread, as one live example of a real book.

Configuration

Flag	Binary	Default	Meaning
--speed N	exchange_sim	0	Replay pace. 1 = real time, 2 = twice as fast, 0 = unthrottled
--group ADDR	both	239.1.1.1	Multicast group
--port N	both	30001	UDP port
--core N	exchange_sim	0	Core to pin the replayer to
--rx-core N	photon	1	Core for the receiver thread
--consumer-core N	photon	2	Core for the consumer thread

On hybrid CPUs, make sure the pinned cores are P-cores.

Project layout

src/common/        wire-format structs, SPSC ring, TSC helpers, core pinning
src/photon/        receiver + consumer threads, latency histogram
src/matcher/       Lithium: order book, memory pool, types
src/exchange_sim/  ITCH file replayer
src/tools/         calibrate_tsc, a standalone TSC sanity check
tests/             GoogleTest suite
docs/              architecture and benchmark write-ups
scripts/           demo runner, data download, perf, format, lint

Tests

cmake --preset debug && cmake --build --preset debug
ctest --preset debug

27 tests cover the ring buffer (including a two-thread stress test), the matcher's fill/cancel/priority rules, wire decoding against hand-built big-endian buffers, and the histogram's percentile math. A sanitize preset runs the same suite under ASan + UBSan. CI runs formatting, clang-tidy, and both presets on every push.

Troubleshooting

Both programs run, no errors, photon shows nothing. Linux doesn't route multicast over loopback by default: everything binds and joins successfully, packets are sent, and the kernel never delivers them locally. Fix:

sudo ip route add 239.1.1.1/32 dev lo

(run_demo.sh does this automatically.) To watch the packets move:

sudo tcpdump -i lo udp port 30001 -c 10

p99 reads 10001ns at low replay speeds. At --speed 1 in pre-market, a window may carry only a handful of events, so a single scheduling hiccup owns the tail. That's the overflow bucket doing its job.

Latency looks worse than the table above. Check the frequency governor (performance), make sure nothing else runs on the pinned cores, and confirm those cores are P-cores. docs/benchmarks.md ranks the usual suspects.

License

MIT. If you build something interesting on top of it, open an issue; happy to link it here.