What did the book look like at 14:03:07? Recording market data for disputes

A client says their fill was off-market. To answer you need two things: the price they were shown when they clicked, and what the market was doing around that moment. If all you have is application logs, you're reconstructing it from memory and guesses.

The recording has to exist before the question comes up, which is why it's worth deciding early what to keep.

Three layers, each answering a different question. The raw messages from each LP as they arrived, with your own receive timestamp. The composite you actually published for each symbol, with a sequence number, since that's what clients saw. And the orders and fills, with the price shown to the client at decision time and the LP's side of each execution.

Recording only the composite is cheaper, but then if something looks wrong you can't tell whether the composite was off or the inputs were. Recording only the raw feeds means rebuilding the composite later, and that assumes the code and config are the same as when it ran. Store the version of both alongside the data.

Disputes are about milliseconds, so the clocks matter. Stamp messages with your own clock at the point of receipt. Use a monotonic clock for ordering events within a process and wall-clock time for display, and store everything in UTC. Keep the host clocks synced (NTP is the minimum, and know how far off it can be).

LPs send their own timestamps too, and they're useful, but they come from the LP's clock. Keep both. Don't overwrite one with the other, or you lose the ability to see how late a message really was.

Tick data adds up quickly, and storing all of it forever isn't realistic for most firms. One way to split it: full raw capture for a short window, the published composite for longer, and everything around an order (some time before and after each fill) for as long as disputes or regulators might ask. Compress it, partition it by day and symbol, and ask compliance what you're actually required to keep in your jurisdiction. That's a rules question, not an engineering one.

Build a way to push recorded raw data back through the same normalization and aggregation code, at whatever speed you like, and compare the output with what was published. If they match, you've got a defensible answer. If they don't, either the recording has a gap or the code changed since. Both are things you want to know.

Start with the published composite and the order records, since those settle most disputes, and add raw capture when you can afford the storage.