post-mortem · nano_match · C++ · 3 pages
The Order That Was in Two Places
a cancelled order went back to the pool while a price level still pointed at it. the next order reused the slot, the book crossed, and nothing crashed.
abstract
A use-after-free that never crashed. The engine kept running and kept producing fills, at prices that were not real and in an order that violated the single guarantee a matching engine exists to provide. That it stayed silent was the problem.
the symptom
The book crossed: a bid resting above the best ask, which cannot survive a matching pass in a correct engine. Fills also came out of sequence within a price level. No crash, no assertion, no allocator complaint. Every pointer being followed was a legal pointer to a live order object. It was simply the wrong one.
root cause
Cancellation after a partial fill released the order to the pool without unlinking it from its price level first, a step the full-fill path did do. The next inbound order acquired that same slot and linked it into a different level. One object, two intrusive lists, one set of link pointers between them. The old level now walks into an order at a price that has nothing to do with it.
the fix
Unlink before release, and then stop trusting call sites to remember: the pool asserts on release that the object is linked nowhere, and a generation counter makes any future instance provable from a log. Fixing the caller fixes an instance. Enforcing at the boundary fixes the class.