Brett Harrison@BrettHarrison
Optimizing Exchange Latency
At HFTs I built low-latency trading systems for traditional and cloud-native exchanges including Nasdaq, NYSE Arca, CME, ICE, SGX, JPX, Coinbase/GDAX, and others. A summary/discussion of the current split in low-latency network infrastructure
Colocation: Traditional exchanges are hosted in physical datacenters. Colocation involves renting rack space with cross-connects to the exchange matching engine. While it is expensive and time-consuming to set up, physical colocation guarantees an equalized path to the exchange. By contrast, it is fast and inexpensive to set up cloud servers but difficult to guarantee the lowest latency. To be colocated with an exchange running in AWS, firms first need to understand not only what region the exchange is deployed in (e.g. us-east-1) but also the availability zone (e.g. use1-az4). Even within a single availability zone, there is variance in how close a randomly placed EC2 instance lands to the matching engine.
Non-determinism: Networking setups for traditional exchanges offer deterministic access to matching engines by running the same cable lengths to all client boxes. Public clouds have a significant degree of non-determinism due to routing within a cloud region. Two different EC2 instances in us-east-1/use1-az4 could have different latencies to the same matching engine in the same zone. Firms mitigate this issue by spinning up multiple instances and choosing the one with lowest ping times, as well as using cluster placement groups and AWS technology such as ENA Express.
Unicast instead of multicast: Traditional exchanges disseminate market data over UDP multicast, which puts a single copy of the book on the wire and delivers it to every participant simultaneously. Public clouds don't offer multicast in any form usable for market data: VPCs have no native support, and Transit Gateway multicast is built for enterprise applications rather than microsecond-sensitive feeds. Cloud-native exchanges therefore publish over TCP unicast, which involves a separate copy for every subscriber and a fan-out order that no longer treats them equally. HFTs recover what they can by bypassing the kernel for their networking stack. They use a poll-mode driver such as DPDK over ENA that lets the NIC DMA frames directly into userspace ring buffers, with a userspace TCP stack handling the protocol above it.
Inter-region latency: Expensive microwave, millimeter-wave, and subsea fiber links carry the lowest-latency paths between physical datacenters, and none of them terminate inside a cloud region. To move data between cloud regions and traditional venues, firms combine the long-haul routes of established market data vendors with specialized cloud on- and off-ramps, paying a penalty at each transition. Since price discovery still originates largely from traditionally colocated exchanges, trading on public cloud venues adds to a firm's market data infrastructure footprint rather than replacing it.
Partnerships between CME and Google and between Nasdaq and Amazon aim to deliver infrastructure that combines the flexibility and global reach of public clouds with the deterministic latency of purpose-built exchange datacenters. If that convergence arrives, most of the practices above stop paying. The advantage in trading on cloud-native exchanges may shift back toward the firms with decades of colocated experience rather than those who have built expertise in cloud-latency engineering.
