Exablaze is now part of Cisco.

Supermicro HFT-Optimized Servers and the ExaNIC

Author David Mann - Date 2014/08/04

Today I want to bring attention to the fact that the ExaNIC is qualifiedfor use on Supermicro's HFT-Optimised seriesof servers. The great news here is that when combined with this server the ExaNIC latency improves dramatically - we're consistently seeing a 100ns reduction in latency, which we think is very impressive.

Testing on the Supermicro HFT-3

We find the best way to get an idea of the relative performance of the ExaNIC in one server compared to another is via our raw performance testing application 'exanic_loopback'. We distribute this utility along with our driver, and it's conceptually very simple. First, we connect together two ports on the card with 1 metre of fibre, and then run the utility, which:

  1. Takes a timestamp using the host's timestamp counter,
  2. Copies a raw frame to the card and triggers a send from one port,
  3. Waits until that frame goes out on the wire, back into second port and up into userspace memory,
  4. Takes another timestamp using the host's timestamp counter

This allows us to measure the total latency all the way from userspace on the host, down to the card, out onto the wire, back into the card and back to userspace.

I've produced the following graph by running exanic_loopback on a Supermicro HFT-3 server that has an ExaNIC X2 installed, and configured it to collect 100,000 latency measurements. The graph is a histogram of these measurements.

Histogram showing 850 ns median latency for ExaNIC loopback in supermicro HFT

From this we can see that the application - wire - application latency for the ExaNIC in the Supermicro server has a median of 850ns, which is certainly very impressive. This compares to the median latency we see in most non-overclocked Sandy Bridge servers of around 950ns.

So how come it's so much faster? A couple of reasons, but we believe it's mostly due to the way Supermicro tune their servers, particularly with regard to overclocking. We've recently rolled some new firmware for the ExaNIC that has better support for overclocked PCIe buses, which we provide via our support portal to existing customers. Feel free to ask us about the specifics if you need more details.

Getting a Supermicro server

We've been working pretty closely with Supermicro in the USA, and also with an IT consultancy and solution provider in the UK - BIOS-IT, who offer Supermicro server based solutions. We recommend both depending on where you are based. Both Supermicro and BIOS-IT can offer a full server solution including the ExaNIC, specifically tuned for high frequency trading applications.