Three cool features of the ExaNIC

Today I wanted to give everyone a flavor for some of the key features of ExaNIC, our 10GbE and 1GbE network interface card. Of course, the primary feature of the ExaNIC is its low latency (950 ns application to wire to application) but it can also do some other cool things. In this post I'll go over precise hardware timestamping, port bridging and port mirroring.

Hardware based timestamping

Every packet that traverses the card (both received and transmitted) is timestamped to nanosecond level resolution (6.2 nanoseconds). This timestamping is done in hardware (on the ExaNIC FPGA), and you can readily synchronise the internal timer to either the host's clock or via a separate pulse-per-second (PPS) input (from a GPS, say) on the bracket of the card. Synchronising to one of these sources is as simple as running our utility (in this case PPS sync):

$ exanic-clock-sync exanic0:pps

The utility then trains the clock on the card to the PPS input.

exanic0: Starting clock discipline using PPS
exanic0: Nominal frequency: 161132800 Hz
exanic0: Using differential PPS input
exanic0: PPS signal detected
exanic0: Clock offset at PPS pulse: -9305478.518 us  drift: 0.000 ppm
exanic0: Clock offset at PPS pulse: -17.253 us  drift: 0.000 ppm
exanic0: Clock offset at PPS pulse: -0.019 us  drift: -16.794 ppm

Once the clock on the card has been trained, there are a number of ways to access the packet timestamps. Perhaps the easiest way to get going is to use our capture utilities to write pcap files that maintain the nanosecond level timestamps. We also provide an API that makes these timestamps available to software.

Port bridging

Port bridging lets the ExaNIC operate as a mini switch. If you choose to enable port bridging, physical ports 1 and 2 are connected together in hardware in under 170ns. This means that any packets that arrive on port 1 which aren't destined for the host will be forwarded out of port 2 (and likewise for packets arriving on port 2). Additionally, the card supports rate matching in hardware (translating between 1GbE and 10GbE when the bridged ports have different speeds).

The original use case for this feature is to allow customers involved in high-frequency trading to put their most latency critical server as close to an exchange as possible, with any downstream servers or switches connected behind the ExaNIC. The ability to connect an optional backup power supply to the NIC means that even if this server goes down, network connectivity to downstream devices is maintained. Additionally, there's zero latency penalty incurred on packets to and from the host.

Diagram showing how bridging can be used to connect the most latency critical
server directly to the exchange

There are other applications for bridging though. For example, in the case of traffic monitoring, you could use the NIC to pass all network traffic traversing a particular network segment to the host. This could then be timestamped, logged or monitored in real time.

Port mirroring

Port mirroring allows you to replicate any traffic received or transmitted on any of the first 3 ports of the four port card, out of the 4th port. This is all done in hardware, so it doesn't incur any overhead on the host processor, and mirroring of both RX and TX data on any of the first 3 ports is individually selectable. The intended use case for this feature is for logging, particularly in trading applications. Importantly, activating logging for a particular port does not introduce any latency overhead.

Diagram showing how mirroing can be used to connect to a logging server

That's all for today, but in a future post I will dive into a bit more detail on some of these and other features of the ExaNIC. As always, if you're interested in any of these features, get in touch.