ExaNIC flow steering and load balancing

This week I wanted to go over another useful feature of the ExaNIC - hardware based flow steering and load balancing (also known as flow hashing). The ExaNIC can steer received traffic to a number of userspace buffers based on rules over IP or MAC address. It can also be configured to calculate a hash that is a function of the IP headers and use this to balance traffic across the host's CPUs.

Selectively steering traffic

By default, all traffic for the ExaNIC arrives in a single ring buffer per port on the card that can be seen by the kernel as well as userspace applications. Our network device driver uses this buffer and integrates in with the rest of the Linux network stack. This allows the ExaNIC to be used like a regular network card. We also provide an API for mapping this buffer to userspace so that user applications can have access to the raw ethernet frames coming off the wire.

Under normal conditions, with flow steering disabled, this buffer will receive all traffic coming off the wire. This means that the kernel driver and user applications may be seing a whole bunch of traffic that isn't relevant. To reduce the load on the kernel and application, we can filter traffic using the ExaNICs onboard flow steering capability. The FPGA onboard the ExaNIC then makes decisions about where to transfer each incoming frame based on user defined rules. A big plus is that there is no latency overhead for enabling any number of flow steering rules.

This functionality can all be configured using our API, and amounts to the following:

  • Obtain a free userspace RX buffer for your application.
  • Define rules and associate them with this buffer.
  • Monitor this buffer for any inbound frames.

The ExaNIC can steer traffic based on IP headers:

Source IP, Destination IP, Source Port, Destination Port, Protocol (e.g. UDP, TCP)

Or L2 ethernet headers:

Destination MAC, Ethertype, VLAN tag

Diagram showing how flow steering can direct traffic to different userspace 

Rules can be very specific - requiring an exact match over all fields - or very broad, with a wildcard for any of the fields. For some examples:

  • All traffic belonging to a TCP connection could be delivered to a buffer by specifying all of the IP fields,
  • Multicast UDP traffic to a specific multicast address, but with any source address,
  • All UDP traffic from and to any address or port combination.

Load balancing

So that's flow steering, now to load balancing. In some cases it is desirable for the NIC to balance load across multiple CPUs on the host. The ExaNIC is capable of doing this using its flow hashing functionality. When placed in flow hashing mode, the ExaNIC calculates a hash over the IP headers of incoming frames. This hash is calculated such that packets belonging to each IP flow will always end up in the same buffer. One example of how this could be used is in for monitoring the connections passing between network segments.

To do this, the ExaNICs first two ports can be bridged together on the card. The two network segments can then be joined using these ports on the ExaNIC. The host can then configure both ports in promiscuous mode and enable flow hashing on the ExaNIC. The host then monitors these buffers and analyses the IP traffic flows passing over the network segment.

Diagram showing ExaNIC with hardware bridge of first two ports, and 
flow hashing used to monitor connections passing through card.

Anyway, that's enough for today.. As always, if you're interested in any of this stuff get in touch.