Exablaze logo

ExaNIC Device Installation

Some motherboards have certain PCI Express slots connected directly to the processor and others connected to the chipset; for optimal performance, install the ExaNIC in a slot directly connected to the processor. If installing in a dual-processor server with only one processor installed, take special care to install in the correct slot(s), otherwise the NIC may not show up at all.

Install SFP+ modules into the ExaNIC as required. There are no limitations on the type or manufacturer of SFP+ modules that can be used.

Software Installation

There are a number of different ways of installing the ExaNIC software: using the pre-built packages provided by Exablaze, building your own packages, or installing from source.

A mailing list is available to be notified when updates to the ExaNIC software are released. Please feel free to add yourself here

Pre-built packages

Pre-built packages are available for some Linux distributions. The easiest way to access these is to add the Exablaze repository to your yum or apt sources, or you can manually download the desired packages directly from the repository.

The yum repository containing .rpm packages for Redhat/CentOS is located at:

https://exablaze.com/downloads/yum/

The apt repository containing .deb packages for Debian/Ubuntu is located at:

https://exablaze.com/downloads/apt/

In both cases there is a README.txt file which explains how to add the repository, what distributions are currently supported and what packages are available.

Build your own packages

You can also build your own packages using the packaging provided by Exablaze.

To build your own .rpm packages, download the latest source rpm from https://exablaze.com/downloads/yum/redhat/el6/SRPMS/ and run rpmbuild --rebuild exanic-*.src.rpm . Packages will be output to ~/rpmbuild/RPMS.

To build your own .deb packages, download and unpack the source from the support website as per the first step below, then run dpkg-buildpackage from within the source directory. Packages will be output to the parent directory.

Install from source

If binary packages are not available for your distribution, the ExaNIC software is also easy to build and install from source. To start, download the ExaNIC source package from the support website and unpack it.

$ tar xzf exanic-1.7.0.tar.gz
$ cd exanic-1.7.0

Running the following commands should build and install the driver, library and utilities:

$ make
$ sudo make install

By default, the library will be installed to /usr/local/lib/libexanic.a, include files to /usr/local/include/exanic and utilities to /usr/local/bin/exanic-*. (You can add a PREFIX= argument to the make install command to use a prefix other than /usr/local.) Additionally the driver (exanic.ko) will be installed into /lib/modules/'uname -r'/extra.

If the driver fails to build, ensure that you have the kernel headers package for your kernel installed (this is typically named kernel-devel [RedHat/CentOS] or linux-headers [Debian/Ubuntu]).

Driver Installation Verification

Assuming installation completed successfully, load the driver and verify that the exanic-config utility works:

$ sudo modprobe exanic
$ sudo exanic-config exanic0

If either of these commands fail, run dmesg and check for errors.

The exanic-config output will show the Linux interfaces corresponding to each port of the ExaNIC. You can verify that packets are being received by bringing up an interface and running tcpdump. For example:

$ sudo ifconfig eth7 up
$ sudo tcpdump -n -i eth7

Information Note 1: Some systems may present the following error in the dmesg output:

modprobe: ERROR: could not insert 'exanic': Required key not available

To resolve this issue, change the BIOS from Windows to "Other OS". This should alter the "Secure boot" setup of the host and the module can now be loaded.

Setting permissions

If the ExaNIC is to be used in kernel bypass mode, it may be useful to set permissions on the ExaNIC device files (/dev/exanic0, etc, and the /dev/exasock device file used by ExaNIC Sockets) so that they can be accessed by certain non-root applications. Permissions can be set with chown/chgrp/chmod as normal, however this will not persist across reboots. A persistent configuration can be achieved by appropriate configuration of the udev daemon. For example, to force the device files to be accessible by a group called exanic, create a /etc/udev/rules.d/exanic.rules file as follows:

KERNEL=="exanic*", GROUP="exanic", MODE="0660"
KERNEL=="exasock", GROUP="exanic", MODE="0660"

In the current software release the ExaNIC device files also expose the device control registers so only trusted users should be given access to the ExaNIC in this way.

Updating firmware

The firmware version currently running on the ExaNIC card is shown under 'Firmware date' in exanic-config. If there is a newer version available on the website, you can update your firmware as follows. First, uncompress the downloaded firmware image:

$ gunzip exanic_x4_20131128.fw.gz

Now run exanic-fwupdate:

$ exanic-fwupdate exanic_x4_20131128.fw

Note that there are different firmware files for each type of ExaNIC. The utility will prevent you from installing firmware incompatible with your hardware.

Warning The exanic-fwupdate program takes the .fw file rather than a .bit, .zip or .tar.gz file. If the wrong type of file is provided to the program you may see the message "ERROR: Invalid start character".

Firmware Recovery

The ExaNIC stores two firmware images onboard. The first is the main "production" image that will be booted normally. The second image is the "recovery" image that can be booted if the main image is corrupted.

The recovery image can be booted into by holding down the recovery button on power up of the NIC. The red LED will illuminate to indicate that the recovery program is being initiated. Recovery mode is only intended to allow a new firmware image to be loaded onto the device and should not be used for standard operation.

Example X10 recovery button

ExaNIC recovery button

Uninstallation

If you installed from packages then you should use the appropriate yum or apt command (sudo yum remove 'exanic' or sudo apt-get remove 'exanic').

If you installed from source the following command should remove everything that was installed by make install:

$ sudo make uninstall

Also remove any files that were manually created while following these instructions (/etc/udev/rules.d/exanic.rules and any scripts that run exanic-clock-sync).

Troubleshooting

The ExaNIC card should follow the following transitions during power on and boot up:

  • When first powered all LEDs should flash on momentarily.
  • With no SFPs/QSFP modules inserted, all LEDs should then switch off. If the LEDs remain on with no modules inserted, and the card does not appear in lspci, attempt the firmware recovery process.
  • Upon SFP/QSFP module insertion, the port LEDs should turn orange. If the LEDs do not light check that the connector is fully inserted, or try a different module.
  • Once enabled from software, the port LEDs should turn green if link is detected. If LEDs remain orange, check that the port is enabled and has correct settings using the exanic-config utility.
  • If the red LED near the PPS connector is lit, or the port LEDs flash with no modules inserted, this indicates that the firmware was corrupt and the card has entered recovery mode. A new firmware must be loaded with the exanic-fwupdate utility.
  • If no LEDs light at any time, the card or slot may be damaged.

Warning When using a server with only one of two possible processors installed, make sure the PCIe expansion slot used is one connected to the installed CPU, otherwise the NIC will not appear in lspci. This is a common installation error in dual-processor servers such as Dell R630 and R730.