Since the Cisco Nexus SmartNIC (formerly ExaNIC) appears to Linux as a normal network card, most
configuration can be performed through the normal Linux mechanisms. For
example, in Redhat-based distributions, configuration is generally
performed using files in the
Alternatively, the interface can be configured temporarily by issuing a
command such as
ifconfig eth7 192.168.1.100.
If the interface is being used through the low-level userspace API
(libexanic) only, it is sufficient to bring the interface up without
assigning an IP address. This can be done through either
ifconfig eth7 up) or
exanic-config exanic0:0 up).
The exanic-config utility
exanic-config utility can be used to inspect diagnostic information
about the card and SFP modules. For example:
$ exanic-config Device exanic0: Hardware type: ExaNIC X4 Board ID: 0x02 Temperature: 53.2 C VCCint: 1.00 V VCCaux: 1.81 V Fan speed: 6987 RPM Function: network interface Firmware date: Thu Nov 28 20:54:56 2013 Bridging: off Port 0: Interface: eth7 Port speed: 10000 Mbps Port status: enabled, SFP present, signal detected, link active Mirroring: off Promiscuous mode: off Bypass-only mode: off MAC address: 64:3f:5f:01:13:18 RX packets: 6 ignored: 0 error: 0 TX packets: 6 ...
$ exanic-config exanic0:0 sfp status Device exanic0 port 0 SFP status: Vendor: FINISAR CORP. PN: FTLX8571D3BCL rev: A SN: ALF0QE7 date: 111006 Wavelength: 850 nm Nominal bit rate: 10300 Mbps Rx power: -2.5 dBm (0.56 mW) Tx power: -1.9 dBm (0.65 mW) Temperature: 33.8 C
exanic-config utility will either accept the Linux interface name
eth7) or the device name of the SmartNIC device (e.g.
for the first SmartNIC in the system by PCI ID, and
exanic0:0 for the
first port of that card). Globbing syntax is also accepted in order to
match several cards/ports. For example,
exanic*:[0-3] will match
ports 0-3 inclusive on any SmartNICs installed. See the output of
exanic-config --help for more details.
Interface packet counters
Packet counters in exanic-config
exanic-config utility will display the packet counter values for traffic being transmitted or received at a particular port.
Port 0: Interface: eth7 Port speed: 10000 Mbps Port status: enabled, SFP present, signal detected, link active Mirroring: off Promiscuous mode: off Bypass-only mode: off MAC address: 64:3f:5f:01:13:18 RX packets: 1514936895 ignored: 20578657 error: 145683 dropped: 0 TX packets: 615149369
RX Packets: refers to the total number of packets received by the NIC hardware.
ignored: refers to packets that did not match the local MAC address. It is incremented when the NIC receives a packet that is unicast but destined to a different destination address. This is not as a result of load or any problem with the NIC/host, but that the device on the other end of the wire is sending packets not destined for this device.
error: refers to packets which failed CRC checking. A common cause of this type of behaviour is poor signal from the other end, e.g. a bad cable or optical splitter. To investigate run
exanic-config exanicX:Y sfp status at both ends of the link.
dropped: refers to packets lost as a result of insufficient PCIe bandwidth to transfer all packets into host memory. If this occurs, check
lspci -vvv and in particular the
LnkSta: line for the SmartNIC to verify that the card is operating at Speed 8GT/s (PCIe Gen 3.0) and Width x8. Dropped packets can also occur due to system topology - for example in a multi-socket system where the card is being accessed from a remote socket and QPI bottlenecks occur - or if the incoming network bandwidth exceeds that available on the PCI Express link (e.g. 8x10G or 2x40G can exceed the bandwidth available on a 8x8Gbit/s link during line rate bursts).
Packet counters in ifconfig
dropped in ifconfig counters can be a result of:
- insufficient CPU cycles to service the NIC
- if jumbo frames are coming in (see note 1 below)
- if IP6 frames are coming in when expecting IP4 only.
Although the kernel driver and exasock don't support jumbo frames, the raw libexanic API can actually receive jumbo frames with no problems. Transmitting jumbo frames using the raw libexanic API is also possible, however you will need to allocate a larger TX buffer than the default size. For sending 4k frames you will need a TX buffer size of 16k.
Setting port speed
The default speed for SmartNIC ports is 10 Gigabit. To set a port to a different speed, use the speed command to either
$ ethtool -s eth7 speed 1000
$ exanic-config exanic0:0 speed 1000
Supported values are 10000, 1000 and 100.
1 Gigabit Ethernet support requires firmware dated 20140206 or later.
100 Mbit Ethernet support requires firmware version 20150327 or later, and one of the following SFP modules: Finisar FCLF-8520-3/FCLF-8521-3/FCMJ-8520-3/FCMJ-8521-3 or Avago ABCU57xxxxxZ.
Some 100 Mbit SFP modules will not configure the speed unless the port has first been enabled (e.g. exanic-config exanic0:0 up). Ensure the module is inserted into the port at the time the port is configured for 100M.
Normally the SmartNIC only passes on received unicast traffic that matches the MAC address of the port to the CPU. If Promiscuous mode is enabled on a port then the network interface controller will instead pass all unicast traffic it receives on that port to the CPU.
$ exanic-config exanic0:0 promisc on
If the interface is being used through the userspace API only and kernel
CPU load on high traffic connections is a concern, the
ethtool) can be used
to detach the kernel driver from the interface. This means that the
kernel driver will not receive any packets. However, tools such as
tcpdump will not function correctly while this is enabled.
Additionally, this setting is not currently persistent across reboots
so, if required, should be added to post-up scripts for the interface.
bypass-only mode should not be enabled for exasock to function correctly.
$ exanic-config exanic0:0 bypass-only on
When an interface is configured to local-loopback mode the SmartNIC will transmit frames through the FPGA to the wire and will also loop the frame back to the host.
When configured to local-loopback mode traffic is disabled from coming in from the wire. That is you will only receive the traffic you transmit on that interface.
$ exanic-config exanic0:0 local-loopback on exanic0:0: local-loopback mode enabled
This then allows the transmitted traffic to be confirmed by observing the received traffic with tools such as
tcpdump. Once validated remember to turn the local-loopback function back off.
$ exanic-config exanic0:0 local-loopback off exanic0:0: local-loopback mode disabled
SmartNICs with a SMA connector (Cisco Nexus SmartNIC K35-S (formerly X10) and onwards) can be configured to drive a PPS out or accept a PPS input - see SmartNIC clock synchronization for full details. SmartNIC clock synchronization
tcpdump can be used to capture SmartNIC traffic that the kernel sees. This would normally be all received traffic
(unless steered away using flow steering) and any traffic transmitted via the kernel.
Note that traffic transmitted via utilities that use libexanic or exasock bypass the kernel, hence will not be shown in tcpdump.
The kernel may also not keep up with received traffic in high load scenarios.
A utility called
exanic-capture is provided that can be used to capture traffic received by an SmartNIC.
This utility receives Ethernet frames from defined interface and dumps to terminal or defined file.
$ exanic-capture Usage: ./exanic-capture -i interface [-w savefile] [-s snaplen] [-p] [-H] [-N] [filter...] -i: specify Linux interface (e.g. eth0) or ExaNIC port name (e.g. exanic0:0) -w: dump frames to given file in pcap format (- for stdout) -s: maximum data length to capture -p: do not attempt to put interface in promiscuous mode -H: use hardware timestamps (refer to documentation on how to sync clock) -N: write nanosecond-resolution pcap format Filter examples: tcp port 80 (to/from tcp port 80) host 192.168.0.1 tcp port 80 (to/from 192.168.0.1:80) dst 192.168.0.1 dport 53 (to 192.168.0.1:53, either tcp or udp) src 192.168.0.5 sport 80 or dst 192.168.0.1 (combine clauses with 'or')
There is a utility called
exact-capture which is designed for high rate capture across multiple SmartNIC interfaces. This utility is open source and available on Exablaze's github account.
This is the recommended utility for capturing traffic in high load environments. Please refer to this page for documentation and further details on
Please note this guide assumes the use of IGMP v3, to check the configured version of IGMP on the host operating system (OS) run:
It is a common source of error that the host OS and the multicast switch operate with different IGMP versions. While the IGMP versions are backwards compatible, please ensure that the implementations of the versions are suitable.
exanic-config should be used to confirm that the SmartNIC card is present and that the interface to be used is appropriately configured. E.g. enabled and link active. The rest of the guide assumes a switch with IP
This should port output similar to.
Port 1: Interface: enp1s0d1 Port speed: 10000 Mbps Port status: enabled, SFP present, signal detected, link active ... IP address: 10.230.91.62 Mask: 255.0.0.0
Confirm the SmartNIC is able to register for group membership successfully at the host. To assist, Cisco provides with the driver source an example multicast registration utility. Browse to the
src/examples/exanic/ folder and build the mcast_register utility for registering to a multicast group.
Then without exasock, run the built utility to register for the multicast group on the SmartNIC host. Providing the interface to utilise and the multicast address to register to. e.g.
$ ./mcast_register 22.214.171.124:enp1s0d1
Next confirm the SmartNIC interface and the multicast address show in the multicast group memberships on the SmartNIC host OS. e.g.
$ netstat -g
This should return a table similar to.
IPv6/IPv4 Group Memberships Interface RefCnt Group --------------- ------ --------------------- lo 1 all-systems.mcast.net enp1s0 1 all-systems.mcast.net enp1s0d1 1 126.96.36.199
Then confirm the SmartNIC interface and the switch are up and have link and routes. If possible ping between the IP addresses on the subnet, note this may not be possible with Fusion objects.
Confirm that multicast registration requests are being sent from the SmartNIC interface e.g.
$ tcpdump -i enp1s0 igmp
IGMP Version 3 membership reports are destined to the address 188.8.131.52. All IGMPv3-capable multicast routers must listen to this address.
09:14:57.915283 IP 10.230.91.62 > 184.108.40.206: igmp v3 report, 1 group record(s) 09:14:58.908281 IP 10.230.91.62 > 220.127.116.11: igmp v3 report, 1 group record(s)
IGMP Version 1 and 2 use a query-response model. Queries are sent to 18.104.22.168.
10:09:25.890209 IP 10.230.91.1 > 22.214.171.124: igmp query v2 10:11:24.291555 IP 10.230.91.1 > 126.96.36.199: igmp query v2
Next please confirm the SmartNIC shows in the multicast group memberships on the switch. For Cisco Nexus 3550-F Fusion switch devices this is the
show igmp groups command.
For further details on the Fusion IGMP groups see :
Then confirm that once registered multicast traffic queries etc are observed at the SmartNIC interface e.g. $ tcpdump -i enp1s0
10:33:57.351212 IP 10.230.91.1 > 188.8.131.52: igmp query v3 10:33:58.352121 IP 10.230.91.1 > 184.108.40.206: igmp query v3
This page was last updated on Dec-10-2020.