Configuration

Since the Cisco Nexus SmartNIC (formerly ExaNIC) appears to Linux as a normal network card, most configuration can be performed through the normal Linux mechanisms. For example, in Redhat-based distributions, configuration is generally performed using files in the /etc/sysconfig/network-scripts directory. Alternatively, the interface can be configured temporarily by issuing a command such as ifconfig eth7 192.168.1.100.

If the interface is being used through the low-level userspace API (libexanic) only, it is sufficient to bring the interface up without assigning an IP address. This can be done through either ifconfig (e.g. ifconfig eth7 up) or exanic-config (e.g. exanic-config exanic0:0 up).

The exanic-config utility

The exanic-config utility can be used to inspect diagnostic information about the card and SFP modules. For example:

$ exanic-config
Device exanic0:
 Hardware type: ExaNIC X4
 Board ID: 0x02
 Temperature: 53.2 C VCCint: 1.00 V VCCaux: 1.81 V
 Fan speed: 6987 RPM
 Function: network interface
 Firmware date: Thu Nov 28 20:54:56 2013
 Bridging: off
 Port 0:
  Interface: eth7
  Port speed: 10000 Mbps
  Port status: enabled, SFP present, signal detected, link active
  Mirroring: off
  Promiscuous mode: off
  Bypass-only mode: off
  MAC address: 64:3f:5f:01:13:18
  RX packets: 6 ignored: 0 error: 0
  TX packets: 6
...

$ exanic-config exanic0:0 sfp status
Device exanic0 port 0 SFP status:
 Vendor: FINISAR CORP. PN: FTLX8571D3BCL rev: A
                       SN: ALF0QE7 date: 111006
 Wavelength: 850 nm
 Nominal bit rate: 10300 Mbps
 Rx power: -2.5 dBm (0.56 mW)
 Tx power: -1.9 dBm (0.65 mW)
 Temperature: 33.8 C

The exanic-config utility will either accept the Linux interface name (e.g. eth7) or the device name of the SmartNIC device (e.g. exanic0 for the first SmartNIC in the system by PCI ID, and exanic0:0 for the first port of that card). Globbing syntax is also accepted in order to match several cards/ports. For example, exanic*:[0-3] will match ports 0-3 inclusive on any SmartNICs installed. See the output of exanic-config --help for more details.

Interface packet counters

Packet counters in exanic-config

The exanic-config utility will display the packet counter values for traffic being transmitted or received at a particular port.

Port 0:
 Interface: eth7
 Port speed: 10000 Mbps
 Port status: enabled, SFP present, signal detected, link active
 Mirroring: off
 Promiscuous mode: off
 Bypass-only mode: off
 MAC address: 64:3f:5f:01:13:18
 RX packets: 1514936895 ignored: 20578657 error: 145683 dropped: 0
 TX packets: 615149369

RX Packets: refers to the total number of packets received by the NIC hardware.

ignored: refers to packets that did not match the local MAC address. It is incremented when the NIC receives a packet that is unicast but destined to a different destination address. This is not as a result of load or any problem with the NIC/host, but that the device on the other end of the wire is sending packets not destined for this device.

error: refers to packets which failed CRC checking. A common cause of this type of behaviour is poor signal from the other end, e.g. a bad cable or optical splitter. To investigate run exanic-config exanicX:Y sfp status at both ends of the link.

dropped: refers to packets lost as a result of insufficient PCIe bandwidth to transfer all packets into host memory. If this occurs, check lspci -vvv and in particular the LnkSta: line for the SmartNIC to verify that the card is operating at Speed 8GT/s (PCIe Gen 3.0) and Width x8. Dropped packets can also occur due to system topology - for example in a multi-socket system where the card is being accessed from a remote socket and QPI bottlenecks occur - or if the incoming network bandwidth exceeds that available on the PCI Express link (e.g. 8x10G or 2x40G can exceed the bandwidth available on a 8x8Gbit/s link during line rate bursts).

Packet counters in ifconfig

dropped in ifconfig counters can be a result of:

• insufficient CPU cycles to service the NIC
• if jumbo frames are coming in (see note 1 below)
• if IP6 frames are coming in when expecting IP4 only.

Setting port speed

The default speed for SmartNIC ports is 10 Gigabit. To set a port to a different speed, use the speed command to either ethtool or exanic-config:

$ ethtool -s eth7 speed 1000

or:

$ exanic-config exanic0:0 speed 1000

Supported values are 10000, 1000 and 100.

Promiscuous mode

Normally the SmartNIC only passes on received unicast traffic that matches the MAC address of the port to the CPU. If Promiscuous mode is enabled on a port then the network interface controller will instead pass all unicast traffic it receives on that port to the CPU.

$ exanic-config exanic0:0 promisc on

Bypass-only mode

If the interface is being used through the userspace API only and kernel CPU load on high traffic connections is a concern, the bypass-only setting in exanic-config (or bypass_only in ethtool) can be used to detach the kernel driver from the interface. This means that the kernel driver will not receive any packets. However, tools such as tcpdump will not function correctly while this is enabled. Additionally, this setting is not currently persistent across reboots so, if required, should be added to post-up scripts for the interface.

$ exanic-config exanic0:0 bypass-only on

Local-loopback mode

When an interface is configured to local-loopback mode the SmartNIC will transmit frames through the FPGA to the wire and will also loop the frame back to the host.

$ exanic-config exanic0:0 local-loopback on
exanic0:0: local-loopback mode enabled

This then allows the transmitted traffic to be confirmed by observing the received traffic with tools such as exanic-capture and tcpdump. Once validated remember to turn the local-loopback function back off.

$ exanic-config exanic0:0 local-loopback off
exanic0:0: local-loopback mode disabled

Pulse-per-second

SmartNICs with a SMA connector (Cisco Nexus SmartNIC K35-S (formerly X10) and onwards) can be configured to drive a PPS out or accept a PPS input - see SmartNIC clock synchronization for full details. SmartNIC clock synchronization

Packet Capture

Using tcpdump

tcpdump can be used to capture SmartNIC traffic that the kernel sees. This would normally be all received traffic (unless steered away using flow steering) and any traffic transmitted via the kernel. Note that traffic transmitted via utilities that use libexanic or exasock bypass the kernel, hence will not be shown in tcpdump. The kernel may also not keep up with received traffic in high load scenarios.

Using exanic-capture

A utility called exanic-capture is provided that can be used to capture traffic received by an SmartNIC.

This utility receives Ethernet frames from defined interface and dumps to terminal or defined file.

$ exanic-capture
Usage: ./exanic-capture -i interface [-w savefile] [-s snaplen] [-p] [-H] [-N] [filter...]
  -i: specify Linux interface (e.g. eth0) or ExaNIC port name (e.g. exanic0:0)
  -w: dump frames to given file in pcap format (- for stdout)
  -s: maximum data length to capture
  -p: do not attempt to put interface in promiscuous mode
  -H: use hardware timestamps (refer to documentation on how to sync clock)
  -N: write nanosecond-resolution pcap format

Filter examples:
  tcp port 80                   (to/from tcp port 80)
  host 192.168.0.1 tcp port 80  (to/from 192.168.0.1:80)
  dst 192.168.0.1 dport 53      (to 192.168.0.1:53, either tcp or udp)
  src 192.168.0.5 sport 80 or dst 192.168.0.1 (combine clauses with 'or')

exact-capture

There is a utility called exact-capture which is designed for high rate capture across multiple SmartNIC interfaces. This utility is open source and available on Exablaze's github account.

This is the recommended utility for capturing traffic in high load environments. Please refer to this page for documentation and further details on exact-capture.

Multicast Traffic

Confirming registration

Please note this guide assumes the use of IGMP v3, to check the configured version of IGMP on the host operating system (OS) run:

cat /proc/net/igmp

exanic-config should be used to confirm that the SmartNIC card is present and that the interface to be used is appropriately configured. E.g. enabled and link active. The rest of the guide assumes a switch with IP 10.230.91.1

$ exanic-config

This should port output similar to.

Port 1:
   Interface: enp1s0d1
   Port speed: 10000 Mbps
   Port status: enabled, SFP present, signal detected, link active
   ...
   IP address: 10.230.91.62  Mask: 255.0.0.0

Confirm the SmartNIC is able to register for group membership successfully at the host. To assist, Cisco provides with the driver source an example multicast registration utility. Browse to the src/examples/exanic/ folder and build the mcast_register utility for registering to a multicast group.

$ make

Then without exasock, run the built utility to register for the multicast group on the SmartNIC host. Providing the interface to utilise and the multicast address to register to. e.g.

  $ ./mcast_register 226.20.20.21:enp1s0d1

Next confirm the SmartNIC interface and the multicast address show in the multicast group memberships on the SmartNIC host OS. e.g.

$ netstat -g

This should return a table similar to.

IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              1      all-systems.mcast.net
enp1s0          1      all-systems.mcast.net
enp1s0d1        1      226.20.20.21

Then confirm the SmartNIC interface and the switch are up and have link and routes. If possible ping between the IP addresses on the subnet, note this may not be possible with Fusion objects.

Confirm that multicast registration requests are being sent from the SmartNIC interface e.g.

$ tcpdump -i enp1s0 igmp

09:14:57.915283 IP 10.230.91.62 > 224.0.0.22: igmp v3 report, 1 group
record(s)
09:14:58.908281 IP 10.230.91.62 > 224.0.0.22: igmp v3 report, 1 group
record(s)

10:09:25.890209 IP 10.230.91.1 > 224.0.0.1: igmp query v2
10:11:24.291555 IP 10.230.91.1 > 224.0.0.1: igmp query v2

Next please confirm the SmartNIC shows in the multicast group memberships on the switch. For Cisco Nexus 3550-F Fusion switch devices this is the show igmp groups command. For further details on the Fusion IGMP groups see : https://fusion.exablaze.com/detailed/multicast/#igmp-groups

Then confirm that once registered multicast traffic queries etc are observed at the SmartNIC interface e.g. $ tcpdump -i enp1s0

10:33:57.351212 IP 10.230.91.1 > 226.20.20.21: igmp query v3
10:33:58.352121 IP 10.230.91.1 > 226.20.20.21: igmp query v3

This page was last updated on Dec-10-2020.