Hardware based timestamping
Every packet that traverses the card (both received and transmitted) is timestamped to nanosecond level resolution (0.25ns for Cisco Nexus NIC HPT (formerly ExaNIC HPT), 6.2 nanoseconds for all others). This timestamping is done in hardware. To obtain a useful, timestamp, the hardware clock must be synchronized. Synchronizing is possible between the hardware internal timer and the host's clock, a separate pulse-per-second (PPS), or networked PTP grand master.
Synchronizing Cisco Nexus SmartNIC cards in the same host
To synchronize Cisco Nexus SmartNIC (formerly ExaNIC) cards to a host or primary card, the
exanic-clock-sync daemon can be used to keep the SmartNIC card's clock synchronized. Each argument to
exanic-clock-sync specifies a
<device>:<synchronization-source> pair, with the synchronization source being one of host (host clock) or exanicN (another SmartNIC device). For example:
$ exanic-clock-sync exanic0:host
$ exanic-clock-sync exanic0:host exanic1:exanic0
If desired this can be placed in system startup scripts. A sample
init.d script is provided in
exanic-clock-check is provided to check how tightly synchronized the host and NIC clocks are:
$ exanic-clock-check Compares the ExaNIC clock to the host or another ExaNIC Usage: ./exanic-clock-check <exanic> [<exanic>]
should provide output similar too
$ exanic-clock-check exanic0 Device exanic0: 1628741401 ticks (1498447571360653132 ns since epoch) Host clock: 1498447571360653 us since epoch Difference: 132 ns
Synchronizing Cisco Nexus SmartNIC cards in different hosts with Pulse per second signals (PPS)
Utilizing the PPS input
To synchronize SmartNIC cards to this PPS signal the exanic-clock-sync daemon can be used. Again each argument to exanic-clock-sync specifies a
<device>:<synchronization-source> pair, with PPS as the synchronization source in this example:
$ exanic-clock-sync exanic0:pps
will result in output similar to:
exanic0: Starting clock discipline using PPS exanic0: Nominal frequency: 161132800 Hz exanic0: Using differential PPS input exanic0: PPS signal detected exanic0: Clock offset at PPS pulse: -9305478.518 us drift: 0.000 ppm exanic0: Clock offset at PPS pulse: -17.253 us drift: 0.000 ppm ... exanic0: Clock offset at PPS pulse: -0.019 us drift: -16.794 ppm
By default, the SMA input on SmartNIC cards accepts 3.3V TTL and has a standard 50 ohm termination enabled. To turn termination off, specify
pps-no-term instead of
pps. This allows for daisy chaining the PPS inputs of several SmartNIC cards together, where the last card in the chain terminates the PPS signal. The intermediate cards should be configured with
pps-no-term and the final card with
$ exanic-clock-sync exanic0:pps-no-term
$ exanic-clock-sync exanic0:pps
Configuring the Pulse-per-second (PPS) output
SmartNICs with a SMA connector (Cisco Nexus SmartNIC K35-S (formerly X10) and onwards) can be configured to drive a PPS out. The simplest starting point is to install a single SMA coaxial lead from the first SmartNIC device to a second SmartNIC device. With the cable present run:
$ exanic-config exanic0 pps-out on
exanic-config will then show the new state of the PPS output.
Device exanic0: Hardware type: ExaNIC X10 ... PPS out: enabled
The PPS output is via a female SMA connector, capable of driving 3.3V TTL into a 50 ohm load.
SmartNIC support for PPS out requires firmware from about Aug 2016 or later - please refer to the firmware changelog for the SmartNIC for details.
Synchronizing Cisco Nexus SmartNIC cards in different hosts with the Precision Time Protocol (PTP)
The Precision Time Protocol (PTP) is a protocol that allows for very accurate time synchronization between a master and a number of client (slave) systems. The SmartNIC supports the Linux
SO_TIMESTAMPING API, so standard PTP daemons such as ptp4l (LinuxPTP) or ptpd can be used to synchronize the card's clock.
The underlying time standard used by PTP is the International Atomic Time (TAI) standard, which at the time of writing is 37 seconds ahead of the Universal Coordinated Time (UTC) standard. LinuxPTP will set and keep the SmartNIC hardware clock in TAI time, not UTC time. Applications that make use of SmartNIC timestamps will need to be aware of this. Recent kernels are aware of the UTC-TAI offset, and this can be queried using
The host operating system requirements for PTP HW and PHC support are:
- linux kernel >= 3.17, or
- centos 7 with stock 3.10 kernel, or
- centos 6 with stock 2.6.32 kernel and nohz=off
It is not recommended or required to run the PTP HW synchronization and PPS exanic-clock-sync at the same time.
LinuxPTP (PTP4l) works out of the box, however does not support unicast PTP traffic. We have had good results with ptp4l and there is a good deployment guide at the link below:
As of December 2016, the current release of ptpd (2.3.1) does not contain the required hardware timestamping code, so you need to either use the hardware timestamping branch described at http://nwtime.org/new-features-for-ptpd/ or to additionally run exanic-clock-sync to synchronize the time from the host.
As an example, a PTP slave ptp4l can then be run with a command such as the following:
$ ptp4l -i eth4 -m
Configuration File Method
In all the above examples, exanic-clock-sync's parameters and configuration options are set through command line options. These can alternatively be stored in a configuration file for ease of maintenance.
To specify the config file /etc/exanic_clksync.conf, run the following command:
$ exanic-clock-sync --config /etc/exanic_clksync.conf
The configuration file is broken into sections, each section containing key-value pairs for a synchronization target. The following subsections detail the format of the configuration file.
This option takes a string as input and is equivalent to specifying the synchronization source on the command line.
pps_term or pps_termination
This option takes a boolean input. When the key is present, the termination resistor is connected unless the value is "falsey", that is, one of
0 or an empty string. This is equivalent to specifying the source as
pps_no_term or some other PPS option on the command line. The value for this key is case-insensitive.
This option takes a string as input and the value is one of the following:
positive causes the synchronization target to accept rising edge PPS signal, and
negative, falling edge. All of the possible values above are case insensitive.
This option takes an integer input and is equivalent to appending a
- followed by an integer offset after the source on the command line, e.g.
$ exanic-clock-sync exanic0:pps+5
This option specifies the offset, in nanoseconds, to add or subtract from the synchronization source. For example, to compensate for a delay of 5 nanoseconds between the synchronization source to your device, configure PPS with -5.
This option applies to PPS only and specifies the averaging interval for calculating clock jitter and adjustment of the synchronization target. The default is 4 seconds.
Sample Configuration File
Here is a sample configuration file that demonstrates how to set up exanic0 for rising edge PPS time synchronization, with the termination resistor enabled and the averaging interval for jitter calculation set to 5 seconds:
[exanic0] source=pps pps_edge=rising pps_term=yes interval=5
Below are some suggestions for troubleshooting PTP configuration for a PTP client. For troubleshooting suggestions for an Cisco Nexus NIC GM (formerly ExaNIC GM) please look at NIC GM troubleshooting.
1. To confirm the timestamping capabilities are present on a slave SmartNIC run the following on your interface: (requires ethtool 3.4 or above)
$ ethtool -T eth1 Time stamping parameters for eth1: Capabilities: hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE) software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE) hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE) software-receive (SOF_TIMESTAMPING_RX_SOFTWARE) software-system-clock (SOF_TIMESTAMPING_SOFTWARE) hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE)
2. Make sure the SmartNIC port has link up and is configured to the correct speed e.g.
Port 0: Interface: eth1 Port speed: 10000 Mbps Port status: enabled, SFP present, link active
to change the port state or speed use the exanic-config commands.
$ sudo exanic-config exanic0:0 up $ sudo exanic-config exanic0:0 speed 10000
2. Make sure the SmartNIC has been assigned an appropriate IP address and that the master and slave are using the same transport protocol, i.e. multicast or unicast - and be aware of issues arising from hybrid master operation.
3. Generally debugging is simpler with multicast - at which point you can look to see that the SmartNIC port is receiving traffic.
$ sudo exanic-config exanic0:0
Which should show the received packet counters incrementing
RX packets: 708 ignored: 0 error: 0 dropped: 0
4. Then make sure kernel interface is receiving traffic.
$ sudo ifconfig eth1
should also show received packets incrementing
RX packets:587 errors:0 dropped:0 overruns:0 frame:0
$ sudo tcpdump -i eth1 port 319 or port 320
should show PTP traffic being observed. If the traffic is not observed at the kernel confirm that the SmartNIC port has not been configured to bypass-only mode.
5. If traffic is observed at the kernel but no output is observed at the PTP client, confirm that firewalls have been switched off or configured to allow the PTP traffic to flow.
6. Alternatively, if traffic is observed at the kernel interface but no output is observed leaving the PTP client, confirm that an appropriate route exists in the kernel routing table by inspecting
You can determine the route the multicast address will take by using
ip r g <multicast address> if it's not the SmartNIC PTP interface then a route needs to be added.
7. If traffic is being observed but the grandmaster settings are not correctly set make sure that the PTP domain that the slave is operating in matches the grandmaster. Make sure that the PTP is configured for hardware timestamps. Make sure that the PTP grandmaster has obtained a suitable clock synch and is serving a valid time.
This page was last updated on Dec-10-2020.