Exablaze logo

Hardware based timestamping

Every packet that traverses the card (both received and transmitted) is timestamped to nanosecond level resolution (0.25ns for ExaNIC HPT, 6.2 nanoseconds for all others). This timestamping is done in hardware. To obtain a useful, timestamp, the hardware clock must be synchronized. Synchronizing is possible between the hardware internal timer and the host's clock, a separate pulse-per-second (PPS), or networked PTP grand master.

Synchronizing ExaNIC cards in the same host

To synchronize ExaNIC cards to a host or primary card, the exanic-clock-sync daemon can be used to keep the ExaNIC card's clock synchronized. Each argument to exanic-clock-sync specifies a <device>:<synchronization-source> pair, with the synchronization source being one of host (host clock) or exanicN (another ExaNIC device). For example:

$ exanic-clock-sync exanic0:host

or

$ exanic-clock-sync exanic0:host exanic1:exanic0

If desired this can be placed in system startup scripts. A sample init.d script is provided in configs/exanic-clock-sync.

A utility exanic-clock-check is provided to check how tightly synchronized the host and NIC clocks are:

$ exanic-clock-check
Compares the ExaNIC clock to the host or another ExaNIC
Usage: ./exanic-clock-check <exanic> [<exanic>]

should provide output similar too

$ exanic-clock-check  exanic0
Device exanic0: 1628741401 ticks (1498447571360653132 ns since epoch)
Host clock: 1498447571360653 us since epoch
Difference: 132 ns

Synchronizing ExaNIC cards in different hosts with Pulse per second signals (PPS)

Utilizing the PPS input

To synchronize ExaNIC cards to this PPS signal the exanic-clock-sync daemon can be used. Again each argument to exanic-clock-sync specifies a <device>:<synchronization-source> pair, with PPS as the synchronization source in this example:

$ exanic-clock-sync exanic0:pps

will result in output similar to:

exanic0: Starting clock discipline using PPS
exanic0: Nominal frequency: 161132800 Hz
exanic0: Using differential PPS input
exanic0: PPS signal detected
exanic0: Clock offset at PPS pulse: -9305478.518 us  drift: 0.000 ppm
exanic0: Clock offset at PPS pulse: -17.253 us  drift: 0.000 ppm
...
exanic0: Clock offset at PPS pulse: -0.019 us  drift: -16.794 ppm

By default, the SMA input on ExaNIC cards accepts 3.3V TTL and has a standard 50 ohm termination enabled. To turn termination off, specify pps-no-term instead of pps. This allows for daisy chaining the PPS inputs of several ExaNIC cards together, where the last card in the chain terminates the PPS signal. The intermediate cards should be configured with pps-no-term and the final card with pps.

Intermediate card:

$ exanic-clock-sync exanic0:pps-no-term

Final card:

$ exanic-clock-sync exanic0:pps

Configuring the Pulse-per-second (PPS) output

ExaNICs with a SMA connector (ExaNIC X10 and onwards) can be configured to drive a PPS out. The simplest starting point is to install a single SMA coaxial lead from the first ExaNIC device to a second ExaNIC device. With the cable present run :

$ exanic-config exanic0 pps-out on

running exanic-config will then show the new state of the PPS output.

Device exanic0:
  Hardware type: ExaNIC X10
...
  PPS out: enabled

Information The PPS output is via a female SMA connector, capable of driving 3.3V TTL into a 50 ohm load.

Warning ExaNIC support for PPS out requires firmware from about Aug 2016 or later - please refer to the firmware changelog for the ExaNIC for details.

Synchronizing ExaNIC cards in different hosts with the Precision Time Protocol (PTP)

The Precision Time Protocol (PTP) is a protocol that allows for very accurate time synchronization between a master and a number of client (slave) systems. The ExaNIC supports the Linux SO_TIMESTAMPING API, so standard PTP daemons such as ptp4l (LinuxPTP) or ptpd can be used to synchronize the card's clock.

Information The underlying time standard used by PTP is the International Atomic Time (TAI) standard, which at the time of writing is 37 seconds ahead of the Universal Coordinated Time (UTC) standard. LinuxPTP will set and keep the ExaNIC hardware clock in TAI time, not UTC time. Applications that make use of ExaNIC timestamps will need to be aware of this. Recent kernels are aware of the UTC-TAI offset, and this can be queried using adjtimex().

Information The host operating system requirements for PTP HW and PHC support are:

  • linux kernel >= 3.17, or
  • centos 7 with stock 3.10 kernel, or
  • centos 6 with stock 2.6.32 kernel and nohz=off

Warning It is not recommended or required to run the PTP HW synchronization and PPS exanic-clock-sync at the same time.

Information LinuxPTP (PTP4l) works out of the box, however does not support unicast PTP traffic. We have had good results with ptp4l and there is a good deployment guide at the link below:

LinuxPTP Red Hat Deployment Guide

Information As of December 2016, the current release of ptpd (2.3.1) does not contain the required hardware timestamping code, so you need to either use the hardware timestamping branch described at http://nwtime.org/new-features-for-ptpd/ or to additionally run exanic-clock-sync to synchronize the time from the host.

As an example, a PTP slave ptp4l can then be run with a command such as the following:

$ ptp4l -i eth4 -m

Troubleshooting PTP

Below are some suggestions for troubleshooting PTP configuration for a PTP client. For troubleshooting suggestions for an X10-GM please look at ExaNIC X10-GM troubleshooting.

1. To confirm the timestamping capabilities are present on a slave ExaNIC run the following on your interface: (requires ethtool 3.4 or above)

$ ethtool -T eth1
Time stamping parameters for eth1:
Capabilities:
    hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)
    software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
    hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)
    software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
    software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
    hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)

2. Make sure the ExaNIC port has link up and is configured to the correct speed e.g.

Port 0:
   Interface: eth1
   Port speed: 10000 Mbps
   Port status: enabled, SFP present, link active

to change the port state or speed use the exanic-config commands.

$ sudo exanic-config exanic0:0 up
$ sudo exanic-config exanic0:0 speed 10000

2. Make sure the ExaNIC has been assigned an appropriate IP address and that the master and slave are using the same transport protocol. I.e. multicast or unicast - and be aware of issues arising from hybrid master operation.

3. Generally debugging is simpler with multicast - at which point you can look to see that the ExaNIC port is receiving traffic.

$ sudo exanic-config exanic0:0

Which should show the received packet counters incrementing

RX packets: 708  ignored: 0  error: 0  dropped: 0

4. Then make sure kernel interface is receiving traffic.

$ sudo ifconfig eth1

should also show received packets incrementing

RX packets:587 errors:0 dropped:0 overruns:0 frame:0

and

$ sudo tcpdump -i eth1 port 319 or port 320

should show PTP traffic being observed. If the traffic is not observed at the kernel confirm that the ExaNIC port has not been configured to bypass-only mode.

5. If traffic is observed at the kernel but no output is observed at the PTP client, confirm that firewalls have been switched off or configured to allow the PTP traffic to flow.

6. If traffic is being observed but the grandmaster is not correctly set make sure that the PTP domain that the slave is operating in matches the grandmaster. Make sure that the PTP is configured for hardware timestamps. Make sure that the PTP grandmaster has obtained a suitable clock synch and is serving a valid time.