Hardware based timestamping
Every packet that traverses the card (both received and transmitted) is timestamped to nanosecond level resolution (0.25ns for ExaNIC HPT, 6.2 nanoseconds for all others). This timestamping is done in hardware. To obtain a useful, timestamp, the hardware clock must be synchronized. Synchronizing is possible between the hardware internal timer and the host's clock, a separate pulse-per-second (PPS), or networked PTP grand master.
Synchronizing ExaNIC cards in the same host
To synchronize ExaNIC cards to a host or primary card, the
exanic-clock-sync daemon can be used to keep the ExaNIC card's clock synchronized. Each argument to
exanic-clock-sync specifies a
<device>:<synchronization-source> pair, with the synchronization source being one of host (host clock) or exanicN (another ExaNIC device). For example:
$ exanic-clock-sync exanic0:host
$ exanic-clock-sync exanic0:host exanic1:exanic0
If desired this can be placed in system startup scripts. A sample
init.d script is provided in
exanic-clock-check is provided to check how tightly synchronized the host and NIC clocks are:
$ exanic-clock-check Compares the ExaNIC clock to the host or another ExaNIC Usage: ./exanic-clock-check <exanic> [<exanic>]
should provide output similar too
$ exanic-clock-check exanic0 Device exanic0: 1628741401 ticks (1498447571360653132 ns since epoch) Host clock: 1498447571360653 us since epoch Difference: 132 ns
Synchronizing ExaNIC cards in different hosts with Pulse per second signals (PPS)
Utilizing the PPS input
To synchronize ExaNIC cards to this PPS signal the exanic-clock-sync daemon can be used. Again each argument to exanic-clock-sync specifies a
<device>:<synchronization-source> pair, with PPS as the synchronization source in this example:
$ exanic-clock-sync exanic0:pps
will result in output similar to:
exanic0: Starting clock discipline using PPS exanic0: Nominal frequency: 161132800 Hz exanic0: Using differential PPS input exanic0: PPS signal detected exanic0: Clock offset at PPS pulse: -9305478.518 us drift: 0.000 ppm exanic0: Clock offset at PPS pulse: -17.253 us drift: 0.000 ppm ... exanic0: Clock offset at PPS pulse: -0.019 us drift: -16.794 ppm
By default, the SMA input on ExaNIC cards accepts 3.3V TTL and has a standard 50 ohm termination enabled. To turn termination off, specify
pps-no-term instead of
pps. This allows for daisy chaining the PPS inputs of several ExaNIC cards together, where the last card in the chain terminates the PPS signal. The intermediate cards should be configured with
pps-no-term and the final card with
$ exanic-clock-sync exanic0:pps-no-term
$ exanic-clock-sync exanic0:pps
Configuring the Pulse-per-second (PPS) output
ExaNICs with a SMA connector (ExaNIC X10 and onwards) can be configured to drive a PPS out. The simplest starting point is to install a single SMA coaxial lead from the first ExaNIC device to a second ExaNIC device. With the cable present run :
$ exanic-config exanic0 pps-out on
exanic-config will then show the new state of the PPS output.
Device exanic0: Hardware type: ExaNIC X10 ... PPS out: enabled
The PPS output is via a female SMA connector, capable of driving 3.3V TTL into a 50 ohm load.
ExaNIC support for PPS out requires firmware from about Aug 2016 or later - please refer to the firmware changelog for the ExaNIC for details.
Synchronizing ExaNIC cards in different hosts with the Precision Time Protocol (PTP)
The Precision Time Protocol (PTP) is a protocol that allows for very accurate time synchronization between a master and a number of client (slave) systems. The ExaNIC supports the Linux
SO_TIMESTAMPING API, so standard PTP daemons such as ptp4l (LinuxPTP) or ptpd can be used to synchronize the card's clock.
The underlying time standard used by PTP is the International Atomic Time (TAI) standard, which at the time of writing is 37 seconds ahead of the Universal Coordinated Time (UTC) standard. LinuxPTP will set and keep the ExaNIC hardware clock in TAI time, not UTC time. Applications that make use of ExaNIC timestamps will need to be aware of this. Recent kernels are aware of the UTC-TAI offset, and this can be queried using adjtimex().
The host operating system requirements for PTP HW and PHC support are:
- linux kernel >= 3.17, or
- centos 7 with stock 3.10 kernel, or
- centos 6 with stock 2.6.32 kernel and nohz=off
It is not recommended or required to run the PTP HW synchronization and PPS exanic-clock-sync at the same time.
LinuxPTP (PTP4l) works out of the box, however does not support unicast PTP traffic. We have had good results with ptp4l and there is a good deployment guide at the link below:
As of December 2016, the current release of ptpd (2.3.1) does not contain the required hardware timestamping code, so you need to either use the hardware timestamping branch described at http://nwtime.org/new-features-for-ptpd/ or to additionally run exanic-clock-sync to synchronize the time from the host.
As an example, a PTP slave ptp4l can then be run with a command such as the following:
$ ptp4l -i eth4 -m
Below are some suggestions for troubleshooting PTP configuration for a PTP client. For troubleshooting suggestions for an X10-GM please look at ExaNIC X10-GM troubleshooting.
1. To confirm the timestamping capabilities are present on a slave ExaNIC run the following on your interface: (requires ethtool 3.4 or above)
$ ethtool -T eth1 Time stamping parameters for eth1: Capabilities: hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE) software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE) hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE) software-receive (SOF_TIMESTAMPING_RX_SOFTWARE) software-system-clock (SOF_TIMESTAMPING_SOFTWARE) hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE)
2. Make sure the ExaNIC port has link up and is configured to the correct speed e.g.
Port 0: Interface: eth1 Port speed: 10000 Mbps Port status: enabled, SFP present, link active
to change the port state or speed use the exanic-config commands.
$ sudo exanic-config exanic0:0 up $ sudo exanic-config exanic0:0 speed 10000
2. Make sure the ExaNIC has been assigned an appropriate IP address and that the master and slave are using the same transport protocol. I.e. multicast or unicast - and be aware of issues arising from hybrid master operation.
3. Generally debugging is simpler with multicast - at which point you can look to see that the ExaNIC port is receiving traffic.
$ sudo exanic-config exanic0:0
Which should show the received packet counters incrementing
RX packets: 708 ignored: 0 error: 0 dropped: 0
4. Then make sure kernel interface is receiving traffic.
$ sudo ifconfig eth1
should also show received packets incrementing
RX packets:587 errors:0 dropped:0 overruns:0 frame:0
$ sudo tcpdump -i eth1 port 319 or port 320
should show PTP traffic being observed. If the traffic is not observed at the kernel confirm that the ExaNIC port has not been configured to bypass-only mode.
5. If traffic is observed at the kernel but no output is observed at the PTP client, confirm that firewalls have been switched off or configured to allow the PTP traffic to flow.
6. If traffic is being observed but the grandmaster is not correctly set make sure that the PTP domain that the slave is operating in matches the grandmaster. Make sure that the PTP is configured for hardware timestamps. Make sure that the PTP grandmaster has obtained a suitable clock synch and is serving a valid time.