Mirror and timestamping - Cisco Nexus 3550-F Fusion

The mirror configuration provides a mechanism for configuring the logging and timestamping functionality within the Nexus 3550-F. A mirror replicates any data traversing its member objects out of another port. Ports, mux and switch objects can be added to a mirror to replicate traffic that traverses them.

The mirror can also be configured to enable timestamping. All packets that enter the Nexus 3550-F are timestamped on arrival at the ingress port. This timestamp is then inserted into the packet body when packets arrive at the mirror egress port.

Example use of a mirror, showing port A1 and port B3 mirrored to output port C3.

Creating a mirror

Port mirrors can be created after specifying a name, completed in either configuration mode, or prefixing the command with the config keyword.

admin@N3550-F(config)> mirror mymirror
Mirror "mymirror" created

Creating a mirror will place the command line into a config-mirror state. Within this state an output port for the mirror can be assigned using the output command. The output port can be any of the available front panel ports.

admin@N3550-F(config-mirror:mymirror)> output C3
Added output port "C3" to mirror "mymirror"

Note: It is not possible to use a port that is already in use for the output port.

Note: There are only four timestamping/mux outputs available in the FPGA. These can be assigned arbitrarily to front-panel ports. Adding an output port to a mirror will consume one of these outputs. If you need a lot of outputs from a mirror, it's worth considering a mirror with only one output port, and using a tap object from that output port to the other ports.

The mirror will replicate the traffic of any of its member objects. To assign an object to the mirror use the object type followed by the name. Objects that can be mirrored are port, mux and switch objects. For example, to replicate all traffic received on ports A1 and B3:

admin@N3550-F(config-mirror:mymirror)> port A1; port B3
Added port "A1" to mirror "mymirror"
Added port "B3" to mirror "mymirror"

In addition to replicating specific ports, it is also possible to replicate all traffic received by a given mux or switch object. For example, to replicate all of the traffic passing through an existing mux object, use the mux command:

admin@N3550-F(config-mirror:mymirror)> mux mymux
Added mux "mymux" to mirror "mymirror"

Note: It is not possible to explicitly mirror outputs to a port using a mirror object, only inputs to ports or mux/switch objects. In order to take a copy of all traffic leaving a port, it is suggested to use a tap object and tap the output port.

Note: Mirroring is not supported when using fastmux firmware.

To remove an object from the mirror, use the no form of the commands:

admin@N3550-F(config-mirror:mymirror)> no mux mymux
Removed mux "mymux" from mirror "mymirror"
admin@N3550-F(config-mirror:mymirror)> no port A1
Removed port "A1" from mirror "mymirror"

Removing a mirror

When a port mirror is no longer needed it is possible to remove the mirror. Using the no form of the mirror command:

admin@N3550-F> config no mirror mymirror
Mirror "mymirror" deleted

Timestamping

The Nexus 3550-F timestamps packets upon arrival using a 350MHz 64 bit timestamp counter. Mirror output ports can be configured to transmit the timestamp along with packet in a number of different modes using the timestamp command. Timestamps are taken at the moment the first bit of the incoming packet arrives.

A utility that allows reception and decoding of these timestamped streams is available on GitHub. This can be used to write out a pcap file with nanosecond accurate timestamps.

In order to replace the Ethernet FCS (Frame Check Sequence) with the lower 32 bits of the timestamp counter, the timestamp mode should be set to fcs:

admin@N3550-F> config mirror mymirror timestamp fcs
Set timestamp mode "fcs" on mirror "mymirror"
admin@N3550-F> config mirror mymirror no timestamp
Timestamping disabled on "mymirror"

Each increment of the 32 bit counter represents approximately 2.86 ns of time. The lower 32 bits of this counter wrap every 12.3 seconds. To allow the capturing host to reconstruct the absolute UTC (Universal Time Coordinated) time of each packet, the mirror port will send periodic keyframes that are not part of the capture stream. These keyframes contain the full 64 bit counter and the UTC time in nanoseconds associated with the counter value.

These keyframes and FCS timestamps allow the remote host to reconstruct the absolute time of arrival of all packets in the captured feed.

The Nexus 3550-F also supports an FCS Compatibility mode, whereby instead of replacing the FCS with the lower 32 bits of the timestamp counter the FCS is replaced with a timestamp format compatible with Arista devices and systems or applications designed to recognize and process the timestamp format provided by Arista. Similar to the FCS mode, the timestamp command can be used again:

admin@N3550-F> config mirror mymirror timestamp fcs-compat
Set timestamp mode "fcs-compat" on mirror "mymirror"

In this format the timestamp counter's lower 31 bits are written over the FCS, with a gap found in the highest order bit of the lower order byte is a zeroed padding (i.e. 31 bits of data in a 32-bit format). Note that even in FCS Compatibility mode, timestamps are taken immediately upon packet arrival at an ingress port, and not upon arrival of the FCS field.

For instances where it is desirable for the timestamped frame to include a valid FCS, another two modes are supported where the timestamp is inserted in place of the original FCS bytes, and a new FCS for the whole packet is calculated and transmitted, therefore increasing the size of each transmitted frame by 4 bytes. Again, the selection between standard and compat modes are available:

admin@N3550-F> config mirror mymirror timestamp append
Set timestamp mode "append" on mirror "mymirror"
admin@N3550-F> config mirror mymirror timestamp append-compat
Set timestamp mode "append-compat" on mirror "mymirror"

This page was last updated on Feb-19-2021.