You are here: Foswiki>Timing Web>TimingSystemDocumentation>TimingSystemHowTo>TestIeee1588NetworkAdapters (28 Oct 2022, EnkhboldOchirsuren)Edit Attach

Test the interoperability of the IEEE 1588 capable network adapters with the White Rabbit switches

Introduction

This report presents the interoperability between the IEEE 1588 capable network adapters and the White Rabbit switches (WRS).

There is a few network adapters that support the hardware timestamping and clock synchronization according to the IEEE 1588 protocol (Precision Time Protocol, PTP). In this work two Ethernet network adapters, which are equipped with the Intel I350 controller, have been tested.

A similar test is done with typical GbE network adapter (that supports only software timestamping) and results are presented here.

1. Hardware and software components used in this work

  • Linux host with a free PCIe slot
  • linuxptp: software implementation of PTP
  • WR switch (WRS): model WRS-3/18, firmware v6.0.1
  • at least 2 SFPs: AXCEN AXGE-3454-0531 (1490nm) for network adapter, AXGE-1254-0531 (1310nm) for WRS
  • fiber cable: simplex jumper LC-LC

2. Test setup

2.1. Installation of the network adapter (ie., BlueLAN CNA)

Check if the target adapter is recognized by the host system: 
$ lspci | grep -i i350
02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Fiber Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation I350 Gigabit Fiber Network Connection (rev 01)

Check the network interfaces: 
$ ifconfig
ens2f0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 80:61:5f:08:b3:2a  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xf7300000-f73fffff  

ens2f1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 80:61:5f:08:b3:2b  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xf7200000-f72fffff
...

Check the driver and hardware support. Use ethtool to show the timestamping capabilities (-T option): 
$ ethtool -T ens2f0
Time stamping parameters for ens2f0:
Capabilities:
   hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)
   software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
   hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)
   software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
   software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
   hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 1
Hardware Transmit Timestamp Modes:
   off                   (HWTSTAMP_TX_OFF)
   on                    (HWTSTAMP_TX_ON)
Hardware Receive Filter Modes:
   none                  (HWTSTAMP_FILTER_NONE)
   all                   (HWTSTAMP_FILTER_ALL)

Repeat it for the ens2f1 network interface.

2.2. Installation and configuration of linuxptp

The linuxptp package is supported by the Debian based distributions and included in their package repositories.

The installation is done by invoking: 

$ sudo apt update && $ sudo apt install linuxptp

It will install the ptp4l command-line tool. The tool requires additional options, which can also be set in an external configuration file. The mostly used options are:
  • i: network interface device to use
  • s: slave only mode
  • m: logging to standard output
  • E/P/A: delay measurement mechanism (E=end-to-end, P=peer-to-peer, A=automatic selection)
  • 2/4/6: network transport layer (2=IEEE 802.3 or Layer2, 4=UDP/IPv4, 6=UDP/IPv6)
  • f: read configuration from file

The options that were used in this work are set in separate configuration files. Please refer to the attached files for the ens2f0 and ens2f1 interfaces of BlueLAN CNA.

In order to start the tool, invoke: 

$ ptp4l -f [config_file]

Below is shown an output message with successful synchronization: 
$ sudo ptp4l -f /etc/linuxptp/bluelan.conf
ptp4l[8023.119]: selected /dev/ptp1 as PTP clock
ptp4l[8023.168]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[8023.168]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[8023.424]: port 1: new foreign master 70b3d5.fffe.91e56c-6
ptp4l[8027.633]: selected best master clock 70b3d5.fffe.91e56c
ptp4l[8027.633]: foreign master not using PTP timescale
ptp4l[8027.633]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[8028.924]: master offset 62155164271 s0 freq  +23673 path delay       249
ptp4l[8030.097]: master offset 62155164223 s1 freq  +23632 path delay       249
ptp4l[8031.001]: master offset         -1 s2 freq  +23631 path delay       249
ptp4l[8031.002]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[8032.095]: master offset          2 s2 freq  +23634 path delay       249
ptp4l[8033.147]: master offset         -4 s2 freq  +23628 path delay       249
ptp4l[8034.155]: master offset         -5 s2 freq  +23626 path delay       249
ptp4l[8035.128]: master offset         -1 s2 freq  +23629 path delay       249

Description:
  • port 1: is the target network interface (ens2f0)
  • state change: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED indicates successful synchronization with a PTP master clock
  • master offset : measured offset from the master in nanoseconds
  • s0, s1, s2: clock servo state (s0=unlocked, s1=clock step, s2=locked)
  • freq : frequency adjustment of the clock in parts per billion
  • path delay : estimated delay of the synchronization messages sent from the master in nanoseconds

For a helpful guide about usage of linuxptp, please refer to this link.

2.3. Configuration of WRS

As a boundary clock (BC) device an WRS is used. Its WR ports must be configured properly.

Use wrs_menuconfig on WRS to configure its WR ports:

  • Port Timing Configuration -> PORT <number, ie., 6> -> Instance 1
    • Profile: White Rabbit, PTP
    • Desired state: Master
    • Network protocol: IEEE 802.3
    • Delay mechanism: End-to-end, Peer-to-peer

As the PTP packets processed by an embedded CPU of WRS, one can snoop the PTP packets with tcpdump: 

#tcpdump -i wri6 -Q in
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wri6, link-type EN10MB (Ethernet), capture size 262144 bytes
14:22:03.311729 IP 192.168.131.162.319 > 224.0.0.107.319: UDP, length 54   # remote device (BlueLAN CNA) with IP 192.168.131.162 sends PTP packets
14:22:03.313928 IP 192.168.131.162.320 > 224.0.0.107.320: UDP, length 54
14:22:03.982525 IP 192.168.131.162.319 > 224.0.0.107.319: UDP, length 54
...

3. Test results

The hardware timestamping and PTP clock synchronization capabilities are tested under different settings of WRS:
  • delay mechanism: E2E (end-to-end), P2P (peer-to-peer)
  • transport layer: Ethernet (layer 2), UDP/IP (layer 3)
  • profile: WR, PTP
  • VLAN support: disabled, enabled

The test results are evaluated by inspecting the logging output of the ptp4l tool (described in the Section 2.2.2).

The both network interfaces of the BlueLAN CNA can be locked only over the Ethernet (layer 2) transport layer regardless of profile, delay mechanism and VLAN support. If the transport layer is selected to UDP/IPv4, then no PTP clock synchronization happens at all (also for software timestamping).

Table 3.1. Hardware timestamping and PTP clock synchronization of the BlueLAN CNA I350-F2 2xSFP network adapter

WRS model VLAN transport delay mechanism profile is network adapter locked? comment
WRS-3/18 disabled raw Ethernet (Layer 2) E2E, P2P WR, PTP yes  
UDP/IPv4 (Layer 3) yes refer to Note 1
enabled raw Ethernet (Layer 2) E2E, P2P WR, PTP yes  
UDP/IPv4 (Layer 3) no refer to Note 1 and Note 2
Table 3.2. Hardware timestamping and PTP clock synchronization of the Ethernet Server Adapter I350-F2 network adapter

not yet tested!

Note 1: For PTP over UDP/IPv4 the WR ports must have IP addresses. Use a custom boot script to assign a dummy IP address to a designated port and enable the custom boot scripts in the configuration (refer to this post for more details).

Note 2: PTP over UDP/IPv4 is not applicable if a VLAN is assigned to a chosen WR port (refer to this post for a workaround solution)

4. Example logging output of ptp4l when using BlueLAN CNA

4.1. Interface ens2f0

BlueLAN CNA interface: ens2f1 WRS settings: VLAN=enabled, WR port=wri6, profile=WR, transport=raw Ethernet (layer2), delay mechanism=E2E 
$ sudo ptp4l -f /etc/linuxptp/bluelan.conf
ptp4l[7033.289]: selected /dev/ptp1 as PTP clock
ptp4l[7033.348]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[7033.348]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[7035.094]: port 1: new foreign master 70b3d5.fffe.91e56c-6
ptp4l[7038.501]: selected best master clock 70b3d5.fffe.91e56c
ptp4l[7038.501]: foreign master not using PTP timescale
ptp4l[7038.501]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[7040.311]: master offset      -1348 s0 freq  +23689 path delay       249
ptp4l[7041.127]: master offset      -1358 s2 freq  +23677 path delay       249
ptp4l[7041.127]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[7041.992]: master offset      -1366 s2 freq  +22311 path delay       249
ptp4l[7042.993]: master offset          4 s2 freq  +23271 path delay       249
ptp4l[7044.159]: master offset        540 s2 freq  +23808 path delay       181
ptp4l[7045.231]: master offset        322 s2 freq  +23752 path delay       249
ptp4l[7046.183]: master offset        244 s2 freq  +23771 path delay       250
ptp4l[7047.266]: master offset        135 s2 freq  +23735 path delay       253
ptp4l[7048.080]: master offset         83 s2 freq  +23723 path delay       257

The output of the WRS sync monitor (look at the status of wri6 port) 
----- HAL ---|---------------------------------- PPSI --------------------------------------------------------
 Iface| Freq |Inst|     Name     |   Config   | MAC of peer port  |    PTP/EXT/PDETECT States    | Pro | VLANs
------+------+----+--------------+------------+-------------------+------------------------------+-----+------
*wri1 |      |  0 |wri1-1-wr-raw |slave       | 00:00:00:00:00:00 | DISABLED  /IDLE      /NONE   | V-W | 2601
*wri2 |      |  1 |wri2-1-wr-raw |master      | 00:00:00:00:00:00 | DISABLED  /IDLE      /NONE   | R-W | 
*wri3 |      |  2 |wri3-1-wr-raw |master      | 00:00:00:00:00:00 | DISABLED  /IDLE      /NONE   | R-W | 
*wri4 |      |  3 |wri4-1-wr-raw |master      | 00:00:00:00:00:00 | DISABLED  /IDLE      /NONE   | R-W | 
*wri5 |      |  4 |wri5-1-wr-raw |master      | 00:00:00:00:00:00 | DISABLED  /IDLE      /NONE   | R-W | 
 wri6 |      |  5 |wri6-1-pt-raw |master      | 80:61:5f:08:b3:2a | MASTER    /          /NONE   | R-P |

4.2. Interface ens2f1

BlueLAN CNA interface: ens2f1 WRS settings: VLAN=enabled, WR port=wri6, profile=WR, transport=raw Ethernet (layer2), delay mechanism=E2E 
$ sudo ptp4l -f /etc/linuxptp/bluelan_ens2f1.conf
[sudo] password for ebold:            
ptp4l[10261.200]: selected /dev/ptp2 as PTP clock
ptp4l[10261.260]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[10261.260]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[10262.542]: port 1: new foreign master 70b3d5.fffe.91e56c-6
ptp4l[10266.194]: selected best master clock 70b3d5.fffe.91e56c
ptp4l[10266.194]: foreign master not using PTP timescale
ptp4l[10266.194]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[10268.582]: master offset 134256972232 s0 freq  +23588 path delay       502
ptp4l[10269.550]: master offset 134256972037 s1 freq  +23387 path delay       498
ptp4l[10270.537]: master offset         -7 s2 freq  +23380 path delay       498
ptp4l[10270.537]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[10271.727]: master offset        -20 s2 freq  +23365 path delay       498
ptp4l[10272.722]: master offset          1 s2 freq  +23380 path delay       486
ptp4l[10273.676]: master offset         -4 s2 freq  +23375 path delay       483
ptp4l[10274.675]: master offset        -18 s2 freq  +23360 path delay       482
ptp4l[10275.655]: master offset        -16 s2 freq  +23356 path delay       482
ptp4l[10276.671]: master offset         -6 s2 freq  +23361 path delay       481
ptp4l[10277.581]: master offset         -5 s2 freq  +23361 path delay       481

----- HAL ---|---------------------------------- PPSI --------------------------------------------------------
 Iface| Freq |Inst|     Name     |   Config   | MAC of peer port  |    PTP/EXT/PDETECT States    | Pro | VLANs
------+------+----+--------------+------------+-------------------+------------------------------+-----+------
*wri1 |      |  0 |wri1-1-wr-raw |slave       | 00:00:00:00:00:00 | DISABLED  /IDLE      /NONE   | V-W | 31
*wri2 |      |  1 |wri2-1-wr-raw |master      | 00:00:00:00:00:00 | DISABLED  /IDLE      /NONE   | V-W | 31
*wri3 |      |  2 |wri3-1-wr-raw |master      | 00:00:00:00:00:00 | DISABLED  /IDLE      /NONE   | R-W | 
*wri4 |      |  3 |wri4-1-wr-raw |master      | 00:00:00:00:00:00 | DISABLED  /IDLE      /NONE   | R-W | 
*wri5 |      |  4 |wri5-1-wr-raw |master      | 00:00:00:00:00:00 | DISABLED  /IDLE      /NONE   | R-W | 
 wri6 |      |  5 |wri6-1-wr-raw |master      | 80:61:5f:08:b3:2b | MASTER    /IDLE      /EXT_OFF| R-W |
-- EnkhboldOchirsuren - 26 Oct 2022
I Attachment Action Size Date Who Comment
bluelan.confconf bluelan.conf manage 1 K 26 Oct 2022 - 15:38 EnkhboldOchirsuren linuxptp configuration for the 1st interface of BlueLAN CNA I350-F2
bluelan_ens2f1.confconf bluelan_ens2f1.conf manage 1 K 26 Oct 2022 - 15:39 EnkhboldOchirsuren linuxptp configuration for the 2nd interface of BlueLAN CNA I350-F2
bluelan_udp.confconf bluelan_udp.conf manage 1 K 28 Oct 2022 - 08:54 EnkhboldOchirsuren linuxptp configuration (PTP over UDP/IPv4) for the 1st interface of BlueLAN CNA I350-F2
Edit  | Attach | Print version | History: r2 < r1 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r2 - 28 Oct 2022, EnkhboldOchirsuren
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback