Synchronize the clock of non-WR devices in the WR network

1. Introduction

This document presents specific configurations of White Rabbit (WR) switches, which are used to synchronize clock of non-WR nodes with the precision time protocol (PTP).

2. PTP and WR

The precision time protocol (PTP) is dedicated to synchronize clocks in measurement, control and telecommunication systems communicating by local area networks (LAN) and is defined in the IEEE 1588 standards. It works with a master-slave architecture. The slaves are synchronized to their masters, which may be slaves to their own masters. The standard defines how master and slaves communicate and where timestamps are generated and how the differences are calculated. When the time stamping is assisted by hardware, it provides synchronization accuracy to the sub-microsecond level.

The PTP protocol can be enhanced with profiles to meet the specific use-case requirements. In the accelerator timing systems of CERN and GSI/FAIR a high-accuracy profile of PTP, so called White Rabbit (WR) is deployed. WR can push the standard PTP accuracy to the sub-nanosecond level by utilizing synchronous Ethernet (SyncE) technology and a dedicated phase detection hardware to synchronize clock frequencies of master and slave clocks.

2. WR timing network

The GSI timing system basically consists of GPS displined oscillator (GSPDO), a WR grandmaster and multiple WR boundary clocks. All WR grandmaster and boundary clocks are WR switches (WRS). They implement the IEEE 1588-2008 (PTPv2) and IEEE 1588-2019 (PTPv2.1) standards partially and supports following configurations (user manual in pdf and FAQ):

• 2-step only
• multicast only
• request-response and peer-to-peer delay mechanisms (per port)
• Layer 2 (raw) and Layer 3 (UDP/IPv4) transport (per port)
• assignment of fixed master/slave role (per port)
• rx/tx delays (per port)

In general, if non-WR devices are properly configured and have the Gigabit Ethernet (GbE) network interface, then they can be synchronized by WRS. Refer to this FAQ.

A known issue in the latest software release of WRS is that PTP over UDP/IPv4 cannot be applied to WR ports, which are assigned to VLAN.

The GSI timing network is built on the tree topology to distribute time and logically organized in VLANs as WRSs are aware of VLAN. In addition, the MAC addresses of valid network nodes must be registered for the 802.1x authentication to have a full access to a desired VLAN. For unregistered devices the network access is so limited that only the WR or PTP synchronization and reception of timing messages are allowed.

3. Configurations of WRS

The clock synchronization of non-WR devices in the WR network is demonstrated between a boundary clock (WRS) and an ordinary clock (non-WR device). A clock with only one port is called an ordinary clock (OC) and it can be master or slave. A clock with multiple ports is called a boundary clock (BC) and it can be master on one port and slaves on other ports.

Figure 1. PTP clock synchronization in the WR network

There are several implementations of PTP in software. One of them is the linuxptp project. It includes a command-line tool, ptp4l, that implements the PTP boundary clock and ordinary clock. Hence, it is chosen as PTP OC in this work and runs on a Linux host with GbE network card (software time stamping is used). The connection to WRS is done via an Ethernet cable and a copper SFP module. A sample configuration for ptp4l (used in this demo) can be found in an attached file.

An WRS is chosen as an PTP BC. For PTP following 3 settings are taken into account:

• transport layer: layer 2 (raw Ethernet) and layer 3 (UDPv4)
• delay mechanism: end-to-end (E2E), peer-to-peer (P2P)
• profile: WR and PTP

These settings are tested under two configurations for WRS:

• default: VLAN is disabled, wri1 is slave clock, wri2-18 are master clocks
• access-layer: VLAN is enabled, wri1 is slave clock, wri2-18 are master clocks

PTP clock synchronization is verified with the output message of ptp4l.

PTP over layer 2 (Ethernet) will function with all delay mechanisms and profiles. PTP over layer 3 (UDP/IPv4) will work only in WR network without VLAN. Hence, the layer 2 (Ethernet) protocol is the recommended transport protocol for WRSs, if non-WR devices support it.

The comparisons of clock synchronization under different configurations are listed in Table 1 and 2.

Table 1. Default WRS configurations (without VLAN)

WRS model	transport	delay mechanism	profile	is non-WR device locked?	comment
WRS-3/18	Layer 2 (raw Ethernet)	E2E, P2P	WR, PTP	yes
	Layer 3 (UDP/IPv4)			yes	refer to Note 1

Table 2. Access-layer configurations (with VLAN)

WRS model	transport	delay mechanism	profile	is non-WR device locked?	comment
WRS-3/18	Layer 2 (raw Ethernet)	E2E, P2P	WR, PTP	yes
	Layer 3 (UDP/IPv4)			no	refer to Notes 1 and 2

Note 1: For PTP over UDP/IPv4 the WR ports must have IP addresses. Use a custom boot script to assign a dummy IP address to a designated port and enable the custom boot scripts in the configuration (refer to this post for more details).

Note 2: PTP over UDP/IPv4 is not applicable if a VLAN is assigned to a chosen WR port (refer to this post for a workaround solution)

4. Safety consideration and a buffer WRS

A previous experiment shows that excessive incoming PTP and LLDP traffic can cause frame loss in WRS and lead to failure in WR synchronization. In order to block incoming traffic from the PTP network segment an additional WRS with the default configuration can be connected between an access-layer WRS and non-WR devices as a buffer WRS. With such cascaded connection traffic from the buffer WRS is blocked by the access-layer WRS: the MAC address of the up-link port of the buffer WRS is unknown, therefore it is handled as unregistered device. All outgoing traffic from the access-layer WRS are normally forwarded to the buffer WRS.

-- EnkhboldOchirsuren - 02 Jun 2022