WR PTP synchronization of WRS under excessive PTP and LLDP traffic

1. Introduction

The White Rabbit (WR) technique is specially developed to provide sub-nanosecond time synchronization for the real-time control systems. It employs the IEEE 1588 precision time protocol (PTP) to synchronize time, synchronous Ethernet (SyncE) to achieve syntonization and digital dual-mixer time difference (DDMTD) phase detection module to measure the frequency difference between master reference clock and local clock. This technique is defined as WR profile in PTP and is referred to as WR PTP.

The White Rabbit Switch (WRS) is the main component of the WR networks. Its non-time-critical functions are implemented in software:

• PTP stack of WR PTP: ppsi daemon
• link layer discovery protocol (LLDP) stack: lldpd daemon

In general, excessive PTP event (ie., Sync, Delay Request) or LLDP traffic might cause high CPU load, which results in temporary failure to WR PTP synchronization. There are the RFC 2889 benchmarking results available for WRSs, refer to pdf. The results from the full and part mesh tests present that mostly Ethernet frames with shorter lenghts (64 and 128 bytes) have been lost massively at all possible bandwidths, which ranges from 100Mb/s to 1Gb/s. All PTP event messages have the length of 34 or 44 bytes (excluding the Ethernet header and frame check sequence), refer this external documentation for the message structures. The LLDP frames can vary in length but those sent by most Timing Receivers (TR) lay within 128 bytes. Therefore, the goal of this work are to demonstrate the failure of the WR PTP synchronization in a target WRS under excessive PTP and LLDP traffic and determine the critical packet rates of each traffic.

2. Test setup

This experiment considers only the incoming PTP and LLDP frames for WRSs.

2.1. Components

The setup contains one WRS, one TR and the XenaBay traffic generator.

The WRS is configured as an access-layer WRS according to the Timing Network v2 (TN2) installation scheme. This configuration is designated for the boundary clock (BC) devices, which directly synchronize time of end nodes like TRs. BC devices have multiple ports: one of them will act as a slave to another master port of a different BC or to the grandmaster, all other ports will act as a master to other slaves connected to them. Therefore, the access-layer WRSs are the first targets that are easily attacked by the slave devices. For the access-layer WRSs, ports wri2-18 are configured to the master mode, only port wri1 is in the slave mode.

The network traffic of the PTP event messages and LLPD packets are generated by the XenaBay traffic generator. It includes 3 test modules, each has 6 SFP ports. These ports are labeled as P[chassis][module][port], for example port 2 of test module 2 is labeled as P022.

The given TR is used to monitor the WR PTP synchronization with WRS. Its software stack provides EtherBone tools to monitor WR PTP synchronization locally: the actual WR PTP status is monitored by built-in 'gui' command (first launch 'eb-console dev/wbm0'), for statistics launch 'eb-mon -g dev/wbm0i'.

The setup of test network is given below:

wrs:wri2  <=> TR:wr0
wrs:wri11 <=> Xenabay:P022
wrs:wri13 <=> Xenabay:P040
wrs:wri14 <=> Xenabay:P041
wrs:wri15 <=> Xenabay:P042
wrs:wri16 <=> Xenabay:P043

A brief information about test devices are given below:

1. WRS

Model         : WRS-3/18
SCB HW        : 3.4
FPGA          : LX240T
Software      : WP3a-wrpc_fixes-24-gf313343b (Adam Wujek); Sep 24 2021 03:21:24
Configuration : dot-config_timing_access

2. TR

Platform    : scu3 +comexpress
FPGA model  : Arria II GX (ep2agx125ef29c5)
Source info : fallout-v6.2.1_mini_release-4050
Build type  : fallout-v6.2.1
Build date  : Wed Mar 16 13:27:35 CET 2022

2.2. Test traffic

As mentioned earlier the PTP and LLDP protocol stacks run as system services (ppsi, lldpd) on the embedded Linux system of WRS. In normal case, they deal with protocol traffic that have much lower packet rates:

• PTP Sync: 1 packet/s
• PTP Announce: 0,5 packet/s (or 1 message every 2 seconds)
• LLDP: 0,2 packet/s (or 1 message every 5 seconds)

These rates can be changed in switch configuration (faq and user manual). A logarithm scale is used to set the PTP packet rate according to the IEEE 1588 standard: packet_rate=1/2^log_value. The allowed maximum rates are 128 packet/s for the Sync and Delay Request message and 8 packet/s for the Announce message.

Three network frames are defined in XenaBay for the test traffic:

• PTP Sync: 70 bytes (assigned to port P022)
• PTP Delay Request: 64 bytes (assigned to ports P040-042)
• LLDP: 114 bytes (assigned to port P043)

The PTP Sync and Delay Request frames contain:

• Ethernet header
• PTP header
• PTP body

The LLDP frame contains:

• Ethernet header
• system description

The desired excessive traffic is generated by sending these frames by XenaBay. Each frame is streamed individually for certain period of time (ie., at least 1 minute) at different packet rates, until the TR connected to the target WRS gets out of time synchronization.

3. Test results

The measured critical (minimum) packet rates of the PTP and LLDP traffic, at which TR connected to the target WRS starts to loose PTP lock, are shown in Table.1.

Table.1. Measured critical values of the PTP and LLDP packet rates for WRS

Frame	Size, bytes	Packet rate, pps	Data rate, Mbps
PTP Sync	70	~20200	~14,54
PTP Delay Request	64	~20400	~13,70
LLDP	114	~19000	~20,36

Note that these values are obtained by streaming the test traffic only to single port. In case of multiple slaves, the total achieved packet rate should be considered instead of individual packet rates.

Besides WR PTP failure, any remote access (ie., ssh, SNMP, ping) has been disturbed during the test streaming. This situation is restored to normal (without system crash) once the test streaming is stopped: TR locks again without any trouble.

All configurations of the target WRS and ValkyrieManager (Windows GUI for XenaBay) used in this test can be found in the 'dos_ptp_lldp_2022_5' directory of this Github repository.

-- EnkhboldOchirsuren - 27 May 2022