Run the ordinary and boundary clocks using linuxptp

The linuxptp package is a PTPv2 implementation according to the IEEE 1588 standard for Linux. It includes the certain programs (ptp4l, phc2sys, phc_ctl, timemaster) and utility tool (pmc).

ptp4l is an implementation of the PTP boundary clock (BC) and ordinary clock (OC). If the hardware timestamping is supported it is used to synchronize the PTP hardware clock (PHC) on the network interface card (NIC) to a master clock. Otherwise, with the software timestamping it synchronizes the system time to the master clock.

Moreover, ptp4l is implemented for the user space and assisted by the Linux kernel with its features:
  • socket option SO_TIMESTAMPING for packet timestamping
  • PHC subsystem that allows to access the PHC devices via system calls
  • some drivers support hardware and/or software timestamping

phc2sys is used to synchronize two or more clocks. With the hardware timestamping it is typically used to synchronizes the system clock with PHC on NIC, which itself is synchronized by the ptp4l program.

phc_ctl can be used directly control a PHC device. Typically, it is used for debugging purpose. In general, phc2sys should be used to control the PHC devices.

timemaster is used to run NTP with PTP as reference clocks. It uses ptp4l and phc2sys in combination of chronyd or ntpd to synchronize the system clock to NTP and PTP time sources. The PTP time is provided by phc2sys and ptp4l to chronyd/ntpd, which can compare all time sources and use the best sources to synchronize the system clock.

The pmc tool implements a PTP management client and is used to send PTP management messages (get/set/command) to ptp4l in run-time.

This document describes how to build the simple OC and BC using linuxptp.

The components that were used for demonstration:
  • Linux host with linuxptp
    • kernel v5.15.0
    • linuxptp v3.1.1
    • EB tools (eb-info, eb-console) and the wishbone driver
    • (optional) saftlib tools v2.5.1
  • network interface card (NIC) with the PTP hardware clock (PHC):
    • BlueLAN converged network adapter with dual interface (recognized as enp2s0f0, enp2s0f1), link
  • White Rabbit (WR) switch
    • model WRS-3/18 from Creotech
    • software version: v6.0.1
    • configuration version: default and v1.5.2
      • 802.1x in WRS does not care of the PTP frames => PTP will always function
  • SFPs: AXCEN AXGE-3454-0531 (1490nm), AXGE-1254-0531 (1310nm)
  • fiber optic cables (LC-LC, simplex)
  • (optional) timing receivers (TR)
    • Pexaria5 installed in the Linux host (dev/wbm0) and/or SCU3
    • gateware version: fallout-v6.1.2
  • (optional) digital oscilloscope
  • (optional) LEMO cables

1. Prerequisites

1.1. Confirm no time synchronization on a local system

In order to avoid mess, the time synchronization of a local system must be disabled.

On Debian-based distros (Ubuntu, Linux Mint) the network time synchronization is handled by timesyncd (instead of ntpd). The status of timesyncd can be queried by timedatectl:
$ timedatectl
               Local time: Thu 2022-11-03 11:27:46 CET
           Universal time: Thu 2022-11-03 10:27:46 UTC
                 RTC time: Thu 2022-11-03 10:27:46
                Time zone: Europe/Berlin (CET, +0100)
System clock synchronized: yes                            # system time is successfully synchronized
              NTP service: active                         # timesyncd is up and running
          RTC in local TZ: no

With timedatectl the NTP service can be turned off/on.
$ sudo timedatectl set-ntp false # turn off
$ sudo timedatectl set-ntp true  # turn on

1.2. BlueLAN CNA I350-F2

It is a dual port, converged network adapter (CNA). In general, the CNA adapters combines two functions of storage area network (SAN) and general-purpose computer networking into a single hardware:
  • host-bus adapter (HBA)
  • network interface controller (NIC)

These adapters support the Fiber-Channel over Ethernet (FCoE) and Gigabit Ethernet (GbE) technologies and enable an access to SAN and LAN via single interface.

The chosen CNA provides two network interfaces for Gigabit SFP trancievers:
$ ifconfig
...
enp2s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.38.124  netmask 255.255.0.0  broadcast 169.254.255.255
        ether 80:61:5f:08:b3:2a  txqueuelen 1000  (Ethernet)
        RX packets 53382  bytes 3947163 (3.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 51124  bytes 8750019 (8.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0x72300000-723fffff

enp2s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.131.24  netmask 255.255.0.0  broadcast 169.254.255.255
        ether 80:61:5f:08:b3:2b  txqueuelen 1000  (Ethernet)
        RX packets 57199  bytes 13727442 (13.7 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 84617  bytes 10944351 (10.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0x72200000-722fffff
...

Moreover, each network interface is capable of hardware timestamping:
$ ethtool -T enp2s0f0                    # repeat it also for 'enp2s0f1'
Time stamping parameters for enp2s0f0:
Capabilities:
   hardware-transmit
   software-transmit
   hardware-receive
   software-receive
   software-system-clock
   hardware-raw-clock
PTP Hardware Clock: 0                    # its PHC is /dev/ptp0
Hardware Transmit Timestamp Modes:
   off
   on
Hardware Receive Filter Modes:
   none
   all

1.3. Usage of ptp4l

The typical usage of ptp4l with hardware timestamping is to synchronize a PHC device on NIC to a PTP master. At least following options/arguments are required:
  • -i: specify a network interface to send/receive PTP packets
  • -s: slave clock mode (default is the master mode)
  • -E: end-to-end delay measurement mechanism (default)
  • -2: raw Ethernet (802.3) as transport protocol (default is UDP/IPv4)
  • hardware timestamping (default)

A command given below synchronizes the PHC clock (/dev/ptp0) of the first network interface (ie., enp2s0f0) to a PTP master (ie., WR switch):

$ sudo ptp4l -i enp2s0f0 -E -2 -s -m --step_threshold=0.00002
ptp4l[8875.899]: selected /dev/ptp0 as PTP clock
ptp4l[8875.943]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[8875.943]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[8876.992]: port 1: new foreign master 70b3d5.fffe.91e37e-2
ptp4l[8881.348]: selected best master clock 70b3d5.fffe.91e37e
ptp4l[8881.348]: foreign master not using PTP timescale
ptp4l[8881.348]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[8884.494]: master offset         -4 s0 freq  +31255 path delay       454
ptp4l[8885.585]: master offset        -39 s2 freq  +31223 path delay       461
ptp4l[8885.585]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[8886.617]: master offset        -36 s2 freq  +31187 path delay       461
ptp4l[8887.723]: master offset         27 s2 freq  +31239 path delay       457
ptp4l[8888.783]: master offset         18 s2 freq  +31238 path delay       458
ptp4l[8889.627]: master offset         22 s2 freq  +31248 path delay       456
ptp4l[8890.725]: master offset         33 s2 freq  +31265 path delay       456
ptp4l[8891.531]: master offset         15 s2 freq  +31257 path delay       458
ptp4l[8892.483]: master offset          1 s2 freq  +31248 path delay       458
ptp4l[8893.649]: master offset        -17 s2 freq  +31230 path delay       459
ptp4l[8894.816]: master offset         -4 s2 freq  +31238 path delay       458

The '--step_threshold=n' option is used to adjust the time faster. If the time jump is greater than 'n' seconds, time is adjusted by stepping the clock instead of changing the frequency. Use '--summary_interval ' option to get summary statistics, instead of log output every clock update. The interval is a power of two in seconds (ie., 5 specifies 32-second interval)

Status of the PTP clock synchronization in the log output includes:
  • offset between master and slave clocks, nanoseconds
  • clock servo state, s0=unlocked, s1=clock step, s2=locked
  • frequency adjustment of clock, parts per billion
  • path delay, nanoseconds

The state change "UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED" indicates successful synchronization with the PTP master clock.

1.4. Usage of phc2sys

Its typical usage is to adjust the system clock based on PHC of a slave clock (network interface that synchronizes to a PTP master). This way a local system can operate as an OC.

As it is used to synchronize two or more clocks, it is also used to synchronize the PHC devices of other master clocks in a BC. Required options/arguments are:
  • -s: specify a source clock by a network interface (or PHC device), it is a slave clock in a BC
  • -c: specify other clock by a network interface (or PHC device), it is a master clock in a BC
  • -w: wait until ptp4l instance is synchronized

A command given below synchronizes the PHC devices on each network interface of BlueLAN CNA. Here, the PHC device (/dev/ptp1) on the second network interface (enp2s0f1) is synchronized to the PHC device on the first network interface (enp2s0f0).
$ sudo phc2sys -s enp2s0f0 -c enp2s0f1 -w -m -S 0.00002
phc2sys[8987.646]: enp2s0f1 phc offset    381755 s0 freq  +29187 delay 233433
phc2sys[8988.648]: enp2s0f1 phc offset    382261 s1 freq  +29691 delay 236393
phc2sys[8989.650]: enp2s0f1 phc offset      2710 s2 freq  +32401 delay 235513
phc2sys[8990.652]: enp2s0f1 phc offset      1161 s2 freq  +31665 delay 236201
phc2sys[8991.654]: enp2s0f1 phc offset       -92 s2 freq  +30761 delay 234713
phc2sys[8992.656]: enp2s0f1 phc offset     -1591 s2 freq  +29234 delay 235289
phc2sys[8993.658]: enp2s0f1 phc offset     -2838 s2 freq  +27510 delay 222794

The '-S n' option can be used to set the step threshold (in seconds) of the servo. Use '-u ' option to print the summary statistics instead of individual update sample. Requires the number of clock updates.

1.5. Usage of phc_ctl

Get the (PTP) time of PHC devices.

Both commands given below read the time of PHC on the first network interface of BlueLAN CNA.
$ sudo phc_ctl enp2s0f0 -- get; sudo phc_ctl /dev/ptp0 -- get
phc_ctl[6414.864]: clock time is 1631095590.304911375 or Wed Sep  8 12:06:30 2021

phc_ctl[6414.877]: clock time is 1631095590.317937114 or Wed Sep  8 12:06:30 2021

1.6. Usage of pmc

This tool can be used to get the current status/configuration of a slave clock.
$ sudo pmc -u -b 0 'GET CURRENT_DATA_SET'
sending: GET CURRENT_DATA_SET
   80615f.fffe.08b32a-0 seq 0 RESPONSE MANAGEMENT CURRENT_DATA_SET
      stepsRemoved     1
      offsetFromMaster -15.0
      meanPathDelay    476.0

1.7. Other

A few useful commands (not relevant to linuxptp) are given below:
$ lspci | grep -i i350                # check if a BlueLAN CNA is detected by the host system
02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Fiber Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation I350 Gigabit Fiber Network Connection (rev 01)

$ ifconfig                            # list available network interfaces
enp2s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.38.124  netmask 255.255.0.0  broadcast 169.254.255.255
        inet6 fe80::8261:5fff:fe08:b32a  prefixlen 64  scopeid 0x20<link>
        ether 80:61:5f:08:b3:2a  txqueuelen 1000  (Ethernet)
        RX packets 19031  bytes 1406692 (1.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27500  bytes 5641200 (5.6 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0x72300000-723fffff

enp2s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.131.24  netmask 255.255.0.0  broadcast 169.254.255.255
        inet6 fe80::8261:5fff:fe08:b32b  prefixlen 64  scopeid 0x20<link>
        ether 80:61:5f:08:b3:2b  txqueuelen 1000  (Ethernet)
        RX packets 14091  bytes 2684066 (2.6 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 39824  bytes 6479204 (6.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0x72200000-722fffff

$ ls /sys/class/ptp/                  # list available PHCs
ptp0  ptp1  ptp2

$ cat /sys/class/ptp/ptp1/clock_name  # get the name of a PHC device
80615f08b32a                          # returns a value same as MAC address of the associated network interface

2. Run an OC and BC with linuxptp

Here, a host system will operate as an OC and BC by using linuxptp.

Figure 2.1 depicts the structure of the demo OC and BC clocks.

Figure 2.1. Demo PTP clocks with Linuxptp

In each case, a WR switch will be used as a BC. It will be labeled as a parent BC to distinguish it from a BC running on the host system. By default, its wri1 interface is in the slave mode, all other WR interfaces are in the master mode. Itself it can run in the free-running (FR) mode.

2.1. Running an ordinary clock (OC)

Any network interface of the BlueLAN CNA can be used: they have to be connected to the master mode ports of the WR switch. In order to run an OC a single instance of ptp4l has to be started with one of the network interfaces in the slave mode. Of course, another instance of phc2sys is required to synchronize the host system time to PHC on chosen network interface.
$ sudo ptp4l -i enp2s0f0 -E -2 -s --step_threshold=0.00002 -m
(optional) $ sudo phc2sys -s enp2s0f0 -w -S 0.00002         # synchronize the local system time

Look at hier for an example.

2.2. Run a boundary clock (BC)

BC provides multiple clocks: typically it is a switch with one slave clock at one network interface and master clocks at other network interfaces. Therefore, both network interfaces of the BlueLAN CNA are used as follows:
  • PTP slave clock will use the first network interface
  • PTP master clock will use the second network interface

BC will be created in two ways:
  • manual: each PTP clock is created by a separate instance of ptp4l
  • automatic: PTP clocks are defined in an external configuration file for ptp4l

For all demo BCs following options are used:
  • hardware timestamping
  • L2 (802.3) transport
  • end-to-end delay mechanism
  • clock stepping threshold is 20 us (step_threshold=0.00002)
  • poll timeout for TX timestamp is 30 ms (tx_timestamp_timeout=30)

Run a BC manually

In order to run a BC in the host system, at least two instances of ptp4l must run. It is achieved by running ptp4l with each network interface of the BlueLAN CNA and synchronize both instances with phc2sys:
  • launch ptp4l as a slave clock with the first network interface (enp2s0f0), which is connected to the master mode port of the WR switch (parent BC)
  • start ptp4l as a master clock with the second network interface (enp2s0f1)
  • start phc2sys to synchronize these master and slave clocks in our demo BC

All instances of ptp4l and phc2sys can be launched in given order:
  • start phc2sys to synchronize the master and slave clocks (source reference is PHC on enp2s0f0)
  • start ptp4l as a slave clock with the enp2s0f0 interface
  • start ptp4l as a master clock with the enp2s0f1 interface

Figure 2.2 depicts the instances of ptp4l and phc2sys running in the demo BC.

Figure 2.2. Instances of ptp4l and phc2sys

As an example, all necessary commands for running a BC manually are given below (invoke them on separate terminal):
$ sudo phc2sys -s enp2s0f0 -c enp2s0f1 -w -m -S 0.00002
phc2sys[9220.219]: Waiting for ptp4l...
...
phc2sys[9281.288]: enp2s0f1 phc offset    231222 s0 freq  +30400 delay 235417
phc2sys[9282.290]: enp2s0f1 phc offset    230411 s1 freq  +29591 delay 234185
phc2sys[9283.292]: enp2s0f1 phc offset      3199 s2 freq  +32790 delay 233993
phc2sys[9284.294]: enp2s0f1 phc offset        57 s2 freq  +30608 delay 235545
phc2sys[9285.295]: enp2s0f1 phc offset       188 s2 freq  +30756 delay 236120
phc2sys[9286.297]: enp2s0f1 phc offset       591 s2 freq  +31215 delay 234921
phc2sys[9287.299]: enp2s0f1 phc offset      2155 s2 freq  +32957 delay 234697
...

$ sudo ptp4l -i enp2s0f0 -E -2 -s -m --step_threshold=0.00002 --tx_timestamp_timeout=30   # start the slave clock
ptp4l[9236.618]: selected /dev/ptp0 as PTP clock
ptp4l[9236.667]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[9236.667]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[9238.144]: port 1: new foreign master 70b3d5.fffe.91e37e-2
ptp4l[9242.886]: selected best master clock 70b3d5.fffe.91e37e
ptp4l[9242.886]: foreign master not using PTP timescale
ptp4l[9242.886]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[9245.227]: master offset     -12347 s0 freq  +31288 path delay       478
ptp4l[9246.419]: master offset     -12427 s2 freq  +31221 path delay       478
ptp4l[9246.419]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[9247.586]: master offset     -12437 s2 freq  +18784 path delay       478
ptp4l[9248.446]: master offset      -1130 s2 freq  +26360 path delay      -138
ptp4l[9249.398]: master offset       4106 s2 freq  +31257 path delay      -750
...
ptp4l[9257.812]: master offset        -36 s2 freq  +31212 path delay       479
ptp4l[9258.938]: master offset        -58 s2 freq  +31180 path delay       479
ptp4l[9260.004]: master offset        -49 s2 freq  +31171 path delay       479
ptp4l[9261.113]: master offset        -32 s2 freq  +31174 path delay       479
...

$ sudo ptp4l -i enp2s0f1 -E -2 -m --step_threshold=0.00002 --tx_timestamp_timeout=30    # start the master clock
[sudo] password for eochirs:
ptp4l[9272.405]: selected /dev/ptp1 as PTP clock
ptp4l[9272.447]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[9272.447]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[9279.644]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[9279.644]: selected local clock 80615f.fffe.08b32b as best master
ptp4l[9279.644]: port 1: assuming the grand master role

Run a BC automatically

In this setup ptp4l uses an external configuration file, which includes network interfaces, their clock roles and all other necessary options (like as 'boundary_clock_jbod'). Configuration files are available for both versions of ptp4l: v3.1.1 and v1.9.2 It also requires that phc2sys must be run with the '-a' option, so that clock roles can be determined automatically from the active ptp4l process.

Important ptp4l options that are considered for BC:
  • boundary_clock_jbod=1 (must be enabled)
  • clock_type=BC (only for v3.1.1)
  • masterOnly=1 for enp2s0f1 (only for v3.1.1)

$ sudo phc2sys -a -m -S 0.00002
phc2sys[9631.954]: Waiting for ptp4l...
...
phc2sys[9660.985]: reconfiguring after port state change
phc2sys[9660.985]: selecting enp2s0f1 for synchronization
phc2sys[9660.985]: selecting enp2s0f0 as the master clock
phc2sys[9660.986]: enp2s0f1 phc offset   -161497 s0 freq  +33611 delay 234824
phc2sys[9661.988]: enp2s0f1 phc offset   -162427 s1 freq  +32682 delay 236008
phc2sys[9662.990]: enp2s0f1 phc offset     -4930 s2 freq  +27752 delay 237064
phc2sys[9663.992]: enp2s0f1 phc offset     -1292 s2 freq  +29911 delay 234666
phc2sys[9664.994]: enp2s0f1 phc offset      1121 s2 freq  +31937 delay 235433
phc2sys[9665.996]: enp2s0f1 phc offset     -1923 s2 freq  +29229 delay 236937
...

$ sudo ptp4l -f bluelan_bc_v311.conf -m                         # start the BC (summary_interval=32 seconds)
ptp4l[9659.654]: selected /dev/ptp0 as PTP clock
ptp4l[9659.654]: port 2: just a bunch of devices
ptp4l[9659.731]: port 1: INITIALIZING to SLAVE on INIT_COMPLETE
ptp4l[9659.787]: port 2: INITIALIZING to MASTER on INIT_COMPLETE
ptp4l[9659.787]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[9660.360]: port 1: new foreign master 70b3d5.fffe.91e37e-2
ptp4l[9664.053]: selected best master clock 70b3d5.fffe.91e37e
ptp4l[9664.053]: foreign master not using PTP timescale
ptp4l[9664.053]: port 2: master state recommended in slave only mode
ptp4l[9664.053]: port 2: defaultDS.priority1 probably misconfigured
ptp4l[9680.933]: rms  454 max 1036 freq +31060 +/- 259 delay   462 +/-   5
ptp4l[9696.434]: rms   13 max   25 freq +31128 +/-  13 delay   454 +/-   1
ptp4l[9711.554]: rms   13 max   25 freq +31127 +/-  18 delay   454 +/-   1
ptp4l[9728.002]: rms   10 max   18 freq +31090 +/-   8 delay   454 +/-   1
...

2.3. Check the PTP synchronization

The checks are relevant to the demo boundary clock, which is run in Section 2.2.

2.3.1. Check the (PTP) time

PTP uses the International Atomic Time (TAI) as its timescale. With the phc_ctl tool the time of the PHC devices can be directly verified. An example below shows that the PHC clocks of the demo BC are not synchronized.
$ sudo phc_ctl /dev/ptp0 -- get        # PTP time of the slave clock in the demo BC
phc_ctl[14405.321]: clock time is 1587506971.943044312 or Wed Apr 22 00:09:31 2020

$ sudo phc_ctl /dev/ptp1 -- get        # PTP time of the master clock in the demo BC
phc_ctl[14408.640]: clock time is 1631103584.010256074 or Wed Sep  8 14:19:44 2021

Above failure can be easily fixed by running an instance of phc2sys that synchronizes both clocks. All three commands below are equal:
$ sudo phc2sys -s /dev/ptp0 -c /dev/ptp1 -w -m  # specify the PHC devices
$ sudo phc2sys -s enp2s0f0 -c enp2s0f1 -w -m    # or specify the network interfaces
$ sudo phc2sys -s /dev/ptp0 -c enp2s0f1 -w -m   # or mix them

If a TR is used as an OC (connected to the master clock of the demo BC), then PTP time on a parent BC (WR switch) and OC (TR) can be examined. Both of them (WR switch and TR) provide CLI tools to monitor the WR PTP synchronization, which show the TAI time.

On a host system with the target TR (dev/wbm0), use eb-console:
$ eb-console dev/wbm0                # start console for the target TR
Connected to uart at address 60500   # press 'Enter' to get the prompt
wrc# gui                             # invoke the 'gui' command to display the WR PTP core monitor

    TAI Time:                  Tue, Apr 21, 2020, 22:21:34

    Link status:
    wru1: Link up   (RX: 41407, TX: 35765) IPv4: BOOTP running
    Mode: WR Off

    PTP status: slave

    Synchronization status:

    Clock offset:                    -66285 ns
    One-way delay averaged:               0 ns
    Observed drift:                     255 ns

On the WR switch, launch wr_mon (demo BC is connected to wri6 port):
nwt0019m66#wr_mon   # invoke the 'wr_mon' command to display the WR synchronization monitor

    WR Switch Sync Monitor WP3a-wrpc_fixes-24-gf313343b [q = quit]

    WR time (TAI)    : 2020-04-21 22:21:32.005181   Leap seconds:  37
    Switch time (UTC): 2020-04-21 22:20:55.005221   TAI-UTC     : +36.999960
    TimingMode: FR    PLL locking state: LOCKED

Do not wonder if there is a small difference between TAI time. Each tool uses the different interface and layout to show the synchronization status, therefore, the status information are displayed with different delays.

Adjust the WR (TAI) time on the WR switch and check that the WR time of the target TR is changed too.
nwt0019m66#wr_date get             # get the actual WR (TAI) time
1587507794.898556304 TAI
2020-04-21 22:23:14.898556304 TAI
2020-04-21 22:22:37.898556304 UTC
nwt0019m66#date -s 14:00           # change the system time
Tue Apr 21 14:00:00 UTC 2020
nwt0019m66#wr_date set host        # adjust the WR time to the system time
nwt0019m66#wr_date get
1587477663.526469712 TAI
2020-04-21 14:01:03.526469712 TAI
2020-04-21 14:00:26.526469712 UTC

If time of the target TR will not be adjusted directly, then use '--step_threshold=' option with the ptp4l instance for the slave clock. Also consider the '-S ' option for the phc2sys instance.

This option will adjust the time by stepping the clock instead of changing the frequency. Use clock stepping carefully, specially for the real-time systems. All timers that are set to expire between new time and the current time will fire once the time is adjusted.

2.3.2. Synchronization accuracy

2.3.2.1. Log output analysis

Example log messages of ptp4l and phc2sys are shown in the Listing 2.1. It shows the summary statistics in the 32-second interval. The statistics include the root-mean-square (rms) and maximum absolute of the offset between the master and slave clocks (nanoseconds), the frequency offset mean and its standard deviation (parts/billion), and path delay mean and its standard deviation (nanoseconds).

Listing 2.1. Log output of ptp4l and phc2sys (summary statistics are taken in 32-second interval)
$ sudo ptp4l -i enp2s0f0 -E -2 -s -m --step_threshold=0.00002 --summary_interval 5
ptp4l[13102.436]: selected /dev/ptp0 as PTP clock
ptp4l[13102.491]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[13102.491]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[13103.004]: port 1: new foreign master 70b3d5.fffe.91e37e-2
ptp4l[13106.594]: selected best master clock 70b3d5.fffe.91e37e
ptp4l[13106.594]: foreign master not using PTP timescale
ptp4l[13106.594]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[13108.845]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[13139.667]: rms 1066 max 3368 freq +30921 +/- 631 delay   466 +/-   9
ptp4l[13170.775]: rms    9 max   20 freq +31028 +/-  11 delay   455 +/-   1
ptp4l[13202.494]: rms   12 max   31 freq +31021 +/-  17 delay   456 +/-   0
ptp4l[13234.972]: rms    8 max   26 freq +31007 +/-  10 delay   455 +/-   1
ptp4l[13267.485]: rms   12 max   28 freq +30984 +/-  13 delay   457 +/-   1
ptp4l[13299.381]: rms    9 max   24 freq +31004 +/-  10 delay   456 +/-   1
...

$ sudo phc2sys -s enp2s0f0 -c enp2s0f1 -w -m -S 0.00002 -u 32
phc2sys[13108.895]: Waiting for ptp4l
...
phc2sys[13140.957]: enp2s0f1 rms 14245 max 57241 freq +31022 +/- 2303 delay 235010 +/- 2644
phc2sys[13173.019]: enp2s0f1 rms 3887 max 10767 freq +31070 +/- 3459 delay 233107 +/- 4320
phc2sys[13205.081]: enp2s0f1 rms 2862 max 7610 freq +30968 +/- 2491 delay 234898 +/- 3350
phc2sys[13237.143]: enp2s0f1 rms 3135 max 11065 freq +31279 +/- 2832 delay 233998 +/- 3734
phc2sys[13269.205]: enp2s0f1 rms 2915 max 11156 freq +30714 +/- 2440 delay 234247 +/- 3370
phc2sys[13301.267]: enp2s0f1 rms 2095 max 7465 freq +30959 +/- 1819 delay 234849 +/- 1877
phc2sys[13333.328]: enp2s0f1 rms 1714 max 5573 freq +31076 +/- 1475 delay 235021 +/- 1159
phc2sys[13365.390]: enp2s0f1 rms 3178 max 8554 freq +30973 +/- 2757 delay 233854 +/- 2956
...

2.3.2.2. (Optional) Measure the delay between the pulse-per-second (PPS) signals

Since WR switch and the target TR can generate the PPS signal, measuring the delay between the generated PPS signals with an oscilloscope would be sufficient for the accuracy analysis.

Short version: the result of measurement of the delay between PPS signals generated by the parent BC (WR switch) and OC (TR) is shown in Table 2.1 and Table 2.2.

Long version:

Use saftlib tools to generate PPS signal on the target TR device. First, check saftd and make sure that the target TR is attached to the saftd.
$ saft-ctl bla -j                    # display attached TR devices
$ sudo saftd tr0:dev/wbm0            # start saftd with the target TR device, if saftd is not started at all
$ sudo saft-ctl attach tr0:dev/wbm0  # or attach the local TR device (dev/wbm0), if it is not attached

Assume the target TR device is attached to saftd as tr0 and ready for usage. Then, use saft-pps-gen to start the PPS generation on the target TR device:
$ saft-pps-gen tr0 -s                    # starts the PPS generation
ECA configuration done for 11 IO(s)!
Next pulse at: 0x16a2d5dc041be800 -> 2021-09-08 11:39:16.000000000
Next pulse at: 0x16a2d5dc3fb6b200 -> 2021-09-08 11:39:17.000000000
Next pulse at: 0x16a2d5dc7b517c00 -> 2021-09-08 11:39:18.000000000
...

The PPS signal is output at all available output pins. If PPS signal generation fails and the saftlib tool returns "Timing Receiver is NOT locked!", then try it again after rebooting the target TR (fpgareset) with eb-reset.

For the WR switch, the PPS signal is output at the SMC connector labeled as 'PPS OUT'. Make sure that the PPS generation is enabled in the switch configuration:
# wrs_menuconfig
  PPS generation ---> Force PPS for tersting purpose # this option must be checked

Table 2.1. Delays between PPS signals generated by WRS and SCU3

statistics manual BC auto BC
mean 46,14 us -0,42 us 20,31 us 12,81 us -0,25 us 6,55 us
min -71,17 us -18,14 us -4,48 us -3,54 us -8,68 us -5,79 us
max 144,66 us 9,72 us 67,06 us 70,85 us 9,16 us 26,08 us
sdev 43,72 us 5,25 us 17,94 us 14,33 us 3,13 us 6,66 us
counts ~1K ~1K ~1,1K ~1,12K ~1K ~1K

Table 2.2. Delays between PPS signals generated by WRS and Pexaria5

statistics manual BC auto BC
mean 11,18 us 0,68 us 6,17 us -5 us -0,07 us -5,73 us
min -18,28 us -2,85 us -23,24 us -21,85 us -4,85 us -22,04 us
max 28,83 us 3,46 us 0,9 us 1,48 us 2,25 us 1,58 us
sdev 8,78 us 1,1 us 5,3 us 5,14 us 1,01 us 5,05 us
counts ~1K ~1K ~1K ~1K ~1K ~1K

Table 2.3. Delays between PPS signals generated by WRS and SCU3 (directly synchronized to WRS)

statistics optical cable UTP Cat6 cable
mean -86,05 ns -
min -88,97 ns -
max -84,29 ns -
sdev 1,352 ns -
counts ~1K -

A screenshot of measurement is available in this file as an example: last measurement in Table 2.2 (signal on channel 1 (C1, olive) is PPS from WRS, signal on channel 2 (C2, pink) is PPS from TR)

3. External sources

  • [1] Configuring PTP using ptp4l, link
  • [2] Linux PTP using PHC adjust phase quick start guide, link
  • [3] PTP on Linux ~ Introduction to linuxptp, link
  • [4] Synchronizing time with Linuxptp, link

-- EnkhboldOchirsuren - 11 Nov 2022
I Attachment Action Size DateSorted ascending Who Comment
oc_bc_linuxptp.pngpng oc_bc_linuxptp.png manage 15 K 11 Nov 2022 - 15:40 EnkhboldOchirsuren Demo PTP clocks with Linuxptp
pps_wrs_scu2.pngpng pps_wrs_scu2.png manage 22 K 11 Nov 2022 - 15:42 EnkhboldOchirsuren Measurement #2
pps_wrs_scu5.pngpng pps_wrs_scu5.png manage 22 K 11 Nov 2022 - 15:43 EnkhboldOchirsuren Measurement #4
pps_wrs_pexaria5.pngpng pps_wrs_pexaria5.png manage 22 K 17 Nov 2022 - 16:10 EnkhboldOchirsuren PPS generation with WRS and Pexaria5
bluelan_bc_v311.confconf bluelan_bc_v311.conf manage 2 K 17 Nov 2022 - 16:13 EnkhboldOchirsuren BC configuration file for ptp4l v3.1.1 (BlueLAN CNA i350-f2 as NIC)
bluelan_bc.confconf bluelan_bc.conf manage 1 K 17 Nov 2022 - 16:17 EnkhboldOchirsuren BC configuration file for ptp4l v1.9.2 (BlueLAN CNA i350-f2 as NIC)
bluelan_enp2s0f0_slave.confconf bluelan_enp2s0f0_slave.conf manage 2 K 17 Nov 2022 - 16:22 EnkhboldOchirsuren configuration file for ptp4l v3.1.1 (enp2s0f0 as PTP slave clock)
bluelan_enp2s0f1_master.confconf bluelan_enp2s0f1_master.conf manage 2 K 17 Nov 2022 - 16:23 EnkhboldOchirsuren configuration file for ptp4l v3.1.1 (enp2s0f1 as PTP master clock)
instances_ptp4l_phc2sys.pngpng instances_ptp4l_phc2sys.png manage 23 K 17 Nov 2022 - 16:24 EnkhboldOchirsuren Instances of ptp4l and phc2sys
linuxptp_clocks.pdfpdf linuxptp_clocks.pdf manage 497 K 16 Dec 2022 - 16:19 EnkhboldOchirsuren Presentation "PTP OC and BC with Linuxptp"
Topic revision: r5 - 16 Dec 2022, EnkhboldOchirsuren
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback