General Machine Timing System at GSI and FAIR
The FAIR facility involves a long chain of accelerators which need to be tightly synchronized. This is achieved by the General Machine Timing (GMT) system, a distributed event generation system based on the notion of time. Time synchronization is achieved by using White Rabbit (WR)
, a fully deterministic Ethernet-based field bus for clock transfer and synchronization. The key components of the GMT are a so-called Data Master (DM) that schedules actions by broadcasting messages, a WR network and Timing Receiver (TR) nodes executing machine relevant actions on time.
The primary tasks of the timing system are the following.
- Time-Synchronization of ~2000 - 3000 nodes with sub-ns accuracy over fiber lengths of up to 2 km.
- Distribution of TAI counters with ns accuracy.
- Generation of timing events for synchronization of equipment.
The aim of this timing web is to provide an overview on the timing system and its relationship to other systems. Moreover, it is the central point for documentation. Please note, that the timing system is still under development.
Scheduled downtime of the GMT production system as agreed with operations
are marked with bold
Shutdown works by other systems may affect timing system operation and are listed as 'informative' (please don't blame us!).
- 26-30 July 2021
- informative: work at computing network in building BR3; network interruptions are expected
- informative: system updates on central nodes of ACC; some features of the GMT might be unavailable
- 2-6 August 2021, maintenance works, timing system not available
- 10 August 2021, informative: changes to network connectivity core-it, global network interruption (see here)
- 12 August 2021, informative: local interruptions ACC network (see here)
- 17 August 2021, informative: reconfiguration embedded systems, reboot of all scu, supermicro, micrioc required (see here)
- 24 Augsut 2021, informative: reconfiguration acc network, systems not picking up changes via dhcp require reconfiguration and reboot (see here)
- 13-15 September 2021, informative: work at 'Mittelspannungsanlage', black-out in major parts of the facility: building BG ...
- shutdown of timing system on 10 September during the afternoon
- restart of timing system on 16 September
- 18-22 October (date tentative), informative: no cooling water in building BG; impact on timing system operation is possible
- 25-29 October 2021, informative: maintenance ACC network, impact on timing system operation is expected
- 1-5 November 2021, maintenance works, timing system not available
- 8-12 November 2021, informative: SCU, FESA, LSA & APP updates
- 15-19 November 2021, informative: SCU, FESA, LSA, Model updates
- 22 November - 3 December 2021, informative: dry-run
- 15-16 December 2021, maintenance works, timing system not available
- 10-11 January 2022, maintenance works, timing system not available
Schedule for the next weeks based on shutdown planning
. For an overview, click here
- 17-Jul-2021: the 2021 beam time ends today; we had almost 6 months of successful beam operation. We had ...
- very smooth operation with only very few on-call services
- a few failures of the Data Master; this issue is understood and we had no failures in the past two months
- a few failures in the White Rabbit network; this issue is understood and has already been fixed (click)
- a few failures due to unexpected power-cuts
- 16-Jun-2021: some White Rabbit switches lost synchronisation; fixed
- 12-May-2021: Data Master restarted
- first beam transfer between ring machines using White Rabbit based kicker timing
- 'bunch-to-coasting-beam' SIS18 -> ESR
Screenshot by our colleagues from Ring RF. Bunches from the synchrotron SIS18 (magenta) are transferred into the storage ring ESR (orange)
- 03-Mai-2021: Epoch of White Rabbit PTP is back to correct value (UTC-offset 37s)
- 27-Apr-2021: many White Rabbit Switches needed a restart; White Rabbit time is off by one second
- 21-Apr-2021: Data Master restarted
- 19-Apr-2021: restart of a few White Rabbit switches after a power-cut in building BG 1
- 08-Apr-2021: restart of a few White Rabbit switches after a power-cut in building BG 1
- 30-Mar-2021: Data Master restarted
- 09-Mar-2021: White Rabbit grandmaster lost synchronization to GPSDO; reboot of all WRS
- 04-Mar-2021: beam in CRYRING; beam operation in the full facility
- 02-Mar-2021: Data Master restarted
- 28-Feb-2021: in February ...
- ... we had regular operation with SIS18 and ESR and a multitude of experiments
- ... no relevant issues with timing system
- ... but we participate in a bug hunt related to ramped devices
- 07-Feb-2021: beam extracted from SIS18!
Screenshot of equipment by the Beam Instrumentation group in the main control room. green: bunches in SIS18, yellow: differential probe signal from kicker magnet, red: four bunches extracted from SIS and on the way to HHD. The signals pass cables of different length and are not aligned to each other.
- 01-Feb-2021: commissioning of facility starts, on-call service
- 19-Jan-2021: maintenance timing system ends
- 18-Jan-2021: maintenance timing system starts
- updated all WRS to new firmware
- 15-Jan-2021: dry-run ends
- 11-Jan-2021: dry-run starts
- 7 February 2021