Booster-Test November 2021
The so-called 'Booster Mode' shall be used to accumulate beam from multiple SIS18 injections into SIS100 at a rate of about 2.7 Hz. Here, the procedure used for multi-multi injections cannot be used. Instead, the booster mode requires beam from UNILAC to SIS18 at deterministic points in time as required by a BPC for this mode. A short term solution using the existing UNILAC Pulszentrale (UNIPZ) is called 'Booster Mode Lite', where UNILAC produces beam for SIS18 exclusively (click
). Since the UNILAC is bound to 50 Hz mains, UNILAC operation consists of cycles with 20 ms length. The main challenge with 'Booster Mode Lite' is the the drift of the 50 Hz mains frequency while a Booster Mode operation is in progress.
The Booster Mode of the accelerator was first tested with the new control for FAIR during the November 2021 Dryrun. The data presented here have been acquired using a booster pattern operated for more than 18 hours.
Raw Data are available here //.../dryRun_2021-nov/boosterTest booster_2021-11-* .
Description of Analyzed System
The GSI/FAIR GMT is an alarm based system, comprised of central message generation units and endpoints all over the accelerator facility. Current time of all endpoints is synchronized to nanoseconds by the White Rabbit (WR) timing network. The handshake between UNILAC and SIS18 is carried out over WR network by the Datamaster (DM) that is coupled to UNIPZ via a dedicated gateway called DM-UNIPZ; an overview on the setup is shown here
. The main idea behind 'Booster Mode Lite' is to separate the SIS18 operation into two tasks, injection and standard operation.
Technically, this is realized by using multiple threads by the Data Master as described here
- a main thread handles continous operation of SIS18 including all function generators that run without interruptions
- an injection thread that only handles timing of equipment relevant for the injection into SIS18
Sketch of 'Booster Mode Lite'. A main thread (large blue box) within the Data Master (DM) requests beam from UNIPZ (green box) via a gateway DM-UNIPZ (small blue box). A dedicated thread for injection (small blue box) is triggered when beam is delivered by UNILAC.
Injections from UNILAC into SIS18 are scheduled within specific 20 ms time slots. Prior to each slot
- the DM requests beam from UNIPZ via DM-UNIPZ (main thread)
- UNIPZ is supposed to deliver beam within a specific UNILAC cycle
- if beam is delivered, DM-UNIPZ triggers the injection thread
It is of prime importance that the activities by the injection thread and the main thread remain synchronized during the entire booster operation process. This is the subject of this analysis.
Description of Analysis
For the purpose of this analysis, data sets were gained from 'WR Snooper Devices' connected to the White Rabbit timing network. Timing messages are broadcast, so snooping on the message stream is possible at any place in the topology. Timing events mark the preparation and the injection of UNILAC beam into SIS18; CMD_UNI_BREQ_NOWAIT (evtno 0x162) and EVT_MB_TRIGGER (evtno 0x28) were chosen. CMD_UNI_BREQ_NOWAIT is the preparation event sent to the timing network from the main thread prior to synchronization, while EVT_MB_TRIGGER is an event that is sent by the injection thread. The difference of deadlines between these two event is calculated for each injection into SIS18 separately. Here, each injection is identified via its Sequence ID (SID). This allows a separate analysis for each SID (6,7, 8) and thus for each booster injection.
The first injection into SIS18 (dataset of SID 5) is analyzed differently, as the underlying synchronization is different to the three following injections (request of Transfer-Kanal ...). For this reason, it was plotted into its own histogram.
Top: Separate histograms for SID 6,7 and 8 with a time distribution (difference between CMD_UNI_BREQ_NOWAIT (evtno 0x162) and EVT_MB_TRIGGER (evtno 0x28)). Each histogram contains data of 28793 injections from UNILAC into SIS18. Bottom: Statistics.
The figure above shows a histogram of the time difference occurring beam request and delivery for sequence IDs 6, 7 and 8. Each SID shows a strong main peak at around 49 ms. A weaker side peak near 69 ms indicates synchronization is not perfect in all cases; here UNILAC had delivered beam not at the requested time but one cycle (20 ms) later. It is expected, that synchronization becomes more and more difficult with each subsequent injection. The main peak at 49 ms becomes broader with each subsequent injection while the second peak at 69 ms becomes more pronounced.
Statistics is interesting too.
- it is a big success all peaks for SID 6, 7 and 8 contain (almost) the same amount of data, 28793 injections; this means, that the synchronization works very well from the technical point of view; the beam request at UNIPZ, beam delivery and the start of the injection thread works very reliably
- the difference between the max and min value increases with ascending injection number; the windows defined by min and max spans several milliseconds
- the standard deviation increases for the three following SID
- for peak 1, the average standard deviation of SID 6,7 and 8 is about 307 us
- on average, 95% of all injections in the main peak are within a window of 1227 us (assuming a normal distribution with +/- 2 sigma at 95% CL)
- it is interesting to note the ratio of data in peak 2 and peak 1
- for the first booster injection (SID 6) almost 98 % of all injections happened at the UNILAC cycle
- even for the last booster injection (SID 8) still 89% of all injections were timed correctly
- on average 93 % of all injections happened at the 'same' UNILAC cycle.
For comparison, a histogram plot of SID 5 is displayed below. Note that this data set uses the time difference between the CMD_UNI_BREQ (evtno 0x160) and EVT_MB_TRIGGER (evtno 0x28) instead. This first injection is used to 'synchronize the Data Master' to UNIPZ, so the distribution is very much different. Y axis is identical to the previous plot, X axis is was scaled to show the outlying minor peaks down to 40 and up to 75 ms. As SID 5 is the first injection, the separation of peaks is not only determined by the 20 ms cycle length of UNILAC but also by the 'length' of other SIS18 patterns operated concurrently. As the other patterns are each of individual length, activating or deactivating other patterns will shift the position of the peak; remember data have been acquired for more than 18 hours.
Histograms for SID 5.
Informative: Drift UNIPZ Frequency
Frequency drift of UNIPZ starting on noon 24 November 2021.
Informative: The operation rate of UNIPZ has been extracted using log data of a different technical system WR-UNIPZ (click
). The frequency is calculated from the time difference of the start of two subsequent UNILAC cycles and logged at a one minute interval. The result is shown in the figure above. For more information on how UNIPZ is linked to 50 Hz mains please refer to section 19 of the UNIPZ documentation (click
For the first time SIS18 was operated as booster using the scheme 'Booster Mode Lite'. A successful and reliable synchronization of UNIPZ and Data Master has been achieved for more than 18 hours of continuous operation with 'dry beam'. On average, 93% of all injections can be injected at the desired UNILAC cycle. In the future this might be improved. However, better tuning of the procedure will require dedicated tools for on-line analysis.
Addendum: Timing of Booster Requests
Timing Scheme. Shown are activities at Data Master (DM), DM-UNIPZ Gateway(Gateway) and UNILAC Pulszentrale (UNIPZ) (black horizontal lines). Indicated are event deadlines (green vertical lines), start of UNILAC cycles (magenta vertical lines), UNIPZ announcements of the next
cycle (blue vertical lines) and beam delivery to SIS18 (red vertical line). Information on time differences measured by the gateway are shown on the upper left. Details see text.
The figure above shows a simplified sketch of the transfer timing from UNILAC to SIS18. Very shortly (0.5 ms ) after a start of each UNILAC cycle, the 'Superpulszentrale' of UNIPZ announces the virtual accelerator (and other information) on the internal MIL-Eventbus to the seven 'Unterpulszentralen' via an event
. Hence, UNIPZ 'thinks' at least one cycle of 20 ms in advance. A measurement by the gateway yields the following.
Optimal injection yield a time difference of 49 ms (difference deadlines of EVT_MB_TRIGGER and CMD_UNI_BREQ). As EVT_MB_TRIGGER is scheduled at the end of an UNILAC cycle (when UNILAC delivers beam), it becomes clear, that the beam request CMD_UNI_BREQ is scheduled about 9 ms prior the end (= roughly in the middle) of a preceding UNILAC cycle. The processing time in the gateway is just a few microseconds and not relevant on the time scales shown here.
The gateway receives the acknowledgement of the beam request about 14 ms after the request has been sent to UNIPZ. This is very interesting: The acknowledgement is received(!) after the decision for the next cycle has been announcement
. I (db) don't know when UNIPZ actually takes the decision on the next cycle but this observation does smell like a potential race condition. What if the decision by UNIPZ is a little bit delayed? Remember, that UNIPZ needs to do other stuff like reading states of interlocks or handling of beams for other virtual machines too! In case the booster mode request loses the race for the next possible cycle, it will be delayed by one cycle of 20 ms in which case the beam is delivered not 49 ms but 69 ms after the request.
Is there a specific need why the beam request is scheduled just 9 ms prior the end of an UNILAC cycle? If not, it is suggested to schedule it 5 ms earlier.
 Remark: the time difference of 0.5 ms is known from an independent measurements by the White Rabbit UNIPZ
Addendum: On-line Monitoring of Booster Requests with Graylog
On-line monitoring of 'Booster Timing' is straight-forward with Graylog. The main idea is observe the diagnostic information by the Datamaster UNIPZ Gateway. This requires filtering for the relevant SCU plus minor tweaking of the search string. Example for good and bad timing are shown in the figures below.
Occurences of 'good' booster timing seen by Graylog.
Occurences of 'bad' booster timing seen by Graylog.
- 17 Dec 2021