Concept Discussions for the Emergency System

This wiki page documents the discussions that have taken place within the Injector Controls Upgrade Working Group (ICU WG) while trying to come up with concept ideas for the 2024 emergency system and 2025 production system milestones.

1 Motivation

The idea is to find out

• which operation scenarios we absolutely have to support for the Emergency System to potentially be used for the 2025 beamtime
• which scenarios we can support on top of that while still keeping in mind that, ideally, all effort we spend on that should also be viable for the Production System to be definitely used for the 2026 beamtime

From that, we should hopefully be able to develop

• a technical concept for the Emergency System
• a project plan to be carried out in 2024

2 Assumptions and Preconditions

• According to the Pulszentrale documentation, Virtual Accelerators 0 to 13 are available for operating. Currently, we start counting at 1 for the Sequence Index in the new control system ( Why was that again? Did it have something to do with the Digitizer/DAQ interface?), so there would be a maximum of 13 Virtual Accelerators / chains that operators can make resident / use.
• There are devices in several sections of UNILAC that need warming pulses. This includes at least RF cavities in HSI, Poststripper and transfer channel
• For the emergency system, the request mechanism between UNILAC and SIS18 will be kept as it is. The UNILAC Data Master (or a related component) will (to the extent necessary) mimic the UNILAC Pulszentrale's behavior in that regard.
• Non-interruptive timing changes will be implemented to ensure thermal stability of UNILAC and the sources (as documented in the milestone overview). The question whether this new feature should be rolled out for the other machines as well was briefly discussed (would be beneficial as a step towards implementing transactions for setting changes), but postponed until further notice since it would be a rather nice-to-have, low-priority feature.

Decide whether or not implementing the feature "Non-interruptive Pattern Group changes" is required for the Emergency System or not.

Clarify with Mathias and Dietrich whether the assumption is valid that we can keep the request mechanism between UNILAC and SIS18 for the Emergency System compatible with how it works now with the Pulszentrale.

3 Operation Scenarios

3.1 Scenario A: Beam to SIS18

For producing a single beam going through the transfer channel to SIS18, a single chain is needed. However, this chain would have to include:

• Settings and timing for producing the beam
• Settings and timing for warming pulses between beam shots

Consequently, for the Emergency System at least, a chain would not be roughly 20ms long, but a multiple of that, depending on how long it would be supposed to generate warming pulses until it is repeated or a different chain is executed. Is it still necessary to find out how picky devices are about their warming pulse frequencies? Or is that covered because all expected requirements can be supported?

This scenario would already allow for multiple beam destinations behind SIS18 (as from there, beamlines are controlled by the corresponding SIS18 Patterns), assuming that all of these experiments can be served with the same UNILAC beam. However, they would not be receiving at the next possible moment after raising their request, as is it was the case with the Pulszentrale, which dynamically adjusted the Virtual Accelerator schedule according to a priority system. Instead, since the schedule is now fixed, they would receive beam at the time when it is their turn according to the predetermined schedule.

For the emergency system, it will not be necessary to adapt the UNILAC schedule depending on whether beam is actually being requested by SIS18 or not. UNILAC will produce beam regardless every time it is executed according to the schedule. Still, the TK9 chopper ( Correct? What's the proper nomenclature?) will only let the beam pass into SIS18 when the corresponding chain there is also executed.

A chain spanning a longer period of time might need to be synchronized with the mains current frequency while it is running. To be discussed with Mathias if/how this is possible.

Find out how transfer channel preparation can be commenced. There probably should be settings and a preparation event, but the issue is that both has to presumably be sent before the actual chain for producing the beam even begins (how long in advance again?). What was discussed is whether it could "simply" be prepared in every chain, but it's not clear how that can work for potentially different settings.

3.2 Scenario B: Beam to UNILAC experiments

Think of a way to stop the beam from being produced when a UNILAC chain has been disabled by an operator (but is still running to ensure thermal stability). According to Peter, there is a hardware signal preventing the source chopper from opening when the beam is not requested by an experimentor, but that signal/button or similar may not be available to operators.

4 Handling Sources

4.1 Separate or Integrated Chains

4.2 Setting Virtual Accelerators in Timing Events Correctly

Concerning the Virtual Accelerator / Sequence to be used for the sources, the UNILAC Pulszentrale documentation states:

Da über den Timing-Bus in allen Nicht-Datenevents immer eine Nummer eines virtuellen Beschleunigers übertragen wird, muss auch immer eine definiert sein. Es wird daher intern der virtuelle Beschleuniger Nr. 14 verwendet. Diese Nummer tritt aber nach aussen hin nicht in Erscheinung.

However, this apparently does not mean that timing events being played for the sources are generally marked as being VirtAcc 14. Quite contrary, according to what Raphael was able to sniff via the White Rabbit UNILAC PZ, these timing events actually carry the VirtAcc/Sequence of the matching following UNILAC "Betriebsbeschleuniger". So the assumption is that, if a source serves multiple destinations / is used with multiple VirtAccs that follow that source's "Timing-Abschnitt", it is always the VirtAcc/Sequence of the "Betriebsbeschleuniger" that's specified on the timing events for the source as well, although it cannot have different settings for different VirtAccs. According to Peter, this is because for beam instrumentation purposes, one has to be able to determine which source pulse belonged to which beam pulse in the following "Timing-Abschnitt".

In the implementation idea sketched in the sections below, this issue is solved by having different beam processes for each source, containing the same Timing Events, but specifying different Virtual Accelerator numbers depending on which "Betriebsbeschleuniger" the source is currently serving. Accordingly, this should be fine for the Emergency System. However, a suitable solution still has to be found for the production system where the working hypothesis currently is that the source's timing settings shall be stored within the (separate) source chain as well (instead of in a "Special Timing Chain/Pattern").

4.3 Visualization

The diagram below (adapted from the whiteboard sketch from the meeting on 2023-11-14) tries to visualize the two aspects mentioned above for further discussion.

4 Handling Non-multiplexed Devices

Discussion went somewhat back and forth between how to handle timing, sources and non-multiplexed devices. The following overview diagram of non-multiplexed devices has been created at a relatively late state of the discussion, but in hindsight, it might have been useful from the very beginning, so it shall now serve as the introduction to the topic.

4.1 Glossary

Non-multiplexed: Devices that only have one set value (most of the time these devices are not beam property dependent)
Multiplexed: Devices that have set values per virtual accelerator / chain (most ot the time beam property dependent)

4.2 Overview

What the diagram shows is a list of all non-multiplexed devices in UNILAC, separated by Particle Transfer (except for those in the experimental areas where Particle Transfers are by "branches" M, X, Y and Z). DIscussing this diagram, the following points became clear:

• There are quite a lot of non-multiplexed devices in UNILA (i.e. more than expected by some participants of the working group)
• Non-multiplexed devices reside not only in the source areas, but are spread over most of HLI
• Multiplexed and non-multiplexed devices are not guaranteed to be separated Particle Transfer-wise, as the UN chopper (GUN3BC1L) is multiplexed at least timing-wise. Put more precisely: Its voltage setting is multiplexed, but always supplied with identical values for all Virtual Accelerators (VA). On the other hand, chopper windows are different between VA.

5.3.5.2 Categories of Non-multiplexed Devices

All of the following are non-multiplexed, but reside in different PTs (and may be non-multiplexed for different reasons?). Also, we may decide to handle some of these categories differently than others.

Category	Settings Managed by	Resides in Context(s)
Source devices	Ion Source App	none
Other devices in source PTs	LSA	Non-Multiplexed
Non-multiplexed devices in HLI and UCW	LSA	Non-multiplexed
MAZ-dependent devices	LSA	Non-multiplexed
Experimental area magnets UM/UX/UY/UZ	LSA	Multiplexed

Ask Susi about the non-multiplexed devices in HLI and UCW. What do their settings depend on? Only beam parameters or something else, too?

The lists of magnets in the experimental areas are colored in light gray to reflect that settings-wise, they shall not be considered non-multiplexed because their settings depend on multiplexed settings. As an example: To calculate the current setting for one of the magnets in the X branch, one needs to consider the beam's rigidity after acceleration within the Alvarez, which is a multiplexed setting ( correct?). However, calculating non-multiplexed settings from multiplexed settings is not possible in LSA (at the time of writing, no ideas considered sensible for implementation have been conceived and at least some LSA developers consider it a bad idea in general). In practice though, this is not considered a problem so far. The magnets are considered "slow" in the sense that it is not possible use them to serve multiple experiments and/or beams at the same time, so only one experiment is foreseen in each branch by convention during beamtime planning. Thus, simply using a "last value supplied wins" strategy is sufficient at this point for the expected behavior. Similar considerations seem to have been made for the legacy control system as well, as the experimental area magnets use a variation of the "MD" equipment model that is functionally non-multiplexed, but allows any Virtual Accelerator number to be specified when supplying settings (using "last value supplied wins" as well).

These findings heavily influenced the final assessment of the options described in the sections below.

5.1 Option A: "Only Multiplexed Chains"

5.1.1 Synopsis

Pretend that everything is multiplexed (as we are currently). Last setting sent wins (for now), reference multiplexing could be used once legacy front-end software is replaced with FESA.

5.1.2 Requirements

Non-multiplexed devices need to accept any selector (or we must implement a new way of handling conversion)

5.1.3 Pro

• No non-multiplexed context needed (no risk associated with introducing a new concept into the control system)
• All settings for the beam in one context (instead of needing multiple ones, see options B and C)
• No issues with optics

5.1.4 Con

• If a reliable and/or convenient way of ensuring that settings are consistent between beams is required, it has to be invented. This might be a show stopper.
• Everything in LSA will be forced to be multiplexed, you can not immediatly "see" a differences between multiplexed and non-multiplexed parameters (is_multiplexed=true for everything). Another means of distinction would have to be invented if necessary.

5.2 Option B: "Source Chains"

5.2.1 Synopsis

Have a separate (multiplexed/normal) chain from the source up until the last non-multiplexed device (that is supposed to be treated as non-multiplexed)

5.2.2 Requirements

Clear particle transfer-wise separation between multiplexed and non-multiplexed devices (seems not to be the case at least for timing for GUN3BC1L)

5.2.3 Pro

• No non-multiplexed context needed

5.2.4 Con

• Requires manual propagation for certain trims in the source chain
• Issues with optics handling because of separated chains
• Unclear how to handle sending the Virtual Accelerator matching the "Betriebsbeschleuniger" in the source chain
• For one beam, now (at least) two contexts would be relevant to get the full picture for the settings. This would also be a requirement for the UIs

5.2.5 Notes

Raphael mentioned that having overlapping chains (source/accelerator) that are differentiated by the settings that they contain might be considered a variation of the "source chains" idea.

5.2.6 Details

Susi raised the issue that, apart from source devices that are "traditionally" handled in the Ion Source App (and would presumably not have their settings stored in LSA), there are also a number of magnets in the source's "Timingabschnitte" / Timing Groups / Particle Transfers that are non-multiplexed, but will be needing settings to be provided by LSA because they are not handled by the Ion Source App (see e.g. this list, devices concerned are those that do not carry an uppercase "i" / "I" in their nomenclature). Peter added that both warming and beam pulses are exactly the same timing- and settings-wise.

Alternatively, if one thinks about what would happen if source timing and settings to the extent mentioned above were contained in the chain of the following "Betriebsbeschleuniger" (i.e. potentially multiple of those), the issue arises that those settings could potentially be different for the same source, which would not be desirable. In principle, it would be possible to implement a check in LSA to enforce that all settings per source are the same in all resident chains, but then, some way of trimming them consistently when changing a source setting value would still have to be invented.

So coming from this perspective, it would seem natural to have sources as separate chains that can somehow be coupled with different chains representing multiple "Betriebsbeschleuniger". This led the ICU WG members to go for the working hypothesis that sources should be separate chains, which is currently being evaluated. For the sake of establishing terms to work with for these separate chains, they will be called "source chains" from this point on, to be distinguished from "accelerator chains" (which shall begin at the point right after the source chains end). More precisely, the following Timing Groups would belong to source chains:

Source	Timing Groups	Timing Group Number	Comment
UL	UL_TO_GUH1MU2	593	Contains all devices from UL up to (excluding) GUH1MU2
UR	UR_TO_GUH1MU2	592	Contains all devices from UR up to (excluding) GUH1MU2
UN	UN_TO_GUN3BC1L	595	Contains all devices from UN up to (excluding) GUN6MU2 GUN3BC1L
	GUN3BC1L _TO_GUN6MU2	624

The assumption here is that there can be only one resident source chain at a time. Ensuring this would probably require some special handling in LSA.

Susi's assessment was that, in general, propagation of settings (including timing) should not be necessary between source and accelerator chains. However, there may be exceptions to this rule:

• Currently, the "UNILAC-Setzprogramm" (USP) propagates changes to element/isotope and charge settings from the source to all "Betriebsbeschleuniger" served by this source. The participants discussed the issue that it would not be possible to to replicate this behavior without additional measures in the new control system, as LSA does not support propagation of settings accross different chains. Adding specific code for trimming these values in "coupled" chains and/or checking that "coupled" chains match either within the Scheduling Application or the Ion Source Application would be conceivable, but specific use cases would have to be defined more precisely. However, the point was made that e.g. between the other machines (UNILAC to SIS, SIS to ESR, ESR to CRYRING), such means of propagation do not currently exist and were so far deemed unnecessary (or even unwanted). So at least for the Emergency System, the working hypothesis is that propagation/checking of element/isotope and charge can be ignored.
• Timing settings between source and accelerator chains must match with respect to source beam pulse start/end and accelerator beam pulse start/end. Achieving this could be realized via checking the corresponding Timing Events directly and/or looking at extraction/injection markers put into each chain by LSA (to be defined how these work exactly, see below). This could be done e.g. in the Scheduling Application and/or BSS (David's idea was: Have some kind of "check plug-in" mechanism for timing events in BSS that can verify timing / timing changes between chains). Whether or not such a check is necessary for the Emergency System should be discussed.
• Discuss and decide whether additional measures are necessary for the production system.

How is "coupling" of UNILAC Chains done? Extraction / injection markers? How would that work practically?

What is currently unclear is how to handle optics and Transfer Lines for the accelerator chains. Currently, Transfer Lines for UNILAC cover all the beamline from source to target (can also be transfer channel to SIS). The same is true for the optics. When splitting chains, this is not viable anymore, because the Scheduling App's optic selection logic checks that a to-be-created chain covers all Particle Transfers of the selected Transfer Line. Also, as Susi pointed out, the magnets at the beginnings of the accelerator chains would have to somehow know where the beam comes from (and how rigid it is) so that their deflection angle can be set appropriately.

In Holger's "Basic Chain Magic", there is some logic that selects specific Init Beam Processes depending on whether UL or UR is used for a UNILAC chain. To be clarified why this is necessary and how one could handle it with separate source chains.

So far, a couple of possibilities have been discussed, each coming with their specific pros and cons:

Option name	Description	Pros	Cons
Dummy Particle Transfer	Create empty (not containing devices) Particle Transfers "in parallel" to existing source Particle Transfers to specify where the beam comes from	Minor effort	Breaks mapping of Particle Transfers to Timing Groups Potentially confusing for developers/operators/modelers
Leave out source Particle Transfers	Keep Transfer Lines and optics definitions as they are, but leave out source PTs during chain creation	Transfer Lines and optics unchanged	Special handling in Scheduling App Inconsistency between UNILAC and other machines because chains do not match Transfer Lines
Operator chooses optic	UL and UR have specific optics for stand-alone operation already. There would be separate optics for the accelerator chains starting at their first component, one for each source that the beam can come from	No special handling required in Scheduling App	Operator must choose correct optic Optics and Transfer lines must be adapted Unclear how to handle selection of Init BPs depending on source
New information item	Like the previous option, but additionally, there is some item of additional information that specifies which source the beam comes from for this accelerator chain. Might be provided by the application that the operator uses to construct chains with in the future.	Better usability than previous option	Special handling required Optics and Transfer lines must be adapted

Ask Ingrid Kraus about how she handles a potentially similar problem in CRYRING's model. There, the beam could come either from ESR or CRYRING's own injector. Transfer lines seem to reflect this as well, i.e. there's "YRT1_YRT1MH2", "ESR_YRT1MH2" and "YRT1MH2_TO_CRYRING" (Note: "YRT1_YRT1MH2" seems not to exist on INT?).

Ingrid provided the follwing Information:

Am CRYRING gibt es den Merger-Magneten YRT1MH2. Für den Kickwinkel, also den Optikparameter, gibt es zwei Parameter für die beiden Injektionslinien (HICK_LOCAL und HICK_ESR), sowie den Theoriewert.

[...] [D]ie Settings [existieren] in INIT files, weil eigentlich nur ein Optikwert pro Device vorgesehen ist. Falls häufiger mehrere Parameter gebraucht werden, sollten wir das Konzept mal anschauen. Am SIS18 gibt es eine Lösung, weil da für Magnete zusätzlich eine Vorsteuerspannung gebraucht wird.

Note that it must probably be possible to "prepare" accelerator chains based on source chains to be used in the future. Or to put it another way: Operators might say "Currently we're doing argon, but I'd like to prepare the UNILAC accelerator chain for the gold beam already now, because the source will be changed to provide that tomorrow". To be discussed if that's a valid use case and how to handle it.

5.3 Option C: "Non-Multiplexed Context(s)"

5.3.1 Synopsis

Have one or more non-multiplexed chains that contain the non-multiplexed devices (that are supposed to be treated as non-multiplexed)

5.3.2 Requirements

Stay close to CERN's implementation to benefit from their work

5.3.3 Pro

Multiplexed and non-multiplexed devices clearly separated
Handling of MAZ magnets seems to fit the concept relatively nicely
Multiplexed and non-multiplexed contexts can intersect (beneficial e.g. for GUN3BC1L and VirtAcc handling at the source)

5.3.4 Con

New concept to be introduced
For one beam, now (at least) two contexts would be relevant to get the full picture for the settings. This would also be a requirement for the UIs

5.3.5 Details

5.3.5.3 Consequences

• There is no propagation between contexts
• Changes to settings in non-multiplexed contexts do not automatically trigger recalculation of settings in other contexts
• MakeRules can look up values in non-multiplexed contexts during execution
• New/changed values can be automatically picked up during MakeRule calculation
• Propagation within a non-multiplexed context works as in a multiplexed context

What about recalculation of contexts that depend on non-multiplexed settings?

In principle, it would be possible to automatically trigger recalculation of (resident) multiplexed contexts when changing settings in a non-multiplexed contexts. But that would require a considerable amount of additional concept and implementation work. For example:

• How to handle trim flags?
• How to decide which contexts to recalculate?

5.3.5.4 Options

Non-multiplexed Chains or Stand-alone Beam Processes

Having non-mux Beam Processes without chains does not seem to have any (significant) benefits. It's a bit closer to CERN's implementation (because they have that already), but establishing non-mux chains should not be that difficult anyway.

One or multiple Non-multiplexed Chains

We could have only one non-mux chain or multiple ones. But we probably don't need to decide that right now. CERN only uses one non-mux user per accelerator, so that seemed to indicate that having only one non-mux chain might be easier, but in the end, we're not attaching a single user to a chain (like CERN attaches a single user to a cycle), but we use one user for each Beam Process in a chain, so we would need multiple users for a non-multiplexed chain anyway.

User-to-Selector Conversion during Drive

• During make-resident generate one non-multiplexed user per beamprocess (e.g. .NON-MULTIPLEXED) and assign it
• During drive for Unilac map *.NON-MULTIPLEXED users to the empty selector (new Selector(""))

Benefit:

• Easier to implement special treatment for non-multiplexed users
• Consistent non-multiplexed user handling between DB and Java code (up to drive)

Drawback:

• The user (DrivableContext.getUser()) cannot be used by a client as a JAPC Selector (what about EquipState and similar applications? Do they still work for non-mux contexts then?)

User-to-Selector Conversion during Persistence (CERN Style)

• During make-resident generate one non-multiplexed user per beamprocess (e.g. .NON-MULTIPLEXED), but assign each beamprocess the empty user ""
• During context persistence persist per beamprocess the real accelerator user (map "" to .NON-MULTIPLEXED for each bp)
• During context finding map the real non-mux users back to the empty user ""

Drawback:

• Mapping magic for users in the backend is necessary

Benefit:

• No accelerator-specific selector mapper needed
• Clients can use the user as a non-mux JAPC Selector

In theory all (combinations of) these options could also be modeled using multiple non-multiplexed chains. That would add the drawback that propagation between non-multiplexed contexts is not possible, so e.g. it would not be possible to propagate settings from the source to the following particle transfers. On the other hand, it might be more according to "the LSA way of doing things" in the sense that independent things are kept separate (i.e. the sources each have their own context) and keeping settings for later use could be done in LSA chains instead of ParamModi export files.

5.3.5.5 Preferred Approach

• Non-multiplexed contexts only for UNILAC for now (not for other machines)
• One non-multiplexed beam process per particle transfer (where needed) in that chain

To be investigated and discussed:

Only one non-multiplexed chain for the whole UNILAC (for now, could be handled differently later)

5.3.5.6 Questions for Roman

• Why not use AcceleratorUser.is_multiplexed() to determine whether a user is multiplexed?
• Function based index? Does it still work for non-mux users per PT?
  • Not with that FBI. Would probably have to be dropped at GSI? More info: INCA-6459
• Which of the above options (convert user during drive or convert user during find/persist) is better?
• What do non-mux contexts look like at CERN (how long are they etc.)?
• Do you have only scalar values in non-mux contexts?

5.3.5.7 Open Issues

• We probably need BP Types and stuff to try this out (-> Holger?)
• There are no devices in source test benches to be supplied by LSA, right?
• Hopefully, all beam diagnostics devices are multiplexed, right? (-> Ask Susi)

5.4 Upwards Hierarchy / Acquire

5.4.1 Synopsis

Do it more like the legacy control system: When looking at the settings in ParamModi, acquire those from the devices, run them through an upwards hierarchy to calculate affected settings and trim them into the context at hand

5.4.2 Requirements

Unclear

5.4.3 Pro

• Might make handling inconsistent settings a bit easier sometimes
• CERN's INCA might be doing something similar

5.4.4 Con

• Highly complex
• Two separate (but consistent) hierarchies have to be maintained
• Benefits do probably not justify effort for implementation (and maintenance)

6 Implementation Idea

Update to reflect how non-multiplexed devices are handled?

For the "Master Schedule Graph" to be put together by BSS and executed by the Data Master, the idea is to use threads to execute indivdual graph fragments (Chains? Beam Processes?), if possible, in such a way that it leads towards the idea for the "final" (beamtime 2026) implementation.

Ask Mathias about the current state of thread spawning functionality in the Data Master and whether realizing this plan will be possible for the emergency system.

7 Open Issues

Ask Mathias about a UNILAC Data Master for testing in INT

Think about how to implement profile grid protection ("Profilgitterschutz")

Peter: "Ich bin mir ziemlich sicher, dass wir für 2025 einen Ersatz für die Rahmenpulsgeneratoren brauchen, die Rahmen-, Klemm- und Experimentpulse erzeugen"

According to the Pulszentrale documentation, warming pulses should always be scheduled in Virtual Accelerator 15. Must we / do we want to adhere to this? Would this mean that we introduce special handling to set the sequence to 15 for settings / timing events related to warming pulses?