Findings related to Prototypes for Set-Set Value Comparison 2022

This page documents various findings with regard to Set-Set value comparisons (SSVC) that we found while creating prototypes in late 2021/early 2022 (see https://git.acc.gsi.de/fcc-applications/expert-beam-proseco-app-fx/ and https://git.acc.gsi.de/fcc-applications/expert-beam-proseco-monitoring-app-fx , see also ApDevSetSetValueComparisonMonitoringPrototype).

Sample HTML Reports

2022-02-08_SampleSSVC.txt

How to handle detected inconsistencies

"Unimportant"/Masked devices

Should SSVC take into account that certain devices are "not important right now" (e.g. masked in MASP)? If yes, define details.

In Raphaels opinion a device that is "not needed" should also be supplied, but with "neutral" values.

Non-reachable devices

Is it ok to treat Set values for devices that cannot be reached as incorrect/inconsistent, or should they have same kind of special handling/marking/etc.?

E.g. from 2022-02-08_SampleSSVC.txt : Exception: Failed to connect to server 'PowerSupplyRf_DU.scuxl0037' : failed to connect to 'tcp://scuxl0037:10020'

DeviceAccess: Slightly different values due to DAC resolution

E.g. from 2022-02-08_SampleSSVC.txt :

BeamProcess: SIS18_FAST_HHD_20220127_182722.C1.PLTKMH2_TO_SIS18.TRANSFER_INJECTION.1 Selector: FAIR.SELECTOR.C=6:T=499:S=4:P=11

Parameter Name	GS12ME1I/CLVOLTLS#cl1voltl	GS12ME1I/VOLTAGES#voltages
LSA Setting	(float:1) -> 372.5776	(float:1) -> 21494.863
Device Setting	(float:1) -> 372.7073	(float:1) -> 21497.238

P. Kainberger explained that this issue only affects DeviceAccess devices and occurs because of the limited resolution on the DAC (digital/analog converter):

LSA schickt die Gerätesollwerte im Allgemeinen in physikalischen Größen (Ampere, Volt, Frequenz,...). Damit können aber die wenigsten Geräte etwas anfangen, denn die Geräte haben meistens keinen digitalen Sollwerteingang sondern einen analogen. D.h. der gewünschte Sollwert in Ampere muss umgewandelt werden auf eine Spannung (Volt) am DAC (digital/analog converter) und mit dem regelt das Gerät dann, sodass der Magnet mit dem gewünschten Strom (Ampere) versorgt wird (das bewirkt dann ein entsprechendes Feld im Magneten).

Die Genauigkeit bei dieser Umwandlung ist nicht beliebig gut, meistens rechnen wir mit 16 Bit, in seltenen Fällen 18. Viele 'alte' Geräte haben nur eine 12 Bit Genauigkeit.

GS12ME1I (Injektionsseptum Sis18) hat einen Maximalwert von 300000 Volt und der DAC kann bestenfalls 16 Bit (eigentlich nur 15, das 16-te Bit ist das Vorzeichen). Die kleinste sichtbare Änderung im Sollwert ist 1 Bit. Bei 300000 V in 15 Bit (32767) macht das 9,155 V . Andersrum: Der vorgegebene Sollwert kann nur auf vielfache von 9,155V übertragen werden. Meistens weiss das Gerät [... ] auch nicht [welche Auflösung der zugehörige DAC hat]. Die allermeisten nehmen einen analogen Sollwert von einem DAC und letztlich bestimmt der dann die Genauigkeit.

Handling suggestion: Variable delta depending on value range

In the Operations-Application Steering Meeting on 2022-03-21, O. Geithner made a comment about handling such slight value differences. She suggested that, if such differences were to be handled with delta values, we should not use constant delta values everywhere, but instead should take the value range of the device into account: E.g. for a device with a value range from 0 - 10 A, a value difference of 0.5 A is much more significant than for a device with a value range from 0 - 10 000 A.

As a sidenote to that suggestion, J. Fitzek pointed out that if we wanted to do that, there might be the need for Operations to provide additional data about device min/max values or acceptable deltas, as we might not have min/max data available for all relevant devices in our databases or via JAPC.

FESA: Splitting of array values for ramps

Many devices report different arrays for the ramp data of constant (? not sure if only those) functions than what LSA sends down, e.g. from 2022-02-08_SampleSSVC.txt :

BeamProcess: SIS18_FAST_HHD_20220127_182722.C1.SIS18_RING.BEAMOUT_INIT.1 Selector: FAIR.SELECTOR.C=6:T=300:S=4:P=26

Parameter Name	GS02BE1/Setting#coeffArrDetunePhase
LSA Setting	(double[][]:1x6) -> [0.0, 0.212, 0.0, 0.0, 0.0, 0.0]
Device Setting	(double[][]:2x6) -> [0.0, 0.11199999999999999, 0.0, 0.0, 0.0, 0.0], [0.11199999999999999, 0.212, 0.0, 0.0, 0.0, 0.0]

BeamProcess: SIS18_FAST_HHD_20220127_182722.C1.SIS18_RING.BEAMOUT_RESET.1 Selector: FAIR.SELECTOR.C=6:T=300:S=4:P=33

Parameter Name	GS02BE1/Setting#coeffArrDetunePhase
LSA Setting	(double[][]:1x6) -> [0.0, 1.264, 0.0, 0.0, 0.0, 0.0]
Device Setting	(double[][]:2x6) -> [0.0, 1.164, 0.0, 0.0, 0.0, 0.0], [1.164, 1.264, 0.0, 0.0, 0.0, 0.0]

According to P. Kainberger, this is done intentionally and affects only FESA devices: The device adapts the Set value set by LSA to split off a part at the end (up to 100 ms - compare first two values in the array values in the Device settings listed above!). This means that the resulting ramp is identical, it is only split differently. To make sure that this effective array is "visible" somehow when debugging etc., the resulting array is published in the Setting property.

Wir brauchen halt am Ende einer Rampe ein Stück (<=16ms), in dem wir möglichst kurze Polynomlängen haben. Deshalb teilen wir bei Bedarf am Ende bis zu 100ms ab, um die in kleinere Teile (2ms) zu zerstückeln. Wenn wir das verstecken, dann wißt ihr halt gar nicht mehr, was wir da machen. Und wenn da mal was schief läuft kann man nicht mehr so richtig gucken. :thinking_face:

Ideas to avoid this Set value discrepancy:

• Report the value exactly as sent by LSA in the property Setting, and create another property (e.g. AdaptedSetting) that contains the adapted value
  • Not preferred by P. Kainberger:
    • What the device really does is not quite as "visible" as it is right now
    • Double amount of memory used in the FESA class!
• LSA could already split the array as needed by the device (logic for determining when/how to split it would then have to be integrated into the model)
  • Preferred by P. Kainberger: Less work to do for the FESA class, and LSA-Set value and Device-Set value would automatically be consistent!
  • Not preferred by D. Ondreka: Introduces device-/hardware-specific logic into the model
• Change the comparison logic for functions: Instead of using a simple equals check, instead create a value table for the functions with pre-defined steps. If the tables are identical (with a small tolerance for floating point rounding for the function values), we can consider the functions equals.
  • Suggested by D. Ondreka
  • Not preferred by A. Walter / R. Müller: Extra logic in Setting Comparison, not readily available elsewhere (e.g. for other tools like Sequencer etc.)
• Use "RevisionRules" (?) or special Make Rules or a similar concept in the LSA model after calculating the pure physics value, to make clear that this is an adaptation done because of device-/hardware-specific reasons.
  • Suggested by R. Müller, apparently something like "RevisionRules" is used at CERN?
• In Setting#xyz, report the setting as it was Set. In Acquisition#xyzSet, report the Set value as it was actually used by the device.
  • Suggested by R. Müller

Tracked in AP-internal JIRA as https://fcc-app.atlassian.net/browse/ADP-272 .

Uninitialized values that throw exceptions when read

Parameter Name	LSA Setting	Device Setting
!YR01MP1I /Setting#current	(double:1) -> 1914.909925	Exception: INVALID_DATA_RANGE: Invalid range of data provided for access point 'YR01MP1I/Setting': FESA_10016 Invalid value: '0' not in range '[100; 3072]'.. /opt/fesa/fesa-fwk/7.3.0/include/fesa-core/Utilities/RangeChecker.h:94 device="YR01MP1I" property="Setting" selector="FAIR.SELECTOR.C=7:T=203:S=7:P=13" requestType="GET"

To be verified: This device probably has no "set" values yet and starts with a default of 0. When trying to read this value, we receive and out of range exception.

D. Day explained:

Die automatische rangechecks (gegen _min,_max) werden bei Gets und Sets gemacht. Wenn der Zustand ungueltig ist, gibt es ein Exception beim lesen.

Rangecheck exceptions bei Gets ist ein Requirement von InCA. Die originale Requirements hatten nur rangechecks fuer gets, sets wurden nur spaeter geprueft ...

Accessing JAPC from Servlets

Raphael and Anne had originally planned to perform the periodical checks in the LSA REST Servlet. However, they found out that this is not easily possible: JAPC and CMW are usually configured by using system properties, but using system properties in a servlet is discouraged (because it could affect all Servlets running in the same server).

For the JAPC REST service, V. Rapp has created a custom build of JAPC that can be configured programmatically, but he considers this solution in its current state as a bit of hack and would strongly like to avoid having this custom build used by other projects/groups.

As an alternative to using JAPC via Java, it would be theoretically possible to use the JAPC REST API to read the values. It has not yet been evaluated if this would work: The value comparison is currently done using a simply equals check, because both LSA and Java JAPC provide similar data types. The JAPC REST API would provide the data in a different type that could probably not simply be compared using equals.

Further Thoughts

• For SSVC monitoring, how often should the checks be performed to be useful for operations?
• Is there a need for some kind of "synchronisation" with LSA to avoid performing SSVC checks and LSA drives at the same time (which could lead to non-existent inconsistencies being detected)?
  • Example scenario: SSVC retrieves LSA Set values. At almost the same instant, LSA finishes a trim and starts driving settings. SSVC reads Device Set values, some of which have already been supplied by the LSA drive (and are therefore inconsistent with the originally retrieved LSA Set values).
  • Andreas pointed out that to avoid SSVC checks during drives, we would probably have to acquire a database lock on the pattern (just as the LSA drive does) during the SSVC check for the pattern.
  • Would that scenario still be likely to happen once we have transactions?
• As long as we don't have transactions: When non-lenient drive is used in LSA, it is possible that a drive is only performed partially (until any device answers with an exception), so that some devices have received new values and some haven't yet. The trim is then rolled back, and there is no automatic "rollback" / "re-supply" of the old settings. => This should become unproblematic when we have transactions!
• Assuming that the SSVC would only detect "real" inconsistencies (= resolution errors etc. would be properly handled): Should there be some kind of interaction between the SSVC and some kind of alarm system, interlock system/MASP, etc. when Set value inconsistencies are detected?
• Do we want more history (possibly even long-term archiving?) for the Set-Set value comparisons over time?
• D. Ondreka suggests to differentiate between missing values in the device (severe error) and wrong values in the device (still an error, but less severe), and to find a way to explicitly display the existence of missing values to the users in the apps. An "unsupplied" device is almost always not intended
  • Caveat: We can only detect "missing" values for non-primitives, e.g. for functions represented by arrays, because an "unsupplied" device will have null. For primitives, we cannot differentiate: if e.g. the LSA setting is true and device has false, we do not know if the false is a "missing"/"unsupplied" value or an "old" value.
  • If we implement this differentiation in the monitoring app, we might want to keep in mind H. Rödl's suggestion (see below) for more noticeable highlighting.

• H. Rödl suggests a more noticeable highlighting (e.g. red/yellow background) in case of detected inconsistencies. This seems very reasonable, but is only feasible once we have a better detection of "true" inconsistencies.

-- AnnekeWalter - 08 Feb 2022