White Rabbit based Timing

Steps to create a White Rabbit triggered real-time event

Generally

• New operational front-ends first need to be registered by the Infrastructure- and by the Timing-Group:
  • FEC registration Infrastructure-Group
  • FEC registration Timing-Group

Class Design

The design needs:

• RT Action
• Logical Event of type "timing"
• Scheduling unit referencing the logical-event and the RT-action

Deployment Unit

The DU needs a concurrency layer containing a reference to the scheduling unit in the design.

The executable needs to be of type 'mixed'.

Instantiation File

The instantiation file needs:

Timing configuration at: classes->[class name]->device-instance->configuration

• TimingDomain: The id of the TimingDomain has to mach the GroupID of the incoming event. Events of all other domains will be ignored
• accelerator:
• accelerator-zone: required for the settings database
  • Check this link in order to see all GroupID's which are played on a FEC
• mainMuxCriterion 'SEQUENCE' to multiplex all fields per sequence-index, 'PROCESS' to multiplex them per process-index, 'CHAIN; to multiplex them per chain-index, NONE, if you dont want to multiplex your fields.

Hardware-Event

Event configuration at: classes->[class name]->events-mapping->[event name]->event-configuration->Timing->hardware-event

Set the event configuration name and event code (e.g.EVT_START_RF#1)

Optional Attributes

• offset: The attribute 'offset' is optional. An offset in nanoseconds can be applied to an hardware-event to delay the treatment of the event.
• beam-in-flag: The attribute 'beam-in-flag' is optional. This flag is meant to define wheher beam is hopefully available during a whole beam process.

Event Mapping

Event mapping at: classes->[class name]->device-instance->event-mapping->[event name]->event-configuration-ref

Select the event configuration defined above.

Event Behaviour

The FESA framework will register with the timing network to receive events that match the accelerator zone and event numbers given. The framework will always receive Process Start and Sequence Start events to track timestamps, but will not raise an RT-Action unless specified in the instantiation file.

Multiplexing Context

The timing event triggers the creation of a multiplexing context that contains the event timing details that are passed to the RT-Action.

   uint32_t formatId;
   uint32_t groupId;
   uint32_t eventNumber;
   uint32_t sequenceIndex;
   uint32_t sequenceCounter;
   uint32_t processIndex;
   uint32_t beamProductionChainIndex;

   timespec eventsourceExecutionTimestamp;      // Time FESA event source is triggered (Linux)
   uint64_t eventExecutionTimestamp;         // Time SAFTlib executes the event (WR)
   uint64_t sequenceTimestamp;               // Time current sequence started (WR)
   uint64_t processTimestamp;               // Time current beam process started (WR)
   uint64_t beamProductionChainTimestamp;      // Time current beam production chain started (WR)
   uint64_t acquisitionTimestamp;            // Time Event is Scheduled by the timing master (WR)

   bool late;                           // event executed after scheduled time
   bool early;                           // event executed before scheduled time
   bool conflict;                        // simultaneous events - order undefined
   bool delayed;

Use of the Multiplexing Context in a Realtime Action

const fesaGSI::TimingContextWR* contextWR =  dynamic_cast<const fesaGSI::TimingContextWR*>(pEvt->getMultiplexingContext());

Event Flags

The TimingFormatID1 White Rabbit Context contains 4 flag bits.

getEventFlags(): returns the flag bits

The 4th bit,BeamIn, is availaible as:

isBeamIn(): true if the event had the BeamIn flag set

The 3rd bit,ChainStart, is availaible as:

isChainStart(): true if the event had the ChainStart flag set

Bits 1-2 are reserved

Error Flags

The White Rabbit Context contains error flags from the timing receiver:

isEarly(): the Timing Event was triggered before its deadline (to prevent timing receiver from filling)

isLate(): the Timing Event was triggered after its deadline

isConflict(): two Events were triggered simultaneously and their order is undefined

These three errors will typically indicate serious Timing Master or clock synchronization problems

isDelayed(): the timing receiver and software stack was unable to process events fast enough

isTimingError(): true if any error flag is set

Multiplexing Depth

Per default, all multiplexed setting-fields of a multiplexed device will be duplicated:

• times 1024 when the device is multiplexed per Process Index
• times 64 when the device is multiplexed per Sequence Index
• times 64 when the device is multiplexed per Chain Index

This default multiplexing-depth should be used whenever possible. If for some reason (E.g. insufficient memory / system resources) the default cannot be used, please inform the LSA and FESA team about that! Otherwise it may happen that your device fails to handle an incoming out-of-bound Process/Sequence index during operation, which was sent by LSA.

The multiplexing-depth can be adjusted manually by passing the application-argument "-muxDepth <value>":

myDUBinary ...... -usrArgs "-muxDepth 123"

TimingUtility Helper Functions

#include <fesa-core-gsi/RealTime/TimingUtility.h>
//For work with White Rabbit EventIDs. 
generate_event_id(TimingGroupID, EventNumber)
generate_event_id_mask(TimingGroupID, EventNumber)

The bit positions in an EventID are defined in fesa-core-gsi/RealTime/TimingEventSourceWR.h (EVENTID_EVENTNO_OFFSET etc.)

Saftlib Status Monitoring

The FESA Timing EventSource monitors the status of the timing receiver lock via callbacks and regular polling.

The TimingReceiverStatus currently contains:

// true if within the last 30s saftd has generated a timing event or responded to a d-bus request
bool alive
// true if the timing receiver is locked to the timing master
bool locked

#include <fesa-core-gsi/Utilities/Saftlib.h>
fesaGSI::TimingReceiverStatus status = fesaGSI::getSaftlibStatus();
if (status.alive) ...
if (status.locked) ...

Status monitoring is initialized on first use. If the FESA class does not use the White Rabbit Timing EventSource, the status monitor will create a new Proxy to the Timing Receiver and start a new glib mainloop to receive callbacks. If the FESA class has defined the configuration property deviceNameTimingReceiver this name will be used to find the timing receiver. If no timing receiver is found all status items will return false.

Working With Devices in different Timing Groups

Devices in a deploy unit may be configured to be in different TimingDomains and therefore respond to events in different timing groups.

In the instance file events-mapping it is determined which devices will respond to which event, and which timing group each device belongs to.

In an RTAction, the device collection is determined by which devices are listening to an event. If two devices in two different timing groups are listening for the same event number and that event number is sent to both groups by the timing master, two RT Actions will be triggered with the device collection containing both devices.

If a device is only interested in events for its own timing group the device collection can be filtered:

#include <fesa-core-gsi/RealTime/TimingUtility.h>
...
void MyAction::execute(fesa::RTEvent* pEvt)
{
const std::vector<Device*>& alldevices = getFilteredDeviceCollection(pEvt);
const std::vector<Device*>& devices = fesaGSI::FilterDeviceCollectionToTimingGroupID(alldevices, pEvt);

for (auto const& device : devices)
{
   device-> ...
}

Access to SAFTlib functions

White Rabbit functions are available in the TimingEventSourceWR class

Static Access

#include <fesa-core-gsi/RealTime/TimingEventSourceWR.h>
...
Device* device = WRTimingTestServiceLocator_->getDeviceCollection()[0];
...
fesaGSI::TimingEventSourceWR::registerTrigger(device, eventNo, offset, tag);

Instance Access

#include <fesa-core-gsi/RealTime/TimingEventSourceWR.h>
#include <fesa-core/RealTime/AbstractEventSourceFactory.h>
... 
std::string className = this->WRTimingTestServiceLocator_->getClassName(); // substitute your FESAClassName for WRTimingTest
fesaGSI::TimingEventSourceWR* evtSource = dynamic_cast<fesaGSI::TimingEventSourceWR*>(fesa::AbstractEventSourceFactory::getEventSource(className, "Timing")); 
...
evtSource->registerTrigger( device, eventNo, offset, tag );

SCU bus tags

To generate tags on the SCU bus in response to a timing event:

void TimingEventSourceWR::registerTrigger( AbstractDevice* device, int eventNo, uint64_t offset, uint32_t tag );

The groupID is retrieved from the instantiation file via the device pointer.

Where offset is the time delay in ns after event before tag is written

To stop an event responding:

void TimingEventSourceWR::deregisterTrigger( AbstractDevice* device, int eventNo );

To manually generate a tag on the SCU bus:

void TimingEventSourceWR::injectTag( uint32_t tag);

SCU bus tool

The tool saft-scu-ctl can view/create/destroy tag conditions

To view the tags registered in the timing receiver:

saft-scu-ctl tr0 -l

To create tags on the SCU bus in response to a timing event:

saft-scu-ctl tr0 -c id mask offset tag

The condition will be removed when saft-scu-ctl closes.

To remove unowned conditions

saft-scu-ctl tr0 -x

SCU Environment

The SCU must be running the saftlib daemon. Example ps output:

 1623 root       2:47 /sbin/saftd tr0:dev/wbm0

Release 8.0.2 of FESA expects a SAFTlib version of 2.2.0 or newer. To check which version is installed:

saft-ctl tr0 -i

The Timing Receiver must be locked to operate correctly. Check that saft-ctl shows "WR Locked":

saft-ctl tr0 -s

No-lock can be caused by the wrong firmware version as well as a missing network link.

If one of the daemons does not run, check that the right fpga-firmware-version is installed.

• If the command "eb-info" is available:

  • # eb-info dev/wbm0

  • Check the Build Type and Date
  • Name and Date: Timing Release "Doomsday" required

In order to get a up to date fpga-firmware, please contact the Timing Team

https://www-acc.gsi.de/wiki/bin/view/Timing/TimingSystemHowBuildingDeployment

Use evtsnoop to check that White Rabbit Events are played on each Timing Network:

/common/usr/lobi/bin/snoop --config=dev/int/pro

How to configure an SCU

To run the dbus and saftlib daemons automatically on an SCU after reboot create a link to the saftlib initialization script in the nfsinit folder for the SCU:

matthies@asl733:scuxl0052>pwd
/common/export/nfsinit/scuxl0052
matthies@asl733:scuxl0052>ll
lrwxrwxrwx 1 matthies bel  19 Jul  9 13:45 20_timing-rte -> ../global/timing-rte-tg-productive
lrwxrwxrwx 1 root     root 20 Jul 12 11:24 50_fesa -> ../global/fesa_64bit_pro_environment_noRT

The scripts will be run in ascending order indicated by the numbers at the beginning.

For more information on the timing runtime environment see https://www-acc.gsi.de/wiki/FESA/FESA3_TimingRTE_501

Multiple instances per SCU

The SAFTlib daemon controls access to the timing hardware. FESA classes create proxy objects to interact with the SAFTlib daemon. This allows multiple FESA classes to execute on the same SCU and register for the same timing event. The number of events that can be listened to is limited by the condition table of the ECA unit.

Accessing SAFTlib directly

Using the Saftlib Directory a FESA class can connect to the Saftlib Daemon without going through the FESA framework. Saftlib tracks ownership of events and actions so events registered with the FESA framework may be seen but not modified. A class may have its own set of conditions instead of - or in addition to - a timing event source registered by the framework.

Saftbus

Release 7.0.0 uses Saftlib v2. The communication between the saftd process and the client proxies now uses pipes between each process, replacing the previous dbus message bus based system.

There is a migration guide at https://www-acc.gsi.de/wiki/Timing/Saftlib2MigrationGuide.

The major changes are:

Removal of glibmm types

Saftlib's dependency on glibmm was removed. See the migration guide for details.

Basic Types: gint32 -> uint32_t (etc.)

Smart Pointer: Glib::RefPtr -> std::shared_ptr

Removal of glib initialization

Gio::init(): an initialization call is no longer required

Removal of glib::mainloop polling mechanism - replaced by blocking wait call

Previous saftlib required the glib mainloop polling mechanism to run in the main thread or a background thread and manage callbacks and messages.

Saftbus uses the wait_for_signal function which blocks waiting, managing callbacks, until the signal being waited for arrives.

This requires:

A saftlib::SignalGroup object

saftlib::SignalGroup signalGroup;

The SignalGroup is passed when constructing the proxy objects:

timingSink = saftlib::SoftwareActionSink_Proxy::create(timingReceiver->NewSoftwareActionSink(""), signalGroup)
std::shared_ptr<SoftwareCondition_Proxy> condition = SoftwareCondition_Proxy::create(timingSink->NewCondition(true, id, mask, offset), signalGroup2);

The wait call will block until any signal in the group arrives

signalGroup.wait_for_signal(timeout_ms);

UTC/TAI Support

See again the migration guide at https://www-acc.gsi.de/wiki/Timing/Saftlib2MigrationGuide.

Saftlib v2 has changed all methods that accept or generate timestamps to use saftlib::Time objects

The methods using uint64_t timestamps are deprecated.

Signal Callbacks:
void on_action(guint64 id, guint64 param, guint64 time, guint64 execution,  guint16 flags)
* becomes
void on_action(uint64_t id, uint64_t param, saftlib::Time time, saftlib::Time execution,  uint16_t flags)

uint64_t timestamp = time.getUTC() or time.getTAI()

Methods:

receiver->InjectEvent(timestamp)
* becomes
saftlib::Time timeUTC(makeTimeUTC(timestamp))
receiver->InjectEvent(timeUTC

Example

As an example, a skeleton custom event source for a SoftwareAction.

Create a Custom Event Source, link to RTAction, Set EventMapping.

Required Headers:

#include <SAFTd.h>
#include <TimingReceiver.h>
#include <SoftwareActionSink.h>
#include <SoftwareCondition.h>

Class members:

std::shared_ptr<saftlib::TimingReceiver_Proxy> timingReceiver_;
std::shared_ptr<saftlib::SoftwareActionSink_Proxy> timingActionSink_;
std::vector<std::shared_ptr<SoftwareCondition_Proxy> > conditions_;
saftlib::SignalGroup signalGroup_;

Constructor / Init

 std::map<Glib::ustring, Glib::ustring> timingDevices = saftlib::SAFTd_Proxy::create()->getDevices();
 timingReceiver_ = saftlib::TimingReceiver_Proxy::create(timingDevices.begin()->second, signalGroup_);
 timingActionSink_ = saftlib::SoftwareActionSink_Proxy::create(timingReceiver_->NewSoftwareActionSink(""), signalGroup_);


void CustomTiming::connect(const boost::shared_ptr<fesa::EventElement>& eventElement)
{
 uint64_t eventID = 0x0fa0001000000000;
 uint64_t eventMask = 0xfffffff000000000;
 int64_t offset = 0;
 try
 {
 std::shared_ptr<SoftwareCondition_Proxy> condition = SoftwareCondition_Proxy::create(timingActionSink_->NewCondition(true, eventID, eventMask, offset), signalGroup_);
 condition->Action.connect(sigc::mem_fun(this, &CustomTiming::on_action)); // register object method as callback function
 conditions.push_back(condition); // the smart pointer must stay alive for actions to occur
 }
 catch (const saftbus::Error& error)
 {
   // invalid parameters   
 }
}

void CustomTiming::wait(boost::shared_ptr<fesa::RTEvent>& eventToFire)
{

 signalGroup.wait_for_signal(10000);
 // read timing information that was written during callback
 // fire event 
 createEvent(eventToFire, CustomTiming::defaultEvent);
}

// callback function
void CustomTiming::on_action(uint64_t id, uint64_t param, saftlib::Time time, saftlib::Time execution, uint16_t flags)
{
 // unpack error flags
 const bool late = (flags & 1) != 0;
 const bool early = (flags & 2) != 0;
 const bool conflict = (flags & 4) != 0;
 const bool delayed = (flags & 8) != 0;

 // read timestamps
 uint64_t timestamp = time.getUTC();

 // do something with timing information
}

More Information

https://www-acc.gsi.de/wiki/pub/Timing/TimingSystemDocumentsSaftlib/saftlib.pdf

https://www-acc.gsi.de/wiki/Timing/TimingSystemEventNumbers

https://www-acc.gsi.de/wiki/Timing/TimingSystemGroupsAndMachines

SAFTlib Version History

• 10.2020 Saftlib 2.2.2

Released as part of Timing Fallout v6.0.0 No changes to Saftlib API

• 8.2019 Saftlib 2.1.0

Released as part of Timing Enigma v5.0.1

d-bus replaced with saftbus

Support for UTC timestamps

Shared-memory interface to Master Function Generator driver

Function Generator Rescan

• 4.2018 Saftlib Version 1.4.0

Released as part of Timing Doomsday v4.0.4

Interface Changes:

Timing Receiver: Firmware Version

• 12.2017 Saftlib Version 1.2.0

Released as part Timing cherry-v4 Release

Rebuilt with glibmm2.50 after asl maintenance update

Interface Changes:

Master Function Generator - Functions returning vector<bool> now return vector<int> (glibmm2.50 vector<bool> requires c++14 support)

• 07.2017 Saftlib Version 1.1.0

Released as part Timing "C" Release

Interface Changes:

Function Generator uses File Descriptor d-bus interface

Master Function Generator Added

• 12.2016, Saftlib Version 1.0.8

Released as part of Timing Run-Time Environment Release 8

• 11.08.2016, Saftlib Version 1.0.8

ECA2 Support

• 18.01.2016

Fixes:

Event triggers for SCUbus Tags with adjacent EventIDs are accepted

Action offsets greater than 2^32ns work correctly.

Negative offsets for actions work correctly.

No Late Errors for events that have not been subscribed to

High 32-bit of payload is correct, no longer copied from eventid.

Changes:

saft-ctl tool replaces the functionality of dm-snoop and eca-ctl

Interface:

Fixed inconsistent signedness of guard and offset parameters.

Usage of local timing

-- due to a bug local timing is not usable in FESa v4.0.0 --

While adding a FEC ( press "add FEC" in the deployment unit ) in the folder of this new FEC a file called <MyFEC>Timing.xml will be generated.

By default this file looks like that:

<?xml version="1.0" encoding="UTF-8"?><fec name="zui">
<local-timing>
<event code="4711" name="MY_TIMING_EVENT_1"/>
<event code="4712" name="MY_TIMING_EVENT_2"/>
</local-timing>
</fec>

So when picking a hw-event in the instance-file, at the very end of the list you as well should see "MY_TIMING_EVENT_1#4711" and "MY_TIMING_EVENT_2#4712".

You are free to modify this xml file to add any number of self-defined events. When you finished your modifications, you need to trigger "Synchronize Source Code" in the deployment unit, in order to update the schema of the instance-file and to make use of your local timing configuration.

Simulation of Timing Events

For easy simulation, best use the following tool: /common/usr/fesa/tools/SimulateEvents/SimulateEvents

Copy it to your FEC and use it from there. Use -h to see its documentation and all possible parameters.

The source-code can be found here: https://www-acc.gsi.de/svn/fesa/framework/branches/SimulateEvents

Generating Timing Events

The saft-ctl inject function will generate a timing event equivalent to that from the timing master.

saft-ctl tr0 -f inject <eventID> <param> <time>

Tool to help generating the eventID and parameter (using FormatID 1):

/opt/fesa/nfs/global/scripts/inject-event-id.sh <group> <event_no> <sequence> <beam process> <chain> <flags>

In order to decode events take a look on FESAToolsDecodeWRTimingEvents

Simulated Data Master

Events can also be simulated using the saft-dm tool:

saft-dm tr0 <schedule_filename> where schedule_filename contains lines: <eventID> <param> <time>

-- DominicDay - 28 Mar 2018