White Rabbit based Timing
Steps to create a White Rabbit triggered real-time event
- New operational front-ends first need to be registered by the Infrastructure- and by the Timing-Group:
The design needs:
- RT Action
- Logical Event of type "timing"
- Scheduling unit referencing the logical-event and the RT-action
The DU needs a concurrency layer containing a reference to the scheduling unit in the design.
The executable needs to be of type 'mixed'.
The instantiation file needs:
Timing configuration at: classes->[class name]->device-instance->configuration
- TimingDomain: The id of the TimingDomain has to mach the GroupID of the incoming event. Events of all other domains will be ignored
- accelerator-zone: required for the settings database
- Check this link in order to see all GroupID's which are played on a FEC
- mainMuxCriterion 'SEQUENCE' to multiplex all fields per sequence-index, 'PROCESS' to multiplex them per process-index, 'CHAIN; to multiplex them per chain-index, NONE, if you dont want to multiplex your fields.
Event configuration at: classes->[class name]->events-mapping->[event name]->event-configuration->Timing->hardware-event
Set the event configuration name and event code (e.g.EVT_START_RF#1)
- offset: The attribute 'offset' is optional. An offset in nanoseconds can be applied to an hardware-event to delay the treatment of the event.
- beam-in-flag: The attribute 'beam-in-flag' is optional. This flag is meant to define wheher beam is hopefully available during a whole beam process.
Event mapping at: classes->[class name]->device-instance->event-mapping->[event name]->event-configuration-ref
Select the event configuration defined above.
The FESA framework will register with the timing network to receive events that match the accelerator zone and event numbers given. The framework will always receive Process Start and Sequence Start events to track timestamps, but will not raise an RT-Action unless specified in the instantiation file.
The timing event triggers the creation of a multiplexing context that contains the event timing details that are passed to the RT-Action.
timespec eventsourceExecutionTimestamp; // Time FESA event source is triggered (Linux)
uint64_t eventExecutionTimestamp; // Time SAFTlib executes the event (WR)
uint64_t sequenceTimestamp; // Time current sequence started (WR)
uint64_t processTimestamp; // Time current beam process started (WR)
uint64_t beamProductionChainTimestamp; // Time current beam production chain started (WR)
uint64_t acquisitionTimestamp; // Time Event is Scheduled by the timing master (WR)
bool late; // event executed after scheduled time
bool early; // event executed before scheduled time
bool conflict; // simultaneous events - order undefined
Use of the Multiplexing Context in a Realtime Action
const fesaGSI::TimingContextWR* contextWR = dynamic_cast<const fesaGSI::TimingContextWR*>(pEvt->getMultiplexingContext());
White Rabbit Context contains 4 flag bits.
getEventFlags(): returns the flag bits
The 4th bit,BeamIn, is availaible as:
isBeamIn(): true if the event had the BeamIn
The 3rd bit,ChainStart, is availaible as:
isChainStart(): true if the event had the ChainStart
Bits 1-2 are reserved
The White Rabbit Context contains error flags from the timing receiver:
isEarly(): the Timing Event was triggered before its deadline (to prevent timing receiver from filling)
isLate(): the Timing Event was triggered after its deadline
isConflict(): two Events were triggered simultaneously and their order is undefined
These three errors will typically indicate serious Timing Master or clock synchronization problems
isDelayed(): the timing receiver and software stack was unable to process events fast enough
isTimingError(): true if any error flag is set
Per default, all multiplexed setting-fields of a multiplexed device will be duplicated:
- times 1024 when the device is multiplexed per Process Index
- times 64 when the device is multiplexed per Sequence Index
- times 64 when the device is multiplexed per Chain Index
This default multiplexing-depth should be used whenever possible. If for some reason (E.g. insufficient memory / system resources) the default cannot be used, please inform the LSA and FESA team about that! Otherwise it may happen that your device fails to handle an incoming out-of-bound Process/Sequence index during operation, which was sent by LSA.
The multiplexing-depth can be adjusted manually by passing the application-argument "-muxDepth <value>":
myDUBinary ...... -usrArgs "-muxDepth 123"
TimingUtility Helper Functions
//For work with White Rabbit EventIDs.
The bit positions in an EventID are defined in fesa-core-gsi/RealTime/TimingEventSourceWR.h (EVENTID_EVENTNO_OFFSET etc.)
Saftlib Status Monitoring
The FESA Timing EventSource
monitors the status of the timing receiver lock via callbacks and regular polling.
// true if within the last 30s saftd has generated a timing event or responded to a d-bus request
// true if the timing receiver is locked to the timing master
fesaGSI::TimingReceiverStatus status = fesaGSI::getSaftlibStatus();
if (status.alive) ...
if (status.locked) ...
Status monitoring is initialized on first use. If the FESA class does not use the White Rabbit Timing EventSource
, the status monitor will create a new Proxy to the Timing Receiver and start a new glib mainloop to receive callbacks. If the FESA class has defined the configuration property deviceNameTimingReceiver
this name will be used to find the timing receiver. If no timing receiver is found all status items will return false.
Working With Devices in different Timing Groups
Devices in a deploy unit may be configured to be in different TimingDomains and therefore respond to events in different timing groups.
In the instance file events-mapping it is determined which devices will respond to which event, and which timing group each device belongs to.
In an RTAction, the device collection is determined by which devices are listening to an event. If two devices in two different timing groups are listening for the same event number and that event number is sent to both groups by the timing master, two RT Actions will be triggered with the device collection containing both devices.
If a device is only interested in events for its own timing group the device collection can be filtered:
void MyAction::execute(fesa::RTEvent* pEvt)
const std::vector<Device*>& alldevices = getFilteredDeviceCollection(pEvt);
const std::vector<Device*>& devices = fesaGSI::FilterDeviceCollectionToTimingGroupID(alldevices, pEvt);
for (auto const& device : devices)
Access to SAFTlib functions
White Rabbit functions are available in the TimingEventSourceWR class
Device* device = WRTimingTestServiceLocator_->getDeviceCollection();
fesaGSI::TimingEventSourceWR::registerTrigger(device, eventNo, offset, tag);
std::string className = this->WRTimingTestServiceLocator_->getClassName(); // substitute your FESAClassName for WRTimingTest
fesaGSI::TimingEventSourceWR* evtSource = dynamic_cast<fesaGSI::TimingEventSourceWR*>(fesa::AbstractEventSourceFactory::getEventSource(className, "Timing"));
evtSource->registerTrigger( device, eventNo, offset, tag );
To generate tags on the SCU bus in response to a timing event:
void TimingEventSourceWR::registerTrigger( AbstractDevice* device, int eventNo, uint64_t offset, uint32_t tag );
The groupID is retrieved from the instantiation file via the device pointer.
Where offset is the time delay in ns after event before tag is written
To stop an event responding:
void TimingEventSourceWR::deregisterTrigger( AbstractDevice* device, int eventNo );
To manually generate a tag on the SCU bus:
void TimingEventSourceWR::injectTag( uint32_t tag);
The tool saft-scu-ctl can view/create/destroy tag conditions
To view the tags registered in the timing receiver:
saft-scu-ctl tr0 -l
To create tags on the SCU bus in response to a timing event:
saft-scu-ctl tr0 -c id mask offset tag
The condition will be removed when saft-scu-ctl closes.
To remove unowned conditions
saft-scu-ctl tr0 -x
The SCU must be running the saftlib daemon. Example ps output:
1623 root 2:47 /sbin/saftd tr0:dev/wbm0
Release 8.0.2 of FESA expects a SAFTlib version of 2.2.0 or newer. To check which version is installed:
saft-ctl tr0 -i
The Timing Receiver must be locked to operate correctly. Check that saft-ctl shows "WR Locked":
saft-ctl tr0 -s
No-lock can be caused by the wrong firmware version as well as a missing network link.
If one of the daemons does not run, check that the right fpga-firmware-version is installed.
- If the command "eb-info" is available:
# eb-info dev/wbm0
- Check the Build Type and Date
- Name and Date: Timing Release "Doomsday" required
In order to get a up to date fpga-firmware, please contact the Timing Team
Use evtsnoop to check that White Rabbit Events are played on each Timing Network:
How to configure an SCU
To run the dbus and saftlib daemons automatically on an SCU after reboot create a link to the saftlib initialization script in the nfsinit folder for the SCU:
lrwxrwxrwx 1 matthies bel 19 Jul 9 13:45 20_timing-rte -> ../global/timing-rte-tg-productive
lrwxrwxrwx 1 root root 20 Jul 12 11:24 50_fesa -> ../global/fesa_64bit_pro_environment_noRT
The scripts will be run in ascending order indicated by the numbers at the beginning.
For more information on the timing runtime environment see https://www-acc.gsi.de/wiki/FESA/FESA3_TimingRTE_501
Multiple instances per SCU
The SAFTlib daemon controls access to the timing hardware. FESA classes create proxy objects to interact with the SAFTlib daemon. This allows multiple FESA classes to execute on the same SCU and register for the same timing event. The number of events that can be listened to is limited by the condition table of the ECA
Accessing SAFTlib directly
Using the Saftlib Directory a FESA class can connect to the Saftlib Daemon without going through the FESA framework. Saftlib tracks ownership of events and actions so events registered with the FESA framework may be seen but not modified. A class may have its own set of conditions instead of - or in addition to - a timing event source registered by the framework.
Release 7.0.0 uses Saftlib v2. The communication between the saftd process and the client proxies now uses pipes between each process, replacing the previous dbus message bus based system.
There is a migration guide at https://www-acc.gsi.de/wiki/Timing/Saftlib2MigrationGuide
The major changes are:
Removal of glibmm types
Saftlib's dependency on glibmm was removed. See the migration guide for details.
Basic Types: gint32 -> uint32_t (etc.)
Smart Pointer: Glib::RefPtr -> std::shared_ptr
Removal of glib initialization
Gio::init(): an initialization call is no longer required
Removal of glib::mainloop polling mechanism - replaced by blocking wait call
Previous saftlib required the glib mainloop polling mechanism to run in the main thread or a background thread and manage callbacks and messages.
Saftbus uses the wait_for_signal function which blocks waiting, managing callbacks, until the signal being waited for arrives.
A saftlib::SignalGroup object
is passed when constructing the proxy objects:
timingSink = saftlib::SoftwareActionSink_Proxy::create(timingReceiver->NewSoftwareActionSink(""), signalGroup)
std::shared_ptr<SoftwareCondition_Proxy> condition = SoftwareCondition_Proxy::create(timingSink->NewCondition(true, id, mask, offset), signalGroup2);
The wait call will block until any signal in the group arrives
See again the migration guide at https://www-acc.gsi.de/wiki/Timing/Saftlib2MigrationGuide
Saftlib v2 has changed all methods that accept or generate timestamps to use saftlib::Time objects
The methods using uint64_t timestamps are deprecated.
void on_action(guint64 id, guint64 param, guint64 time, guint64 execution, guint16 flags)
void on_action(uint64_t id, uint64_t param, saftlib::Time time, saftlib::Time execution, uint16_t flags)
uint64_t timestamp = time.getUTC() or time.getTAI()
As an example, a skeleton custom event source for a SoftwareAction.
Create a Custom Event Source, link to RTAction, Set EventMapping.
std::vector<std::shared_ptr<SoftwareCondition_Proxy> > conditions_;
Constructor / Init
std::map<Glib::ustring, Glib::ustring> timingDevices = saftlib::SAFTd_Proxy::create()->getDevices();
timingReceiver_ = saftlib::TimingReceiver_Proxy::create(timingDevices.begin()->second, signalGroup_);
timingActionSink_ = saftlib::SoftwareActionSink_Proxy::create(timingReceiver_->NewSoftwareActionSink(""), signalGroup_);
void CustomTiming::connect(const boost::shared_ptr<fesa::EventElement>& eventElement)
uint64_t eventID = 0x0fa0001000000000;
uint64_t eventMask = 0xfffffff000000000;
int64_t offset = 0;
std::shared_ptr<SoftwareCondition_Proxy> condition = SoftwareCondition_Proxy::create(timingActionSink_->NewCondition(true, eventID, eventMask, offset), signalGroup_);
condition->Action.connect(sigc::mem_fun(this, &CustomTiming::on_action)); // register object method as callback function
conditions.push_back(condition); // the smart pointer must stay alive for actions to occur
catch (const saftbus::Error& error)
// invalid parameters
void CustomTiming::wait(boost::shared_ptr<fesa::RTEvent>& eventToFire)
// read timing information that was written during callback
// fire event
// callback function
void CustomTiming::on_action(uint64_t id, uint64_t param, saftlib::Time time, saftlib::Time execution, uint16_t flags)
// unpack error flags
const bool late = (flags & 1) != 0;
const bool early = (flags & 2) != 0;
const bool conflict = (flags & 4) != 0;
const bool delayed = (flags & 8) != 0;
// read timestamps
uint64_t timestamp = time.getUTC();
// do something with timing information
SAFTlib Version History
Released as part of Timing Fallout v6.0.0
No changes to Saftlib API
Released as part of Timing Enigma v5.0.1
d-bus replaced with saftbus
Support for UTC timestamps
Shared-memory interface to Master Function Generator driver
Function Generator Rescan
- 4.2018 Saftlib Version 1.4.0
Released as part of Timing Doomsday v4.0.4
Timing Receiver: Firmware Version
- 12.2017 Saftlib Version 1.2.0
Released as part Timing cherry-v4 Release
Rebuilt with glibmm2.50 after asl maintenance update
Master Function Generator - Functions returning vector<bool> now return vector<int> (glibmm2.50 vector<bool> requires c++14 support)
- 07.2017 Saftlib Version 1.1.0
Released as part Timing "C" Release
Function Generator uses File Descriptor d-bus interface
Master Function Generator Added
- 12.2016, Saftlib Version 1.0.8
Released as part of Timing Run-Time Environment Release 8
- 11.08.2016, Saftlib Version 1.0.8
Event triggers for SCUbus Tags with adjacent EventIDs
Action offsets greater than 2^32ns work correctly.
Negative offsets for actions work correctly.
No Late Errors for events that have not been subscribed to
High 32-bit of payload is correct, no longer copied from eventid.
saft-ctl tool replaces the functionality of dm-snoop and eca-ctl
Fixed inconsistent signedness of guard and offset parameters.
Usage of local timing
-- due to a bug
local timing is not usable in FESa v4.0.0 --
While adding a FEC
( press "add FEC
" in the deployment unit ) in the folder of this new FEC
a file called <MyFEC>Timing.xml will be generated.
By default this file looks like that:
<?xml version="1.0" encoding="UTF-8"?><fec name="zui">
<event code="4711" name="MY_TIMING_EVENT_1"/>
<event code="4712" name="MY_TIMING_EVENT_2"/>
So when picking a hw-event in the instance-file, at the very end of the list you as well should see "MY_TIMING_EVENT_1#4711" and "MY_TIMING_EVENT_2#4712".
You are free to modify this xml file to add any number of self-defined events. When you finished your modifications, you need to trigger "Synchronize Source Code" in the deployment unit, in order to update the schema of the instance-file and to make use of your local timing configuration.
Simulation of Timing Events
For easy simulation, best use the following tool: /common/usr/fesa/tools/SimulateEvents/SimulateEvents
Copy it to your FEC
and use it from there. Use -h to see its documentation and all possible parameters.
The source-code can be found here: https://www-acc.gsi.de/svn/fesa/framework/branches/SimulateEvents
Generating Timing Events
The saft-ctl inject function will generate a timing event equivalent to that from the timing master.
saft-ctl tr0 -f inject <eventID> <param> <time>
Tool to help generating the eventID and parameter (using FormatID
/opt/fesa/nfs/global/scripts/inject-event-id.sh <group> <event_no> <sequence> <beam process> <chain> <flags>
In order to decode events take a look on FESAToolsDecodeWRTimingEvents
Simulated Data Master
Events can also be simulated using the saft-dm tool:
saft-dm tr0 <schedule_filename> where schedule_filename contains lines: <eventID> <param> <time>
- 28 Mar 2018