How-To: White Rabbbit UNILAC PZ (wr-unipz) - Coding and Deployment
"wr-unipz" is a component of the MIL-based UNILAC 'Pulszentrale' (UNIPZ). As a field-bus, it does not use the MIL 'Event' bus but a White Rabbit network. Logically, wr-unipz
is identical and on the same level as the seven 'Pulszentralen' (PZ) of the UNILAC timing system. As for the other PZ, data supply and 50 Hz synchronization are provided through the 'Super Pulszentrale' (SPZ) of the UNILAC timing system. For technical and conceptual reasons, the UNILAC timing system and the GSI/FAIR General Machine Timing system (GMT) must use two distinct White Rabbit networks.
- Users at UNILAC
- identical interface as for the FAIR timing system
- same timing receivers (SCU, Pexaria...)
- same drivers, Etherbone tools, saftlib
- timing messages have a 'UNILAC dialect', see here
- there are just UNILAC events and no such thing as CMD_SEQ_START ...
Integration of White Rabbit Pulszentrale into UNILAC Pulszentrale. Shown are the existing components Settings Management
(grey box), Super Pulszentrale
(light green box), Pulszentralen 1..7 (green boxes) and the new White Rabbit Pulszentrale
(blue box). As well shown are MIL event bus
lines (green arrows), a White Rabbit network
(blue arrows) as well as connections from Super Pulszentrale to PZs (black arrows). Details see text.
Today (2020) the UNILAC is still operated using the 'old GSI control system' Device Access
. The timing system is based on a so-called MIL Event Bus
with a Pulszentrale
(PZ) as master. UNILAC has seven timing areas with one PZ each. Thus, there are seven distinct MIL cables, one for each PZ. The seven PZs are coordinated via the Super Pulszentrale
(SPZ), which has two tasks.
- It supplies the PZs with data. The data contain information for up to 16 independent virtual accelerators (VACC). For each VACC there are two distinct sets of data (called 'Kanäle'), one for normal operation and one for low intensity operation ('verkürzt', 'Profilgitterschutz'). In total the SPZ needs to provide 7 * 16 * 2 = 224 event sequences. Each sequence is a list of event data and a time offset relative to the beginning of a UNILAC cycle. SPZ transmits the data to the PZs using Device Access via the standard ACC network.
- SPZ and the PZs are interconnected via an internal MIL event bus. SPZ uses this internal bus for various purposes:
- announce event: announces the virtual accelerator number that must be played for a specific PZ during the next cycle; there may be up to seven of these events per cycle (one for each PZ)
- cycle start event: starts the next UNILAC cycle. Each cycle has a length of ~20ms (50 Hz); there is one event per cycle starting the event sequences at (up to) seven PZs simultaneously
- service event: service events are played after the event sequence (at a PZ) has been completed, there is one event for each service event to be sent
- synch data event: if new data have been supplied via Device Access, they will become active during the next cycle; this event is processed by all PZs
The White Rabbit PZ
just behaves like the seven PZ. It is supplied via Device Access and is connected to the internal MIL Event Bus to receive the events from SPZ. There are two main difference to the seven PZs.
- there is only one White Rabbit network for all seven timing areas at UNILAC
- White Rabbit PZ uses the concept of an alarm based timing system unlike the seven PZs, that are event based
Stack of the White Rabbit Pulszentrale
Overview on the White Rabbit Pulszentrale. Shown are SCU (blue box), host system and FPGA (orange boxes), standard IP cores instantiated in the FPGA (light green boxes), components by the WR-UNIPZ
(yellow boxes) and components of the old GSI Control System (dark green boxes). Indicated is data flow from the Super Pulszentrale
(thick black arrows), a Lemo cable (thin blue arrow), Wishbone and Etherbone communication (thin black arrows) and Timing Messages
sent to the White Rabbit network (thick blue arrow). Details see text.
One of the main challenges of this development was to bring together two different worlds: The old GSI Device Access
control system and the very different ecosystem of the White Rabbit based developments. The low level interface to timing system components is Etherbone. The high level standard API is saftlib. However, WR UNIPZ uses a third approach, namely the use of a shared library. This disentangles the two distinct ecosystem as much as possible:
- Developing, building of the firmware and tools works independently of the old GSI control system.
- Developing, building of the Device Access class only requires including one header file and linking against a single shared library.
The basis of WR-UNIPZ is a regular SCU as Frontend with Timing Receiver. Hardware, gateware and the RunTime Environment (RTE: drivers + tools) are identical to other frontends. This is configured and rolled-out like all other SCU based frontends.
WR-UNIPZ: Firmware and Tools
Very tightly coupled to the gateware is firmware and low-level software tools of WR UNIPZ that are hosted in the same source code control and build environment as gateware and RTE. It is part of the White Rabbit based ecosystem. The tools contain all a script and helper software to upload the firmware to a lm32 soft CPU embedded in the VHDL of the regular SCU during runtime. Moreover, the script performs basic configuration of firmware, IOs ports and the ECA. Configuration and roll-out works via nfs-init (as usual) and includes a shared library
as API. This shared library hides implementation details as much as possible from the user. Included as well is a CLI for testing and diagnosis.
Precise timestamping needs to be done for the 'cycle start event' received from the SPZ via MIL bus. This is achieved by the Timestamp Latch Unit (TLU) of the ECA. The 'MIL Piggy' generates a TTL pulse on an output when the cycle start event is received. Via a LEMO cable, this TTL signal is connected to an input configured as TLU input of the ECA. This solution provides timestamping in hard real time by the gateware, indepently of the firmware.
PZUD: Device Access Class
The equipment class
is part of the Device Access environment and part of the equipment model pzu.
only uses the shared library
as the sole interface to the lm32 firmware in the FPGA. Basically,
can be seen as a wrapper around that shared library.
I (db) prefer the following workflow
- Coding, building on my main linux development machine. For testing the firmware and
socat enabled on a test SCU, it is sufficient to
eb-fload the firmware and to use the CLI
wrunipz-ctl via tcp.
- Once the firmware, library and tools are mature I commit and push to the GIT repo.
- Rollout needs to be done the ASL7 cluster. I (db) never code on the ASL but just use it as a building and deployment system after pulling the working environment from the GIT repo.
This workflow has two advantages: It allows me to work on my favorite working environment and assures a roll-out is done with sources that committed and pushed.
Coding and Building
For cloning the repo and setting up the tool chain just follow the instructions of the current release
. The gateway must follow the current timing release very closely. It typically resides in a branch that stems from the current release branch.
- Convention: branch is named
wrunipz_dietrich_< YYYY-MM-DD >
Depending on your favorite working environment, consider to follow the directions given here
cd bel_projects/modules/wr-unipz // main working folder of wr-unipz
asl // scripts for roll-out on the asl cluster
fw // firmware
include // header files
Makefile // the big makefile, with directions for ASL on the top of the file
x86 // x86 software
make // builds firmware (calls 'fw/Makefile') and software (calls 'x86/Makefile')
Rollout for ACO
Rollout on a SCU in the ACO environment requires working on the ASL 7 cluster. There are two steps. First, rebuilding of binaries (firmware and CLI). Second, prepare 'nfsinit' for a specific SCU.
Remark: There is no dedicated FPGA image, the gateway uses the default SCU image of the current release
. The startup script (called during nfsinit) replaces the firmware of the user lm32 during run-time.
For cloning the repo and setting up the toolchain just follow the instructions of the current release
. Consider to follow the directions given here
Building and Copying
- here is the Makefile //.../bel_projects/modules/wr-unipz/Makefile
- try head Makefile to check the parameters of the Makefile
- build the Firmware and Software
DEV : try make clean
DEV : try make MASP=YES PRO=NO PREFIX= all
PRO : try make clean
PRO : try make MASP=YES PRO=YES PREFIX= all
- (just in case: building on a non-ASL environment is as easy as for other eb-tools when using MASP=NO)
- deploy Firmware, Gateware and scripts
DEV : try make MASP=YES PRO=NO PREFIX= STAGING=/common/export/timing-rte/wrunipz-dev deploy
PRO : try make MASP=YES PRO=YES PREFIX= STAGING=/common/export/timing-rte/wrunipz deploy
- cd to the nfsinit folder of the relevant SCU
- create the relevent symbolic link
DEV try 30_wrunipz -> ../global/timing-rte-wrunipz-dev
PRO try 30_wrunipz -> ../global/timing-rte-wrunipz
- note the target of the symbolic link and the folder names above
In case you are not familiar with Device Access and the build environment on the ASL >>> 6 <<<
cluster, checkout this
Standard Device Properties
Some info on coding...
provides some data and standard device properties
ebDevice, handle to Etherbone device of pzud
pz, number of PZ (0..6); there is one instance of the PZUD class for each PZ
resetDevice, init of all get-values; here: reset diagnostic values (for all PZ)
reset (for all PZ), but init of all set-values too; here: clears event data (for THIS PZ)
updateStatus, is triggered periodically to get and update the device status; here: get basic info (for ALL PZ, 'opReady'....)
onlineTest, is triggered periodically to get and update the device status; here: get very basic info (for ALL PZ, check if firmware is reachable)
getVersionString, obtains version info; here: PZUD, version, shared library
(User Service Routines) provide methods dedicated to PZUD only; USRs are defined in the file
pzud-write-evtsequ.cc, writes event data for a specific virtual accelerator and channel to the firmware, important method used by the 'UNILAC Super Pulszentrale'
pzud-read-evtsequ.cc, reads event data for a specific virtual accelerator and channel to the firmware
pzud-read-diag.cc, reads diagnostic info of firmware (for ALL PZ)
pzud-write-cmdcleardiag.cc, clears diagnostic info of firmware (for ALL PZ)
pzud-write-cmdsubmit.cc, submits/activats event data previously written to the PZs (for ALL PZ); remark: this method is only useful when playing with PZUD via the PropHelper. The 'Super Pulszentrale' uses a dedicated event on the internal MIL bus to which PZUD is connected.
Interface to the lm32 Firmware in the FPGA
There is only very little code implemented by PZUD. Basically, PZUD is just a wrapper around a shared library 'wrunipzlib.so'. This library provides the interface to the firmware including all communication. The library and headere file are provided here
/common/export/timing-rte/wrunipz-dev ; development system Programmentwicklungsraum
/common/export/timing-rte/wrunipz; production system LSB6
The Device Access class PZUD just includes the header and links against the shared library (see the info on 'SConscript' here
Building and Roll-Out
Follow these directions
- 29 Apr 2020