How-To: White Rabbbit UNILAC PZ (wr-unipz) - Coding and Deployment
Introduction
"wr-unipz" is a component of the MIL-based UNILAC 'Pulszentrale' (UNIPZ). As a field-bus, it does not use the MIL 'Event' bus but a White Rabbit network. Logically,
wr-unipz is identical and on the same level as the seven 'Pulszentralen' (PZ) of the UNILAC timing system. As for the other PZ, data supply and 50 Hz synchronization are provided through the 'Super Pulszentrale' (SPZ) of the UNILAC timing system. For technical and conceptual reasons, the UNILAC timing system and the GSI/FAIR General Machine Timing system (GMT) must use two distinct White Rabbit networks.
How-To
- WR-UNIPZ
- MIL-UNIPZ
- Users at UNILAC
- identical interface as for the FAIR timing system
- same timing receivers (SCU, Pexaria...)
- same drivers, Etherbone tools, saftlib
- differences
- timing messages have a 'UNILAC dialect', see here
- there are just UNILAC events and no such thing as CMD_SEQ_START ...
Documentation
Overview
Figure: Integration of White Rabbit Pulszentrale into UNILAC Pulszentrale. Shown are the existing components
Settings Management (grey box),
Super Pulszentrale (light green box), Pulszentralen 1..7 (green boxes) and the new
White Rabbit Pulszentrale (blue box). As well shown are
MIL event bus lines (green arrows), a
White Rabbit network (blue arrows) as well as connections from Super Pulszentrale to PZs (black arrows). Details see text.
Today (2020) the UNILAC is still operated using the 'old GSI control system'
Device Access . The timing system is based on a so-called
MIL Event Bus with a
Pulszentrale (PZ) as master. UNILAC has seven timing areas with one PZ each. Thus, there are seven distinct MIL cables, one for each PZ. The seven PZs are coordinated via the
Super Pulszentrale (SPZ), which has two tasks.
- It supplies the PZs with data. The data contain information for up to 16 independent virtual accelerators (VACC). For each VACC there are two distinct sets of data (called 'Kanäle'), one for normal operation and one for low intensity operation ('verkürzt', 'Profilgitterschutz'). In total the SPZ needs to provide 7 * 16 * 2 = 224 event sequences. Each sequence is a list of event data and a time offset relative to the beginning of a UNILAC cycle. SPZ transmits the data to the PZs using Device Access via the standard ACC network.
- SPZ and the PZs are interconnected via an internal MIL event bus. SPZ uses this internal bus for various purposes:
- announce event: announces the virtual accelerator number that must be played for a specific PZ during the next cycle; there may be up to seven of these events per cycle (one for each PZ)
- cycle start event: starts the next UNILAC cycle. Each cycle has a length of ~20ms (50 Hz); there is one event per cycle starting the event sequences at (up to) seven PZs simultaneously
- service event: service events are played after the event sequence (at a PZ) has been completed, there is one event for each service event to be sent
- synch data event: if new data have been supplied via Device Access, they will become active during the next cycle; this event is processed by all PZs
The
White Rabbit PZ just behaves like the seven PZ. It is supplied via Device Access and is connected to the internal MIL Event Bus to receive the events from SPZ. There are two main difference to the seven PZs.
- there is only one White Rabbit network for all seven timing areas at UNILAC
- White Rabbit PZ uses the concept of an alarm based timing system unlike the seven PZs, that are event based
Stack of the White Rabbit Pulszentrale
Figure: Overview on the White Rabbit Pulszentrale. Shown are SCU (blue box), host system and FPGA (orange boxes), standard IP cores instantiated in the FPGA (light green boxes), components by the
WR-UNIPZ (yellow boxes) and components of the old GSI Control System (dark green boxes). Indicated is data flow from the
Super Pulszentrale (thick black arrows), a Lemo cable (thin blue arrow), Wishbone and Etherbone communication (thin black arrows) and
Timing Messages sent to the White Rabbit network (thick blue arrow). Details see text.
One of the main challenges of this development was to bring together two different worlds: The old GSI
Device Access control system and the very different ecosystem of the White Rabbit based developments. The low level interface to timing system components is Etherbone. The high level standard API is saftlib. However, WR UNIPZ uses a third approach, namely the use of a shared library. This disentangles the two distinct ecosystem as much as possible:
- Developing, building of the firmware and tools works independently of the old GSI control system.
- Developing, building of the Device Access class only requires including one header file and linking against a single shared library.
Regular SCU
The basis of WR-UNIPZ is a regular SCU as Frontend with Timing Receiver. Hardware, gateware and the RunTime Environment (RTE: drivers + tools) are identical to other frontends. This is configured and rolled-out like all other SCU based frontends.
WR-UNIPZ: Firmware and Tools
Very tightly coupled to the gateware is firmware and low-level software tools of WR UNIPZ that are hosted in the same source code control and build environment as gateware and RTE. It is part of the White Rabbit based ecosystem. The tools contain all a script and helper software to upload the firmware to a lm32 soft CPU embedded in the VHDL of the regular SCU during runtime. Moreover, the script performs basic configuration of firmware, IOs ports and the ECA. Configuration and roll-out works via nfs-init (as usual) and includes a shared library
wrunipzlib.so
as API. This shared library hides implementation details as much as possible from the user. Included as well is a CLI for testing and diagnosis.
Precise timestamping needs to be done for the 'cycle start event' received from the SPZ via MIL bus. This is achieved by the Timestamp Latch Unit (TLU) of the ECA. The 'MIL Piggy' generates a TTL pulse on an output when the cycle start event is received. Via a LEMO cable, this TTL signal is connected to an input configured as TLU input of the ECA. This solution provides timestamping in hard real time by the gateware, indepently of the firmware.
PZUD: Device Access Class
The equipment class
pzud
is part of the Device Access environment and part of the equipment model pzu.
pzud
only uses the shared library
wrunipzlib.so
as the sole interface to the lm32 firmware in the FPGA. Basically,
pzud
can be seen as a wrapper around that shared library.
WR-UNIPZ
I (db) prefer the following workflow
- Coding, building on my main linux development machine. For testing the firmware and
socat
enabled on a test SCU, it is sufficient to eb-fload
the firmware and to use the CLI wrunipz-ctl
via tcp.
- Once the firmware, library and tools are mature I commit and push to the GIT repo.
- Rollout needs to be done the ASL7 cluster. I (db) never code on the ASL but just use it as a building and deployment system after pulling the working environment from the GIT repo.
This workflow has two advantages: It allows me to work on my favorite working environment and assures a roll-out is done with sources that committed and pushed.
Coding and Building
For cloning the repo and setting up the tool chain just follow the instructions of the
current release. The gateway must follow the current timing release very closely. It typically resides in a branch that stems from the current release branch.
- Convention: branch is named
wrunipz_dietrich_< YYYY-MM-DD >
Depending on your favorite working environment, consider to follow the directions given
here. Then
cd bel_projects/modules/wr-unipz // main working folder of wr-unipz
ls
asl // scripts for roll-out on the asl cluster
fw // firmware
include // header files
Makefile // the big makefile, with directions for ASL on the top of the file
Manifest.py
x86 // x86 software
make // builds firmware (calls 'fw/Makefile') and software (calls 'x86/Makefile')
Rollout for ACO
Rollout on a SCU in the ACO environment requires working on the ASL 7 cluster. There are two steps. First, rebuilding of binaries (firmware and CLI). Second, prepare 'nfsinit' for a specific SCU.
Remark: There is no dedicated FPGA image, the gateway uses the default SCU image of the
current release. The startup script (called during nfsinit) replaces the firmware of the user lm32 during run-time.
For cloning the repo and setting up the toolchain just follow the instructions of the
current release. Consider to follow the directions given
here.
Building and Copying
- here is the Makefile //.../bel_projects/modules/wr-unipz/Makefile
- try head Makefile to check the parameters of the Makefile
- build the Firmware and Software
-
DEV
: try make clean
-
DEV
: try make MASP=YES PRO=NO PREFIX= all
-
PRO
: try make clean
-
PRO
: try make MASP=YES PRO=YES PREFIX= all
- (just in case: building on a non-ASL environment is as easy as for other eb-tools when using MASP=NO)
- deploy Firmware, Gateware and scripts
-
DEV
: try make MASP=YES PRO=NO PREFIX= STAGING=/common/export/timing-rte/wrunipz-dev deploy
-
PRO
: try make MASP=YES PRO=YES PREFIX= STAGING=/common/export/timing-rte/wrunipz deploy
nfsinit
- cd to the nfsinit folder of the relevant SCU
- create the relevent symbolic link
-
DEV
try 30_wrunipz -> ../global/timing-rte-wrunipz-dev
-
PRO
try 30_wrunipz -> ../global/timing-rte-wrunipz
- note the target of the symbolic link and the folder names above
PZUD
In case you are not familiar with Device Access and the build environment on the ASL
>>> 6 <<< cluster, checkout
this.
Standard Device Properties
Some info on coding...
pzud-device.cc provides some data and standard device properties
- data
-
ebDevice
, handle to Etherbone device of pzud
-
pz
, number of PZ (0..6); there is one instance of the PZUD class for each PZ
- methods
-
resetDevice
, init of all get-values; here: reset diagnostic values (for all PZ)
-
initDevice
, as reset
(for all PZ), but init of all set-values too; here: clears event data (for THIS PZ)
-
updateStatus
, is triggered periodically to get and update the device status; here: get basic info (for ALL PZ, 'opReady'....)
-
onlineTest
, is triggered periodically to get and update the device status; here: get very basic info (for ALL PZ, check if firmware is reachable)
-
getVersionString
, obtains version info; here: PZUD, version, shared library
USRs (User Service Routines) provide methods dedicated to PZUD only; USRs are defined in the file
pzud.xml
-
pzud-write-evtsequ.cc
, writes event data for a specific virtual accelerator and channel to the firmware, important method used by the 'UNILAC Super Pulszentrale'
-
pzud-read-evtsequ.cc
, reads event data for a specific virtual accelerator and channel to the firmware
-
pzud-read-diag.cc
, reads diagnostic info of firmware (for ALL PZ)
-
pzud-write-cmdcleardiag.cc
, clears diagnostic info of firmware (for ALL PZ)
-
pzud-write-cmdsubmit.cc
, submits/activats event data previously written to the PZs (for ALL PZ); remark: this method is only useful when playing with PZUD via the PropHelper. The 'Super Pulszentrale' uses a dedicated event on the internal MIL bus to which PZUD is connected.
Interface to the lm32 Firmware in the FPGA
There is only very little code implemented by PZUD. Basically, PZUD is just a wrapper around a shared library 'wrunipzlib.so'. This library provides the interface to the firmware including all communication. The library and headere file are provided here
-
/common/export/timing-rte/wrunipz-dev
; development system Programmentwicklungsraum
-
/common/export/timing-rte/wrunipz
; production system LSB6
The Device Access class PZUD just includes the header and links against the shared library (see the info on 'SConscript'
here).
Building and Roll-Out
Follow these
directions.
--
DietrichBeck - 29 Apr 2020