Documentation Data Master Test System

Hardware

SuperMicro fel0069

The data master for the test system is hosted on SuperMicro fel0069 with two PEXARIA5d (fel0069.acc.gsi.de). One of these is the data master. The other one is used to analyse the timing messages with snoop. The SuperMicro is accessible with ssh root@fel0069.acc via ASL cluster or other hosts. The management interface (ILO) is accessible via https://fel0069i.acc.

1. fel0069.acc.gsi.de
   • IP: 10.248.2.137
   • Location: BG2.009, Rack BG2A.A9, Slot 28
2. pexaria248t (dev/wbm0), Data master
   • IP: 192.168.131.184
   • MAC: 00:26:7b:00:08:0b
   • Name: pexaria248t
   • CID: 55 0113 0012 0
   • PEXARIA5d, Serie EE
3. pexaria305t (dev/wbm1), Timing receiver for snoop
   • IP: 192.168.131.241
   • MAC: 00:26:7b:00:08:44
   • Name: pexaria305t
   • CID: 55 0113 0069 4
   • PEXARIA5d, Serie EE

After power on, set the IP addresses of the two pexarrias. This host has no BootP service.

The SuperMicro is configured for PXE boot and nfsinit with links (following https://www-acc.gsi.de/wiki/Timing/Intern/TimingSystemHowToHintsForFECS).

1. On ASL cluster links in folder /common/tftp/csco/pxe/pxelinux.cfg for PXE boot: 8CB5A992 to fel0069, fel0069 to scuxl.fallout
2. On ASL cluster links in folder /common/export/nfsinit/fel0069/ for nfsinit.
3. On ASL cluster create folder /common/fesadata/data/fel0069/.

Connections: network cable to acc network. For management: network cable to acc network.

After reboot of fel0069:

1. Set the IP addresses of the two pexarria5. This is done in the WRPC init scripts of the two pexarria5.
2. Start saftd: saftd tr1:dev/wbm1
3. On each host, which needs access to fel0069, copy the public ssh key to fel0069. This has to be done for every user / host which needs access to fel0069 with ssh.

   ssh-copy-id -i .ssh/id_rsa.pub root@fel0069.acc

4. Test (check before automated testing starts)

   ssh -t root@fel0069.acc.gsi.de "saft-ctl tr1 -xv snoop 0 0 0 3"

   (snoop on tr1 for all events for 3 seconds)

There is a script in bel_projects, modules/ftm/test/tools/datamasterInit.sh. This initializes fel0069 after reboot / power on. It copies some public ssh keys to .ssh/ on fel0069. If this is NOT started from ac0pc042.gsi.de, there are adoptions necessary. datamasterInit.sh also compiles the lm32 firmware located in modules/ftm/ftmfw. Call

datamasterInit.sh FEL0069 ftm.bin 8

for 8 thread firmware or

datamasterInit.sh FEL0069 ftm.bin 32

for 32 thread firmware.

White Rabbit Switch nwt0473m66

Location: BG2.009, Rack BG2A.A9, Slot 29
Configuration: blank. Not an access switch or distribution switch!
Access via tsl101, like other switches.
Name: nwt0473m66.timing.acc.gsi.de,
IP: 192.168.21.219.
Connections: fibre optic cable from wri2 to pexaria248t, fibre optic cable from wri3 to pexaria305t. Network cable to acc network for management.

---

Software

Firmware Images

Datamaster: build with make ftm in bel_projects root folder. Current version (example):

Project     : ftm
Platform    : pexaria5 +db[12] +wrex1
FPGA model  : Arria V (5agxma3d4f27i3)
Source info : fallout-3295
Build type  : developer preview
Build date  : Wed Oct 28 16:45:33 CET 2020
Prepared by : Martin Skorsky 
Prepared on : ACOPC042
OS version  : Linux Mint 19.3 Tricia, kernel 4.15.0-122-generic
Quartus     : Version 18.1.0 Build 625 09/12/2018 SJ Standard Edition

  176fccd2 pmc: changed seed
  d3ab3c45 vetar2a-ee-butis: changed seed
  5419e84e microtca: changed seed
  c0d44bd6 Merge pull request #260 from GSI-CS-CO/dm-fallout-merge-v2
  50444f38 scu4: changed seed

Detecting Firmwares ...

Found 4 RAMs, 4 holding a Firmware ID


********************
* RAM @ 0x04120000 *
********************
UserLM32
Stack Status:
Project     : ftm
Version     : 7.0.1
Platform    :
Build Date  : Wed Oct 28 16:45:19 CET 2020
Prepared by : martin Martin Skorsky 
Prepared on : ACOPC042
OS Version  : Linux Mint 19.3 Tricia  Linux 4.15.0-122-generic x86_64
GCC Version : lm32-elf-gcc(GCC)4.5.3 (build 190527-673a32-f3d6)
IntAdrOffs  : 0x10000000
SharedOffs  : 0x500
SharedSize  : 98304
FW-ID ROM will contain:

   176fccd2 pmc: changed seed
   d3ab3c45 vetar2a-ee-butis: changed seed
   5419e84e microtca: changed seed
   c0d44bd6 Merge pull request #260 from GSI-CS-CO/dm-fallout-merge-v2
   50444f38 scu4: changed seed
*****

Pexaria dev/wbm1 for snoop: https://github.com/GSI-CS-CO/bel_projects/releases/download/fallout-v6.0.1/pexarria5.rpd

Update: https://github.com/GSI-CS-CO/bel_projects/releases/download/fallout-v6.1.2-rc1/falloutv6_1_2-rc1-pexarria5.rpd

Access the Data Master

dm-cmd tcp/fel0069.acc
dm-sched tcp/fel0069.acc

Remote Snoop of Timing Messages

Snoop per remote ssh: Set up of ssh without password: use public key of user@host and transfer it to root@fel0069.acc: /.ssh/authorized_keys with ssh-copy-id. Snoop with Python3: the tests using python3 / pytest read the command for snooping from environment variable SNOOP.

• Example for local environment: saft-ctl tr0 -xv snoop 0 0 0
• Example for remote environment on fel0069.acc: ssh -t root@fel0069.acc 'saft-ctl tr1 -xv snoop 0 0 0'
  saftd on fel0069.acc monitors dev/wbm1 as tr1.

The tests add an additional parameter for the number of seconds to snoop.

Required versions

1. Python 3.6.9
2. pytest 6.1.2

Jenkins on builder.acc.gsi.de / Jenkins Slave tsl021.acc.gsi.de

There are Jenkins jobs on https://builder.acc.gsi.de, which build the datamaster lm32 firmware and the tools and run tests on these binaries. The jobs have the steps

1. Preparation
2. Build the tools
3. Build the lm32 firmware with lm32-toolchain and load this into the datamaster fel0069.acc.gsi.de
4. Run the tests with these components
5. Report test results

There are jobs for the branches fallout, dm-summer-update-2022, dm-merge-Dez23, dm-fallout-tests for 8 threads and 32 threads.

The script for the build steps is

# create links needed for Rocky-9 environment
cd res/rocky-9
./generate_soft_links.sh
export PATH=$PATH:$(pwd)
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)
cd ../..
./fix-git.sh
# make all prerequisites: 1. hdlmake and lm32-toolchain, 2. etherbone, 3. eb-tools, 4. test-tools for ftm.
make
make etherbone
make tools
cd modules/ftm/tests
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$WORKSPACE/ip_cores/etherbone-core/api/.libs/:$WORKSPACE/modules/ftm/lib/
EBPATH1=../../../ip_cores/etherbone-core/api make prepare
export PATH=$PATH:$WORKSPACE/tools/:$WORKSPACE/modules/ftm/bin/:$WORKSPACE/modules/ftm/analysis/scheduleCompare/main/
# build ftm lm32 firmware
THR_QTY=8 PATH=$PATH:$HOME/.local/bin:$WORKSPACE/lm32-toolchain/bin/ make -C $WORKSPACE/syn/gsi_pexarria5/ftm/ ftm.bin
# load the required lm32 firmware into fel0069
$WORKSPACE/syn/gsi_pexarria5/ftm/fwload_all.sh tcp/fel0069.acc.gsi.de $WORKSPACE/syn/gsi_pexarria5/ftm/ftm.bin
# run all tests
OPTIONS='--runslow' make remote

Links for the Jenkins Jobs

1. https://builder.acc.gsi.de/jenkins/job/timing/view/tests/job/test_module_ftm_datamaster_dm-fallout-tests/ All tests in modules/ftm/tests, branch dm-fallout-tests.
2. https://builder.acc.gsi.de/jenkins/job/timing/view/tests/job/test_module_ftm_datamaster_dm-fallout-tests_thread32/ All tests in modules/ftm/tests, branch dm-fallout-tests, ftm.bin modified for 32 threads.
3. https://builder.acc.gsi.de/jenkins/job/timing/view/tests/job/test_module_ftm_datamaster_dm-merge-Dez23/ All tests in modules/ftm/tests, branch dm-merge-Dez23.
4. https://builder.acc.gsi.de/jenkins/job/timing/view/tests/job/test_module_ftm_datamaster_dm-merge-Dez23_thread32/ All tests in modules/ftm/tests, branch dm-merge-Dez23, ftm.bin modified for 32 threads.
5. https://builder.acc.gsi.de/jenkins/job/timing/view/tests/job/test_module_ftm_datamaster_dm-summer-update-2022/ All tests in modules/ftm/tests, branch dm-summer-update-2022.
6. https://builder.acc.gsi.de/jenkins/job/timing/view/tests/job/test_module_ftm_datamaster_dm-summer-update-2022_thread32/ All tests in modules/ftm/tests, branch dm-summer-update-2022, ftm.bin modified for 32 threads.
7. https://builder.acc.gsi.de/jenkins/job/timing/view/tests/job/test_module_ftm_datamaster_fallout/ All tests in modules/ftm/tests, branch fallout.

The jobs use a lockable resource named 'fel0069' to ensure that only one job at a time runs tests against datamaster fel0069.acc.gsi.de. Running jobs concurrently against a datamaster breaks the tests.

Preparation of user jenkins on tsl021 for remote snoop. This has to be done once with a Jenkins job since there is no way to log on as user jenkins without sudo rights.

1. Add the known host key for fel0069.acc.gsi.de to ~/.ssh/known_hosts for user jenkins on tsl021.

   cat modules/ftm/tests/tools/fel0069_known_host >> ~/.ssh/known_hosts

2. If not existing, generated the ecdsa id for ssh for user jenkins on tsl021.

   test -f ~/.ssh/id_ecdsa || ssh-keygen -t ecdsa -f $HOME/.ssh/id_ecdsa

3. Copy the id to root@fel0069.acc.gsi.de.

   ssh-copy-id -i $HOME/.ssh/id_ecdsa.pub root@fel0069.acc.gsi.de

4. Test that remote snoop works (run saft-ctl for a second).

   ssh root@fel0069.acc.gsi.de 'saft-ctl tr1 -xv snoop 0 0 0 1'