Testing and Commissioning Guide for Pexaria5

---

---

---

Required components

• Pexaria5 under test (named as P5UT in this guide)
• PC with Intel Quartus (version 18.0+) and Xilinx ISE
• Oscilloscope (to measure 200 MHz pulses)
• USB blaster (Altera or terasIC)
• Xilinx USB programmer (Platform Cable USB II, Model: DLC10) + adapter cable
• PROMO5 adapter
• USB cables A/B and A/micro-B
• LEMO cable(s)
• 12V power supply
• SFP (green or purple) + fiber cable
• timing receiver with 3 LVTTL IOs (AMC, PEXP, Exploder5, ...)
• White Rabbit switch (at least with firmware v4.2)

Prepare the required binary images and CLI tools

The gateware release Fallout has been used for testing. For older commissioning guide click here.

1. Build and install required Etherbone and Saft tools

$ git clone https://github.com/GSI-CS-CO/bel_projects.git # clone a repo
$ cd bel_projects
$ git checkout fallout                          # check out a release branch
$ make clean                                    # clean repo
$ make etherbone                                # Etherbone library
$ make tools                                    # Etherbone tools
$ make driver                                   # Linux drivers
$ sudo make install                             # install them
$ make saftlib-clean
$ make saftlib                                  # build Saftlib library, tools
$ make saftlib-install                          # install them
(optional) Use insmod to load the drivers oder restart

Testing steps

Setup

1. Place powered-off Pexaria5 on ESD-desk, ESD dischare!
2. Make sure the board has a serial number and a CID label
3. Remove the Pexaria5dDBx add-on board (if any is mounted)

Attention:

• $dev_usb is a placeholder for the USB device: eg, export dev_usb="dev/ttyUSB0"
• $dev_pcie is a placeholder for the Wishbone device: eg, export dev_pcie="dev/wbm0"

Programming the CPLD device

1. Plug Xilinx USB Programmer to the JTAG1 connector
2. Verify switch positions of the PROMO5 adapter: SEL1=1, SEL2=4
3. Launch Xilinx ISE and load a projec 'bel_projects/syn/gsi_pexarria5/cpld/pexaria5_prog.xise'
4. Build the CPLD configuration: Process Menu -> Implement Top Module
5. Start the programmer tool: Tools Menu -> Impact
6. Attach 12V power cable to the AUXPOW1 input plug and turn on power
7. Detect on-board CPLD: double-click Boundary Scan
8. Load the previously generated configuration: Control-I -> prog1.jed
9. Start programming of CPLD: Operations Menu -> Program
10. Turn power off when programming is successfully completed
11. Un-plug Xilinx USB Programmer

Programming the FPGA device (not-persistent)

1. Plug the Altera/terasIC USB Blaster to the JTAG1 connector
2. Verify switch positions of the PROMO5 adapter: SEL1=2, SEL2=8
3. Turn power on
4. Invoke following commands (or use GUI commands):

$ quartus_pgm -c 1 -m jtag -o 'p;pexarria5.sof'

1. Unplug the USB Blaster
2. If Quartus Programmer (GUI) is unable to scan device chain (Processing -> Auto Detect), then verify the USB Blaster configuration

$ lsusb | grep Altera
Bus 003 Device 004: ID 09fb:6001 Altera Blaster
$ dmesg | grep Altera
[  107.989799] usb 3-3: Manufacturer: Altera
$ dmesg | grep 'usb 3-3'
[  107.837138] usb 3-3: new full-speed USB device number 4 using xhci_hcd
[  107.989791] usb 3-3: New USB device found, idVendor=09fb, idProduct=6001
[  107.989795] usb 3-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  107.989797] usb 3-3: Product: USB-Blaster
[  107.989799] usb 3-3: Manufacturer: Altera
[  107.989801] usb 3-3: SerialNumber: 91d28408

Programming the USB chip

1. Make sure no other timing receiver is connected with USB
2. Now connect P5UT to PC via USB by plugging an USB cable to the USBCON1 connector
3. Change to 'bel_projects/ip_cores/etherbone-core/hdl/eb_usb_core'
4. Invoke following commands:

$ make
$ sudo ./flash-fx2lp.sh -E # erase the USB controller (as root)
$ sudo ./flash-fx2lp.sh    # program the USB controller (as root)

1. Turn power off

Configuring (and programming) the SPI Flash chip

1. Turn power on
2. Program the FPGA again:

$ quartus_pgm -c 1 -m jtag -o 'p;pexarria5.sof'

1. Configure the SPI flash chip:

$ sudo eb-config-nv $dev_usb 10 4

Programming the FPGA device (persistent)

1. Write *.rpd bit-stream into the SPI flash, if USB and flash chips are programmed:

$ sudo eb-flash $dev_usb pexarria5.rpd

1. Do power cycle
2. Verify the gateware version

$ sudo eb-info $dev_usb
Project     : pci_control
Platform    : pexaria5 +db[12] +wrex1
FPGA model  : Arria V (5agxma3d4f27i3)
Source info : fallout-3264
Build type  : fallout-v6.0.0               # gateware version
...

Formatting the WR EEPROM (MAC address is stored here)

1. Go to 'bel_projects/ip_cores/wrpc-sw/tools'
2. Invoke following commands:

$ make
$ sudo ./eb-w1-write $dev_usb 0 320 < sdb-wrpc.bin

Check EEPROM (and set MAC address)

1. Run the EB console and set a MAC address:

$ sudo eb-console $dev_usb
mac setp <aa:bb:cc:dd:ee:ff>  # set real MAC address for aa:bb:cc:dd:ee:ff

1. Press 'CTRL+c' to quit the EB console
2. Check given MAC address (if address is just set, then it can be checked only after power cycle!)
3. Do power cycle
4. Run the EB console and check the previously set MAC address:

$ sudo eb-console $dev_usb
mac                 # you should see the MAC address

1. Quit the EB console
2. Turn power off
3. Mount the Pexaria5bDBx add-on board
4. Turn power on (FPGA power ~600mA/12V)

Check OneWire Devices

check with 'onewire-scanner'

1. Go to 'bel_projects/tools/commissioning/onewire-scanner'
2. Invoke following commands:

$ make
$ sudo ./onewire-scanner $dev_usb

Output should look like this (OWID should be slightly different): Scanning for OneWire controller(s) on dev/ttyUSB0 now... ID Wishbone Address OWID Serial Code Type ------------------------------------------------------------------------------- 00 0x0000000000060600 -- --- --- -- --- 00 0x3700000a62e07928 DS18B20 - Digital Thermometer -- --- 01 0x9c0000006cfb1e43 DS28EC20 - 20Kb EEPROM Found 1 OneWire controller(s) on dev/ttyUSB0. Found 2 OneWire device(s)/slave(s) on dev/ttyUSB0.

check with 'eb-mon'

1. Get the serial code of temperature sensor:

$ sudo eb-mon $dev_usb -w0 -b0 -f0x28  # should return 0x3700000a62e07928

1. Get the actual temperature:

$ sudo eb-mon $dev_usb -w0 -t0 -f0x28  # return like 45.1875

1. Get the serial code of EEPROM:

$ sudo eb-mon $dev_usb -w0 -b0 -f0x43  # should return 0x9c0000006cfb1e43

Check White Rabbit

1. Insert SFP and connect DUT to a WR port (in master mode) of an operating White Rabbit switch
2. Start the EB console and show the White Rabbit status:

$ sudo eb-console $dev_usb
gui

1. Check 'Mode: WR slave Locked Calibrated'
2. Check 'PTP status: slave'
3. Check 'Clock Offset <= 10ps' for at least 30 seconds
4. Check if the board gets an IP address: if the MAC address is unknown or not yet registered, then 'IPv4' status has BOOTP RUNNING message
5. Un-plug and re-plug the fiber cable several times and make sure WR locks again and 'Servo state' gets 'PHASE_TRACK' (state transitions are SYNC_NSEC -> SYNC_PHASE -> WAIT_OFFSET_STABLE -> TRACK_PHASE)
6. Press 'ESC' to quit the gui environment
7. Power cycle the device and repeat step #4 > Look at on-board LEDs D1-D4

Synchronization status should be: + Servo state: TRACK_PHASE + Phase tracking: ON You should also see 4 leds at the front panel: + red (D1) = traffic/no-link + blue (D2) = link + green (D3) = timing valid + white (D4) = PPS

1. Quit only the gui environment (press 'ESC') and check the timing mode

mode            # should return 'running; e2e slave'

1. Remove fiber cable
2. Set the master mode:

mode master     # should return 'PTP stop Locking PLL...' if it could be able to lock PLL and get master

1. Check if P5UT could be able to get 'master' mode
2. Quit the console (CTRL+C)
3. Turn power off

Check IOs (using SAFT tools)

1. Start the Saftd daemon with P5UT and TR2 (eg, test PC hosts TR2):

$ sudo saftd p5ut:$dev_usb tr2:$dev_pcie # ensure that you have chosen right devices by checking the gateware info

Front Panel IOs

1. Make 1:1 connections between P5UT and secondary timing receiver (TR2): connect IO1 (P5UT) to IO1 (TR2) and so on for IO2 and IO3
2. (for P5UT) Start 1 MHz clock on each IO:

$ saft-io-ctl p5ut -n IO1 -o 1            # turn on the output enable (red LED is on)
$ saft-clk-gen p5ut -n IO1 -f 1000000 0   # start clock

1. (for TR2) Snoop inputs:

$ saft-io-ctl tr2 -s

1. Measure jitter and signal quality with a scope (FTRN specification should be matched)

$ saft-clk-gen p5ut -n IO1 -s             # stop clock

1. Repeat the test for IO2 and IO3

Slew Rate

1. Measure the slew rate with a scope (for EVERY IO), should be equal to Exploder5 (TODO: provide specification)

IO Bandwidth

1. Generate a 200MHz clock and measure the clock with a scope:

$ saft-clk-gen p5ut -n IO1 -f 200000000 0     # start clock
$ saft-clk-gen p5ut -n IO1 -s                 # stop clock

1. ( Not needed) Generate a 125MHz clock with TR2 and measure it with the ECA:

$ saft-io-ctl tr2 -n IO1 -o 1                 # TR2: turn on the output enable (red LED is on)
$ saft-clk-gen tr2 -n IO1 -f 125000000 0      # TR2: generate 125 MHz clock
$ saft-io-ctl p5ut -n IO1 -o 0                # P5UT: turn off the output enable (red LED is off)
$ saft-io-ctl p5ut -s                         # P5UT: snoop inputs

IO Load Test

1. Drive every IO with termination and measure voltage level with a scope (3,04V/3,36V with/without termination):

$ saft-io-ctl p5ut -n IO1 -o 1 -t 0 -d 1      # termination off
$ saft-io-ctl p5ut -n IO1 -o 1 -t 1 -d 1      # termination on

1. Make sure that termination functions
2. Repeat the test for IO2 and IO3

Output Enable Test

1. Drive every IO with and without output enable
2. Make sure you don't see a level change when output enable is turned off, output should stay at 0 voltage
3. Check the LEDs, they should indicate every activity (blue) and the output enable status (red)

Check LEDs

1. Generate a pulse per second:

$ saft-pps-gen p5ut -s

1. Leave the application running and check all LEDs
2. Press 'CTRL+c' to stop the pulse generation

Check PCIe/PCI (P5UT is installed into the test PC)

1. Start the Saftd daemon with P5UT:

$ sudo saftd p5ut:$dev_pcie        # ensure that you have chosen a right device

1. Invoke the following commands:

$ sudo eb-console $dev_pcie
$ sudo eb-info $dev_pcie
$ sudo eb-ls $dev_pcie

1. Write a test pattern to internal shared RAM and read it back:
   • Get the LM32 shared RAM address:

$ sudo eb-ls $dev_pcie | grep RAM  # eg, 12.14.5   0000000000000651:54111351   4060000  LM32-RAM-User

• • Create a dummy file (which will be written into the RAM):

$ dd if=/dev/urandom of=/tmp/foo bs=4k count=1

• • Write dummy file to the lm32 shared ram:

$ sudo eb-put $dev_pcie 0x4060000 /tmp/foo

• • Get the data from the shared ram:

$ sudo eb-get $dev_pcie 0x4060000/4096 /tmp/bar

• • Compare both files:

$ cmp /tmp/foo /tmp/bar

• • Both files should contain the same data
  • Repeat this test with the attached 'test_ram.sh' script (edit device name if needed)

Check PCIe/PCI interrupts

1. Make sure P5UT is connected to a White Rabbit switch and WR is locked
2. Create your own schedule (or use the attached 'schedule.txt')
3. Start snooping for events:

$ saft-ctl p5ut snoop 0 0 0 -d

1. Inject events (from the second terminal):

$ saft-dm p5ut -fp -n 1 schedule.txt

1. Verify that all events are snooped
2. Press 'CTRL+c' to stop the snooper

-- EnkhboldOchirsuren - 07 Oct 2020

• test_ram.sh: Shell script to test the shared RAM via the host PCIe/PCI interface