How-To: WB_FEC

This wiki provides information about

• WB FEC VHDL module and integration in Bel_Projects
• Synthesis of Timing Receiver Gateware with the WB FEC
• Simulation environment of the WB FEC
• Demonstration of the WB FEC

The theory and technical specs of the FECo (Forward Error Correction) will be provided in an incoming PhD Thesis.

WB_FEC

The FEC here presented is based in the Fixed Rate Code. The FEC encoder divides the original frame in 4 blocks uses these blocks to encoded the data. The encoding process happens on the fly and only needs to know 1 cycle in advance the length of current frame to calculate the encoding parameters. Finally 4 FEC packets are created and send to the network. The pic below shows a Ethernet and frame and FEC frame.

The doc FEC_Encoding present in a table how the information of the original frame is encoded into 4 frames and how the decoding process is done depending on which frames has reached the decoder.

The pic below shows the integration of the FEC in VHDL System of GSI Front End Controller. The FEC placed between the WR Core and the Etherbone Core encodes and decodes the incoming encoded frames. If an incoming frame is not encoded, the WR Fabric MUX used in the FEC will forward the frame through the WB FEC without decoding. In transmission works in the same way. The WR Fabric MUX routes frames according to the packet class specified in the Status Word of the incoming frame on the WR Fabric. The Status Word for Rx Frames is added by the End Point in the EPFilter module, and it be configured modifying the rules programmed by the _wrpc-sw. For Tx Frames the WR Frabric master must add themselves the Status Word defining the packet class.

For more information about the format and uses of the WR Frabric Status Word check the WR doc.

For more information about the EPFilter rules, read the file wrpc-sw/tools/pfilter-builder.c It is self-documented.

A system using the configuration of WR/FEC/EB in the previous pic can handle only 2 streams of data. A stream of data encoded y another not encoded. If you need more than 1 stream of FEC data you may add another decoder. The pic below shows an example.

In the case of >2 streams of FECs (B2B System or Machine Protection) you need to work with VLANs. The EPFilter while not using VLAN, classifies the frames checking the Ethertype and MAC address of the frames and it is able only to classify and forward two streams of traffic. In turns, using VLAN enables the possibility of classifying and forwarding more than 2 streams. For more information about VLANs in the WR Core check the docu.

The Table below shows the memory map of the Wishbone Register of the WB_FEC:

WB SLAVE	Register	Value
GSI:FEC	0x0	Enable/Disable FEC
		0x0 Disable Enc/Dec
		0x1 Enable Enc/Disable Dec
		0x2 Disable Enc/Enable Dec
		0x3 Enable Enc/Dec
	0x4	FEC Type
	0x8	FEC Ethertype
	0xC	Ethernet Ethertype
	0x10	Num of Encoded Frames
	0x14	Num of Decoded Frames
	0x18	Err Jumbo Frames in Rx
	0x1C	Decoding Errors > 2 FEC pkt lost
	0x20	Encoding Errors & Jumbo Frames

Synthesis the Timing Receiver Gateware with the FEC

I have integrated the WB_FEC in the BEL_PROJECTS repository under the module folder. The code is not in Master yet, I let you decide when/who to merge it with the master branch. So far is in the wb_fecs branch

https://github.com/GSI-CS-CO/bel_projects

The gateware synthesized and tested use the Explder5 and Pexaria projects. It fits perfectly in the FPGA and Quartus doesn't complain about timing violations. If you want to synthesize the design, just follow the usual procedure to build the bitstream in bel_projects.

Simulation environment of the WB FEC

If in the future you find any bug in the WB FEC module, the System Verilog WB FEC simulation environment is your best friend.

Before you start fiddle around with the environment I advice you to check a short tutorial of System Verilog. It will take you a couple of hours but it pays off. Besides, I advice you to clone my fork of the WR Core, my simulations uses modified components from it (e.g. WB driver and modules).

You can see the architecture of the test-bench is the image below:

System Verilog Test Bench

• It provides the modules to create Ethernet packets and send them over the White Rabbit fabric.
• It implements Wishbone protocol, it is quite handy for changing the config parameters of the WB_FEC during the simulation
• It controls the White Rabbit fabric
• It implements and automatic check to validate the encoding/decoding process and data

WB_FEC

• Encoder and decoder the incoming frames/packets

Packet Dropper

• It simulates loss of packets in the network.
• You can configure which packets are going to drop. It can also drop more than 2 packets to force errors.

WR Fabric Loop-Back

• It receives the encoded packets over the the WR fabric and send its back to the decoder.

Setting up the Simulation Environment

First of all you have to clone a fork of the wr-cores project in the :

https://github.com/GSI-CS-CO/wr-cores-fec

The WB_FEC repository is added as a module to the wr-core fork:

https://github.com/GSI-CS-CO/wb_fec

If you don't know how to set up WR simulation environment, please check this tutorial:

Once you have done the previous steps:

cd wr-cores-fec/ip_cores/wb_fec/testbench/fec
hdlmake
make
sudo vsim

Modelsim will start and then type in the console window of the Modelsim

do run.do

the simulation will start you will see the stream of frames getting encoded and decoded.

DEMO

I have developed a demo to show in operation the WB FEC. As you can see in the pic below the Exploder5 A is going to send encoded broadcast, EB commands, over a WR Switch to the Exploder5 B and Standar PC. You can also interact with the WB slaves on the Exploder5 over the WR network from a standar PC using the EB commands eb-ls, eb-read etc... This EB traffic is not encoded and it is not going to be decoded in the Exploder5 B. This proves full-duplex and multiplexing of streams between encoded and non-encoded packets in the WB FEC module. In order to simulate the losses of packets in a network a special VHDL component is added to the tx path, the Packet Dropper module. This module drops packets after being encoded. You can configure the packets that you want to drop over a WB Interface. A Test Module is also added to the design, Test_Module. This module is the target of the EB commands send by the Exploder5 A. The EB commands are write commands containing the value of the number of FEC packets sent by the Exploder5 A. Finally a LM32 firmware is used to send the EB commands from the Exploder5 A.

To set up the demo, please grasp 2 Exploder5 connect them to a WR Switch. Program the Exploder5 with the this firmware. If you want to build yourself the firmware checkout from the bel_project repo the tag wb_fec_demo

git clone https://github.com/GSI-CS-CO/bel_projects
git checkout fec_demo

and keep building following the usual procedure to synthesize a form factor in bel_projects. Otherwise you can use the bitstreams already available:

• pci_control.rpd.xz: Demo Pexaria Bitstream
• pci_control.sof.xz: Demo Pexaria Bitstream - Altera
• exploder5_csco_tr.rpd.xz: Demo Exploder Bitstream
• exploder5_csco_tr.sof.xz: Demo Exploder Bitstream - Altera

One of the Exploder is going to be the sender of Etherbone commands and it is going to encode them. So that you have to load the firmware in the_ User LM32_ to send the Etherbone commands. Please connect the Exploder to the EB USB port, open the serial console

$minicom -D /dev/ttyACM0

and type in the console:

wrc#ip set 192.168.1.5

program the second Exploder5 and connect to the USB serial port and assign another IP

$minicom -D /dev/ttyACM1
wrc#ip set 192.168.1.3

the open another terminal and load the firmware for sending encoded EB commands:

$cd bel_projects/module/wb_fec/code/
$make
#eb-fwload dev/ttyUSB0 u 0x0 fectest.bin

You will see a message in the WR console:

FEC Test Packet Gen Ready, sending

Now the Exploder5 A (192.168.1.5) is sending encoded frames to the Exploder5 B over the switch.

Checking the Encoding/Decoding

Once the Exploder5 A is sending and encoding the EB commands frames, you can check the counters of both the encoder and decoder.

#while true; do eb-read dev/ttyUSB0 0x8020010/4; sleep 1; done

#while true; do eb-read dev/ttyUSB1 0x8020014/4; sleep 1; done

You should see the counters with the some values and increasing at the same time.

The address of the WB Slaves that I'm using can vary, please check with eb-ls the address of the WB FEC.

Testing Loss of Packets

Once that the Exploder5 A is sending and encoded frames to the Exploder5 B, you can simulate the the losses of packets during operation. A special module, Packet Dropper, drops packets according to the desire configuration. The table below shows the WB register available for configuring the drops of packets:

WB SLAVE	Register	Value
GSI:PACKET_DROPPER	0x0	Drop Packets
		0x0 Drop No Packets
		0x2 Drop FEC Packet 1
		0x6 Drop FEC Packet 1 and 2
		0x1 Drop FEC Packet 0
		0x5 Drop FEC Packets 0 and 2
		0x3 Drop FEC Packets 0 and 1
		0x7 Drop FEC Packets 0,1 and 2 = FEC Error
	0x4	Enable/Disable Packet Dropper
		0x0 Disabled
		0x1 Enabled

Now we can test the FEC forcing the lost of packets 0 and 2.

eb-write dev/ttyUSB0 0x8040000/4 0x5

You can check that that the Exploder5 A is sending just 2 FEC packets if you connect a PC to the WR Switch snoop with Wireshark the traffic. If you check again the Encoder/Decoder counters it should keep growing at the same time.

Now lets see how the FEC fails. If more than 2 packets get lost, the decoder can't decode and retrieve the original packets, let's try it with three packets losses:

eb-write dev/ttyUSB0 0x8040000/4 0x7

If you check again the Encoder/Decoder counters, only the encoder counter keep growing, the decoder is idle, and the error counter of the decoder will show that the decoder can't decode original packet:

while true; do eb-read dev/ttyUSB1 0x802001C/4; sleep 1; done

If you set again a Packet Dropper configuration with only 2 packets losses, you will see how the error counter is idle and decoder counter is growing again.

FEC Terminal Interface

I have developed a small eb program to check the configuration and status information of the FEC. You can find it in our repo:

$cd bel_projects/tools/fec
$make
#./eb-fec dev/ttyUSBX

the daemon starts and will show you all the information about the FEC:

------- FEC Info dev/ttyUSB1 ------
# FEC Configuration
Encoder: ON -- Decoder: ON
FEC Ethertype: 0xbabe 
Etherbone Ethertype: 0x800 
# FEC Stats
Encoder Packets: 0
Decoder Packets: 42714
Error Decoder: 12366
Error Encoder: 0
Jumbo Packet Rx: 0

Testing Non-Encoded Traffic

During normal operation of the encoding/decoding you may want to read/write a WB Slave register in a Front End Controller, or just get information of the firmware flashed, list of WB slaves etc... Since the FEC routes the encoded and non-encoded traffic this is possible. Check it out:

eb-ls udp/192.168.1.5

You should see the a list of WB Slaves devices.

Potential Issuses

Since the WR Frabric is shared between the WR PTP and external modules like the Etherbone Slave or Master, it may happen that the EB components block for two long the transmission of PTP frames and causes the lost of synchronization. This happens if the tx rate of the EB devices is high, how much? I have tested with 300 Mb/s and I lost sync in the nodes in 120 sec... One solution could be to reduce the rate of the PTP handshake in the WR PTP, so the probability of blocking the fabric for the PTP frames is lower, as result less probability of screwing the synchronization.

How-To: WB_FEC (OLD WIKI: TO BE MERGED WITH THE STUFF ABOVE)

This wiki provides the information about

• Status of the WB_FEC
• Synthesis the Timing Receiver Gateware with the WB_FEC
• Simulation environment of the WB_FEC

The theory and technical specs of the FECo (Forward Error Correction) will be provided in an incoming PhD Thesis.

Status of the WB_FEC

The FEC here presented is based in two codes:

• Golay Code
• Fixed Rate Code

The Golay Code is used to repair bit errors in the Ethernet frame and the Fixed Rate Code to overcome the loss of packets in the network.

The Fixed Rate Code encodes the original information contained in one frame into four frames. The doc FEC_Encoding present in a table how the information of the original frame is encoded into 4 frames and how the decoding process is done depending on which frames has reached the decoder.

Both codes are working reliable (>48h test) in simulation. The tests carried out with real hardware is limited since there is not so a easy way for testing the encoder, although it has been tested and it works reliable.

Basically the Fixed Rate Code needs to know the length of the incoming frame. The FEC is a layer 2 encoder/decoder. The FEC encoder divides the original frame in 4 blocks. The FEC uses these blocks to encoded the data. The encoding process happens on the fly and needs to know 1 cycle in advance the length of current frame to calculate the encoding parameters. If the information it is not provided in the Ethertype field (which is still standard today) should store the frame payload in another fifo.

So far the Data Master doesn't provide this information in the Ethertype yet. The length of frame is needed to fulfill the FARI requirements of encoding latency and RAM allocation in the FPGA.

TODO List:

• WB_FEC Documentation (PhD Thesis)
• Saftlib tool (?)

Synthesis the Timing Receiver Gateware with the FEC

I have integrated the WB_FEC in the BEL_PROJECTS repository under module folder. The code is not in Master yet, I let you decide when/who to merge it with the master branch. So far is in the wb_fecs branch

https://github.com/GSI-CS-CO/bel_projects

The gateware synthesized and tested use the Explder5 and Pexaria projects. It fits perfectly in the FPGA and Quartus doesn't complain about timing violations.

If you want just to see how is integrated check the monster module. The pic below shows how the WB_FEC is connected in the design:

If you want to synthesize the design, just follow the usual procedure to build the bitstream in bel_projects.

Simulation environment of the FEC

If in the future you find any bug in the FEC module, the System Verilog FEC simulation environment is your best friend.

Before you start fiddle around with the environment I advice you to check a short tutorial of System Verilog. It will take you a couple of hours and you will understand better the code. Besides I advice to check the simulation project of the WR Core, my environment uses some components from it (e.g. WB driver and modules).

You can see the architecture of the test-bench is the image below:

System Verilog Test Bench

• It provides the modules to create Ethernet packets and send them over the White Rabbit fabric.
• It implements Wishbone protocol, it is quite handy for changing the config parameters of the WB_FEC during the simulation
• It controls the White Rabbit fabric
• It implements and automatic check to validate the encoding/decoding process and data

WB_FEC

• Encoder and decoder the incoming frames/packets

Packet Dropper

• It simulates loss of packets in the network.
• You can configure which packets are going to drop. It can also drop more than 2 packets to force errors.

WR Fabric Loop-Back

• It receives the encoded packets over the the WR fabric and send its back to the decoder.

Setting up the Simulation Environment

First of all you have to clone a fork of the wr-cores project in the :

https://github.com/GSI-CS-CO/wr-cores-fec

The WB_FEC repository is added as a module to the wr-core fork:

https://github.com/GSI-CS-CO/wb_fec

If you don't know how to set up WR simulation environment, please check this tutorial:

https://www-acc.gsi.de/wiki/Timing/WRSimulation

Once you have done the previous steps:

cd wr-cores-fec/ip_cores/wb_fec/testbench/fec
hdlmake
make
sudo vsim

Modelsim will start and then type in the console window of the Modelsim

do run.do

the simulation will start you will see the stream of frames getting encoded and decoded.

DEMO

A Demo has been set up to show the FEC in operation.

To set up a basic Demo, please grasp 2 Exploder5 connect them to a WR Switch. Program the Exploder5 with the this firmware. If you want build yourself the firmware checkout from the bel_project repo the tag FEC_DEMO

git clone https://github.com/GSI-CS-CO/bel_projects
git checkout fec_demo

and keep building following the usual procedure to synthesize a form factor in bel_projects.

One of the Exploder is going to be the sender of Etherbone messages and it is going to encode them. You have to load the firmware in the User LM32 to send the Etherbone messages. Connect to the EB USB port, open the serial console a

$minicom -D /dev/ttyACMX 

and type in the console:

wrc#ip set 192.168.1.3

the open another terminal:

$cd bel_projects/module/wb_fec/code/
$make
#eb-fwload dev/ttyX u 0x0 fectest.bin

You will see a message in the WR console:

FEC Test Packet Gen Ready, sending

Q&A

Mathais Kreider - I know I said it can be done, but I still don't get why. Eth type/len field is at offset 20, IP TOL at 24 (ebm doesn't use VLAN). Why would 2 cycles more latency matter enough to use non standard format?

• If the FEC relies on the IP header can't encode Ethernet layer 2 frames and there are other layer 2 applications (B2B, Machine protection) that would like to use the FEC. CERN is also interested in FEC and they are not going to use IP
• The FEC encodes on the fly the payload as soon as it gets to the encoder. Since I have to split the payload in 4 equal blocks I need to know the length before the payload starts. If I wait to get the length from the IP header I have to store the payload in a FIFO (another FIFO) and you well know that the generic FIFOs nowadays allocate 1 Kbit no matters how small we want them. I wanted to keep the footprint of the FEC as small as possible since the DM is running out of RAM...
• Finally, the length in the ethertype is also standard, actually you told me this, I didn't know it

-- CesarPrados - 14 Dec 2018

-- CesarPrados - 14 Dec 2018