Remote debugging of a running saft-daemon with GDB

It is possible to connect GDB to a running SCU, set breakpoints, step through the code, print variables, etc. This is a step-by-step guide

Deployment

The normal deoployment of timing-RTE contians a saftlib that is build with the "-O2 -g" compiler options. That means, debug symbols are already there, but optimization is switched on. It is possible to connect to such a saftlib build, but some symbols may be missing.

In order to get the best debugging experience, a special depolyment with "-O0 -g" should be done.

1. Login to asl cluster
2. clone the ci_cd repository: https://github.com/GSI-CS-CO/ci_cd
3. cd ci_cd/scripts/deployment/RTE_Timing
4. modify the build-rte.sh
   1. At the top of the file, set the BEL_BRANCH variable to the desired branch in the bel_projects git repository, and the DEPLOY_TARGET directory. You have to create a new deploy directory or re-use an existing one. In this example: DEPLOY_TARGET="/common/export/timing-rte/tg-saftlib-gdb
   2. Further down in the file (in the section #building saftlib), modify the call to make by adding CXXFLAGS='-O0 -g'. The line should now look like this: make CXXFLAGS='-O0 -g' DESTDIR=$RTE_DIR install
5. ./build-rte.sh

Preparation

1. On the asl cluster, go to the nfsinit directory of the target SCU (scuxl0089 in this example): cd /common/export/nfsinit/scuxl0089/
2. set the 20_timing-rte symlink to point to the debug deployment that was just created: 20_timing-rte -> ../global/timing-rte-tg-saftlib-gdb
3. reboot the target SCU
4. login to the target SCU
5. start gdbserver and leave it running: gdbserver scuxl0089:2345 --attach `pidof saftd`

Debugging

1. In a different window: login to asl cluster and cd into the directory where the debug deployment was done: cd ci_cd/scripts/deployment/RTE_Timing
2. launch gdb: gdb
   
3. enter the following commands in the GDB-console:
   1. set sysroot rte-build
   2. file rte-build/sbin/saftd
   3. set solib-search-path rte-build/lib
   4. target remote scuxl0089:2345

Now, the saftd process on the SCU is interrupted, and you can set breakpoints (break saftlib::MasterFunctionGenerator::Flush), step through the code (next), or continue the execution (continue)

-- MichaelReese - 13 Oct 2021