microIOC

Inbetriebnahme-Tipps fuer neue Schrittmotor-Systeme

• neue MicroIOCs / PMACs am Testsystem vollstaendig (!!!) anschliessen und mit LocalControl GUI konfigurieren.
• FE:
  • MAC-Adressen LAN1, LAN2, x-port fuer DHCP-server (-> ACO INN)
  • PXE-Verzeichnis, Links fuer IP (LAN1), OS Image-link
  • NFSINIT-Verzeichnis /common/export/nfsinit/ anlegen, Links entsprechend setzen
  • entsprechende FESA-Software (einzelne Motoren oder Schlitze???) in /common/export/fesa/local// bereitstellen und konfigurieren, Instanz mit Geraetenomenklaturen in DB eintragen
  • Schrittmotor-Konfigurationsdateien in /common/fesadata/data//slits-conf/ bereit stellen, Geraetenomenklaturen eintragen

Checkliste für Inbetriebnahme und Konfiguration von neuen MicroIOCs

Quick Setup of new MicroIOCs (cloning from tstci26)

./ibn.sh <hostname> <IP address> # setup microioc
./ibn.sh <hostname> <IP address> del # remove setup of microioc

Local Control GUI

• Launch in HKR
  • open Launcher: /common/usr/cscoap/bin/launcher pro
  • Tab Verschiedenes > Experten Apps > Schrittmotor-LocalControl GUI
  • Tab Verschiedenes > Terminal, run
  • /usr/bin/webstart https://websvcdev.acc.gsi.de/groups/cscofe/applications/StepperMotor_LocalControl/StepperMotor_LocalControl.zip
  • see: ~workspaceMotionControl/LocalControlGUI_webserver/StepperMotor_LocalControl
  • since 2022: needs to be uploaded to webserver https://websvcdev.acc.gsi.de/groups/cscofe/applications/StepperMotor_LocalControl/

LCDisplay Control

• activation:
  • old: press fn button + button for two seconds
  • new: press green tick button + right arrow button for two seconds

MicroIOC System Control

Backup of properties.ini files

./backup_properties.ini -t/c/f/u/s

Status Checks

./status_test.sh
./status_cry.sh
./status_frs.sh
./status_unilac.sh
./status_sfrs.sh

Check of initialization files/software installation

./check_init_files_microIOCs.sh -t/c/f/u/s # check installation for asl74x-cluster
./check_init_files_microIOCs.sh -t/c/f/u/s -n # check installation for asl75x-cluster/yocto

Web Interface

Reboot MicroIOCs

./reboot_microiocs.sh -t/c/f/u/s

Sources

Driver + FESA Software

Local Control GUI

System Documentation By Manufacturer

System Control

Troubleshooting

Network Connection

MicroIOC nicht im ACC Netzwerk erreichbar

• Verkabelung (fehlerhafte/falsche LAN-Kabel), ACC Switch ports
• CF Karte noch eingesetzt, Boot-Konfiguration im BIOS fehlerhaft
• Keine DHCP-Addresse: Reset ACC Network Switch
• Fehlerhafter DHCP-Eintrag, MicroIOC erscheint mit falschem Hostnamen, oder gar nicht. (-> ACO/IN)
• Fehlerhafter PXE-Eintrag (-> ACO/IN/FEC)
• DHCP im BIOS nicht aktiviert?
• Falsches Datum/falsche Uhrzeit im BIOS
• Falsches Boot-Setting (Launch PXE OpROM: disabled -> enabled)
• Verlängerungskabel / Adapter funktioniert nicht (cross-over Adapter nicht nutzbar!)

MicroIOC bootet nicht mehr nach hardware powercycle

• BIOS-Einstellungen fehlerhaft (falsches Datum, falsches Boot-Medium eingestellt (sollte sein: PXE))
• Mainboard-Batterie leer/Spannung zu gering, sollte ~3 V sein, nicht angeschlossen am Mainboard (!?)

MicroIOC bootet nicht, Netzteil-Lüfter läuft, keine Ausgabe der Boot-Mitteilungen

• Stromversorgungskabel an Grafikkarte?
• Mainboard defekt -> Cosylab

Keine Verbindung möglich

• Ping und Verbindung mit MicroIOC funktioniert nicht: ACO/IN pruefen lassen, ob MAC-Adressen richtig eingetragen sind
• Verbindungsversuch mit LocalControl GUI scheitert:
  • Netzwerkkabel ist abgezogen
  • System ist fehlerhaft konfiguriert, z.B. fehlende Parameter in Konfigurationsdatei (-> Fehlermeldung deutet fälschlicherweise auf serielle Verbindung hin, bis 2020)
  • Encoder type in properties.ini ist falsch eingestellt. Muss ggfs. in properties.ini direkt eingetragen werden da Verbindung mit LocalControl GUI nicht möglich ist wegen seriellem Verbindungsfehler. (bis 2020)

• • System ist fehlerhaft verkabelt (aussen). Darauf achten dass jeweilige Buchsen für Potentiometer oder SSI-Gerät verwendet werden.
  • PMAC/MicroIOC ist ausgeschaltet...
  • Serielle Stecker oder Stromversorgungskabel innen (!) im MicroIOC nicht richtig aufgesteckt
  • Konfiguration: wenn nicht alle Motoren via Kabel angeschlossen sind, sollten sie z.B. als PMAC konfiguriert sein (z.B. motor8.encoderType=3).

PDC Connection

Serielle Schnittstelle

• Tauchen Fehler beim Zugriff auf die serielle Schnittstelle auf, etwa der Art

  ERROR Exception at: src/SerialComm.cpp:610, Description: SerialComm::serialRead: /dev/ttyCTI1 read timeout
  
        kapci01          Error, command: readSystemConfig  Exception: Exception at: src/Device.cpp:72, Description: Driver initialization failed: Exception at: src/SerialComm.cpp:610, Description:       SerialComm::serialRead: /dev/ttyCTI1 read timeout      

• Error 300

14:54:25 - An error occurred while sending the following command to the server: readSystemConfig 

ERROR: Exception at: src/Device.cpp:70, Description: Driver initialization failed: Exception at: src/SharedMemory.cpp:34, Description: Failed to create MBox driver: Exception at: src/SerialComm.cpp:76, Description: SerialComm::connect: /dev/ttyCTI0 open failed: No such file or directory

[root@microioc~]# dmesg | grep tty
[    0.000000] Command line: initrd=scuxl/current/initramfs panic=10 console=ttyS3,115200 8250.nr_uarts=20 BOOT_IMAGE=scuxl/current/kernel 
[    0.000000] Kernel command line: initrd=scuxl/current/initramfs panic=10 console=ttyS3,115200 8250.nr_uarts=20 BOOT_IMAGE=scuxl/current/kernel 
[    0.000000] console [ttyS3] enabled
[    1.886977] 00:04: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    1.913313] 00:05: ttyS2 at I/O 0x3e8 (irq = 10) is a 16550A
[    1.939718] 00:06: ttyS3 at I/O 0x2e8 (irq = 10) is a 16550A
[    1.966072] 00:07: ttyS4 at I/O 0x2c0 (irq = 10) is a 16550A
[    1.992533] 00:0f: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.002274] 0000:04:07.0: ttyS5 at MMIO 0xfe700000 (irq = 19) is a XR17D15X
[    2.009652] 0000:04:07.0: ttyS6 at MMIO 0xfe700200 (irq = 19) is a XR17D15X
[    2.017068] 0000:04:07.0: ttyS7 at MMIO 0xfe700400 (irq = 19) is a XR17D15X
[    2.024449] 0000:04:07.0: ttyS8 at MMIO 0xfe700600 (irq = 19) is a XR17D15X
[   23.751852] 0000:04:07.0: ttyCTI0 at MMIO 0xfe700000 (irq = 19) is a XR17xx5x
[   23.761101] 0000:04:07.0: ttyCTI1 at MMIO 0xfe700200 (irq = 19) is a XR17xx5x
[   23.770495] 0000:04:07.0: ttyCTI2 at MMIO 0xfe700400 (irq = 19) is a XR17xx5x
[   23.779416] 0000:04:07.0: ttyCTI3 at MMIO 0xfe700600 (irq = 19) is a XR17xx5x
[root@microioc~]# echo "gv" > /dev/ttyCTI0 | cat /dev/ttyCTI0
Rgv:HW-REV.B+FW-2.2

[root@microioc~]# echo "gv" > /dev/ttyCTI1 | cat /dev/ttyCTI1
gv
[root@microioc~]# dmesg | grep tty
[    0.000000] Command line: initrd=scuxl/current/initramfs panic=10 console=ttyS3,115200 8250.nr_uarts=20 BOOT_IMAGE=scuxl/current/kernel 
[    0.000000] Kernel command line: initrd=scuxl/current/initramfs panic=10 console=ttyS3,115200 8250.nr_uarts=20 BOOT_IMAGE=scuxl/current/kernel 
[    0.000000] console [ttyS3] enabled
[    1.879477] 00:04: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    1.905747] 00:05: ttyS2 at I/O 0x3e8 (irq = 10) is a 16550A
[    1.932122] 00:06: ttyS3 at I/O 0x2e8 (irq = 10) is a 16550A
[    1.958489] 00:07: ttyS4 at I/O 0x2c0 (irq = 10) is a 16550A
[    1.984901] 00:0f: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    1.994585] 0000:04:07.0: ttyS5 at MMIO 0xfe000000 (irq = 19) is a XR17D15X
[    2.001965] 0000:04:07.0: ttyS6 at MMIO 0xfe000200 (irq = 19) is a XR17D15X
[    2.009372] 0000:04:07.0: ttyS7 at MMIO 0xfe000400 (irq = 19) is a XR17D15X
[    2.016733] 0000:04:07.0: ttyS8 at MMIO 0xfe000600 (irq = 19) is a XR17D15X

• Die fehlende PDC SSI version HW-REV.B+FW-2.2 deutet auf ein Problem mit der PDC hin
  • entweder PDC selbst (defekt / veraltete Version / inkompatibler ADC)
  • Verkabelung
  • serielle Schnittstelle innerhalb der MicroIOC
• Ursache für die fehlerhafte Kommunikation mit der seriellen Schnittstelle kann auch ein fehlender Treiber sein. Dieser kann nur über eine http-Verbindung geladen werden, nicht über eine https-Verbindung:

<b>OK:</b>
[root@microioc~]# opkg update || opkg install kmod-scu-ctiserial
Downloading http://packages/opkg/el7/x86_64/Packages.
Updated list of available packages in /usr/lib/opkg/lists/gsi-arch.
Downloading http://packages/opkg/el7/all/Packages.
Updated list of available packages in /usr/lib/opkg/lists/gsi-all.
Package kmod-scu-ctiserial (3.10.101-03) installed in root is up to date.

List available drivers

opkg list | grep kmod

Check Adressbelegung serielle Schnittstellen

root@<FEC-name>:dmesg | grep tty
[    0.000000] Command line: initrd=scuxl/current/initramfs panic=10 console=ttyS3,115200 8250.nr_uarts=20 BOOT_IMAGE=scuxl/current/kernel 
[    0.000000] Kernel command line: initrd=scuxl/current/initramfs panic=10 console=ttyS3,115200 8250.nr_uarts=20 BOOT_IMAGE=scuxl/current/kernel 
[    0.000000] console [ttyS3] enabled
[    1.919329] 00:04: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    1.945671] 00:05: ttyS2 at I/O 0x3e8 (irq = 10) is a 16550A
[    1.972038] 00:06: ttyS3 at I/O 0x2e8 (irq = 10) is a 16550A
[    1.998414] 00:07: ttyS4 at I/O 0x2c0 (irq = 10) is a 16550A
[    2.024823] 00:0f: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.034455] 0000:04:07.0: ttyS5 at MMIO 0xfe700000 (irq = 19) is a XR17D15X
[    2.041828] 0000:04:07.0: ttyS6 at MMIO 0xfe700200 (irq = 19) is a XR17D15X
[    2.049211] 0000:04:07.0: ttyS7 at MMIO 0xfe700400 (irq = 19) is a XR17D15X
[    2.056563] 0000:04:07.0: ttyS8 at MMIO 0xfe700600 (irq = 19) is a XR17D15X
[   24.083838] 0000:04:07.0: ttyCTI0 at MMIO 0xfe700000 (irq = 19) is a XR17xx5x
[   24.093475] 0000:04:07.0: ttyCTI1 at MMIO 0xfe700200 (irq = 19) is a XR17xx5x
[   24.102557] 0000:04:07.0: ttyCTI2 at MMIO 0xfe700400 (irq = 19) is a XR17xx5x
[   24.112141] 0000:04:07.0: ttyCTI3 at MMIO 0xfe700600 (irq = 19) is a XR17xx5x
root@<FEC-name>:setserial -g /dev/ttyS[0..8]
[root@microioc~]# setserial -g /dev/ttyS0
/dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
[root@microioc~]# setserial -g /dev/ttyS1
/dev/ttyS1, UART: 16550A, Port: 0x02f8, IRQ: 3
[root@microioc~]# setserial -g /dev/ttyS2
/dev/ttyS2, UART: 16550A, Port: 0x03e8, IRQ: 10
[root@microioc~]# setserial -g /dev/ttyS3
/dev/ttyS3, UART: 16550A, Port: 0x02e8, IRQ: 10
[root@microioc~]# setserial -g /dev/ttyS4
/dev/ttyS4, UART: 16550A, Port: 0x02c0, IRQ: 10
[root@microioc~]# setserial -g /dev/ttyS5
/dev/ttyS5, UART: undefined, Port: 0x0000, IRQ: 19
[root@microioc~]# setserial -g /dev/ttyS6
/dev/ttyS6, UART: undefined, Port: 0x0000, IRQ: 19
[root@microioc~]# setserial -g /dev/ttyS7
/dev/ttyS7, UART: undefined, Port: 0x0000, IRQ: 19
[root@microioc~]# setserial -g /dev/ttyS8
/dev/ttyS8, UART: undefined, Port: 0x0000, IRQ: 19

dmesg | grep tty
[    0.000000] console [tty0] enabled
[    0.780225] 00:04: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    0.801057] 00:05: ttyS2 at I/O 0x3e8 (irq = 10) is a 16550A
[    0.821851] 00:06: ttyS3 at I/O 0x2e8 (irq = 10) is a 16550A
[    0.842663] 00:07: ttyS0 at I/O 0x2c0 (irq = 10) is a 16550A
[   24.016347] 0000:04:07.0: ttyCTI0 at MMIO 0xfe000000 (irq = 19) is a XR17xx5x
[   24.028275] 0000:04:07.0: ttyCTI1 at MMIO 0xfe000200 (irq = 19) is a XR17xx5x
[   24.040097] 0000:04:07.0: ttyCTI2 at MMIO 0xfe000400 (irq = 19) is a XR17xx5x
[   24.052005] 0000:04:07.0: ttyCTI3 at MMIO 0xfe000600 (irq = 19) is a XR17xx5x

• Eine alternative Fehlerquelle ist das verwendete serielle Kabel. Es sollte das richtige sein. -> SD/LOBI/AOBI
• (vor 2020: Möglicherweise ist auch die properties.ini-Datei die Fehlerursache. Es sollten keine Syntaxfehler enthalten sein. Es sollte der jeweils aktuelle Stand verwendet werden, z.B. inklusive Angaben für 'named positions' etc!!!)

SSI encoder comm / Potentiometer comm error

• PDC off
• PDC not connected to MicroIOC (cables?)
• PDC broken

Potentiometer comm error / PMAC comm error

• 'old' PDCs may have an incompatible ADC version, serial connection errors become visible on console when mode is switched to SSI; motors can still be operated in PMAC mode (disadvantage: position unknown)
• inside microIOC:
  • serial connection cable might not sit properly on two-lined plug socket
  • sometimes only reconnecting the plugs of the cables on the PCI-104-board helps
  • PCI-104 serial card might be broken
  • MicroIOC -internal cables, especially in older systems, might have an 'invisible corrosion' issue ("crevice corrosion"?!). It might help to simply reconnect the cables at their plug sockets.
  • cabling between PCI-104 card and connector on backside might have a defect
• software configuration, /dev/ttyCTI0 appears as /dev/ttyCTI1 for an unknown reason

SSI Encoder Comm Error

• set both encoder type settings to 'potentiometer' only in local configuration

No connection to PMAC

• socket connection problems: MicroIOC of a more recent generation (2nd) have an additional network interface built-in, the network interfaces have to be configured differently compared to the first version

log 'configuring network'
# configure internal network interface
if /sbin/ifconfig eth2; then
 log "Device eth2 exists, dealing with new version M-Box (LCD CFA533), setting eth2 as PMAC's interface."
 /sbin/ifconfig eth2 192.6.94.1 netmask 255.255.255.0 up
else
 log "Device eth2 doesn't exist, dealing with old version M-Box (LCD LCM02), setting eth1 as PMAC's interface."
 /sbin/ifconfig eth1 192.6.94.1 netmask 255.255.255.0 up
fi

[root@microioc~]# ifconfig
eth0     Link encap:Ethernet HWaddr 00:18:7D:A7:22:68
         inet addr:140.181.168.12 Bcast:140.181.191.255 Mask:255.255.192.0
         UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
         RX packets:19149182 errors:0 dropped:4606237 overruns:0 frame:0
         TX packets:10377500 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:1587088574 (1.4 GiB) TX bytes:14005087882 (13.0 GiB)
eth1     Link encap:Ethernet HWaddr 00:18:7D:A7:22:69 
         UP BROADCAST MULTICAST MTU:1500 Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000 
         RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth2     Link encap:Ethernet HWaddr 00:0B:AB:BB:8B:F4 
         inet addr:192.6.94.1 Bcast:192.6.94.255 Mask:255.255.255.0
         UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
         RX packets:92579581 errors:0 dropped:0 overruns:0 frame:0
         TX packets:34732985 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000 
         RX bytes:5611335928 (5.2 GiB) TX bytes:2264005048 (2.1 GiB)
         Memory:fe300000-fe400000 
lo       Link encap:Local Loopback 
         inet addr:127.0.0.1 Mask:255.0.0.0
         UP LOOPBACK RUNNING MTU:65536 Metric:1
         RX packets:0 errors:0 dropped:0 overruns:0 frame:0
         TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0 
         RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

PMAC connection issues

2020-09-23T10:52:23.361Z  yr07ci01             lcd                 
yr07ci01 lcd: Exception at: lcd.cpp:724, Description: main: error:Exception at: src/SharedMemory.cpp:34, Description: Failed to create MBox driver: Exception at: src/PmacComm.cpp:86, Description: PmacComm::doConnect: connect failed: No route to host

• this could be an internal network interface card issue (hardware broken, internal cables of microioc, misconfiguration?).

2024-07-05T08:37:00.13836 tstci26             lcMonitor_cfa53    Exception at: src/lcMonitor.cpp:157, Description: Failed to create MBox driver: Exception at: src/PmacComm.cpp:86, Description: PmacComm::doConnect: connect failed: Connection timed out Hint: Check eth network interfaces are configured correctly or PMAC may not be available

• this is a symptom of a missing serial driver, check with

dmesg | grep tty

Axis Interlock / Amplifier Fault

• Interlock-Stecker (2-polig Lemo am PDC Eingang des Kanals) zum Überbrücken des Interlock-Ausgangs
• cable between MicroIOC and PMAC controller connected?

Amplifier Fault

• 2023: may lead to an interlock which makes motor unusable remotely via DeviceControl
• if the amplifier fault is the only error displayed per motor it could be that it can be removed by moving the motor locally via LocalControl
• if moving the motor via LocalControl is not the solution to remove the amplifier mode it may help to configure the motor for PMAC mode. -> encoder problem

Software Troubleshooting

OS Image cannot be loaded

• error message during boot: Could not find kernel image: rescue/kernel
  • CSCOIN needs to set a cryptic link to point the IP (gethostip) to the hostname of the Micro IOC in /common/tftp/csco/pxe/pxelinux.cfg

FESA/!DeviceControl/FESA Explorer: partial setting not allowed

• [RdaException] FESA_13008 Partial setting not allowed:
  • set-rda3-c++ -d -p MoveToPosition -v / JAPC call: uebergebener Japc-Parameter: /!MoveToPosition#name
    uebergebener Wert: (String)
    
    [RdaException] FESA_13008 Partial setting not allowed. src/fesa-core/Server/ServerData.cpp:254
    • wrong/missing parameter: >set-rda3-c++ -d -p MoveToPosition -f positionName -v
      Sync SET completed: /!MoveToPosition
      RequestContext [selector=; filters=; data=]

Since 2023: How to upload FESA instances to the FESA database

/common/usr/fesa/bin/FESAClassUploader -s pro subset/DeviceData_MotorSlit_DU.instancesubset

Since 2023: how to upload a FEX zip file to the webserver

curl -u matthies:<PWD> -T /home/bel/matthies/lnx/workspaceMotionControl/MotorSlit_DU/src/test/kyr3ci01/MotorSlit_DU_kyr3ci01.zip https://websvcdev.acc.gsi.de/groups/fesa/fex/

Device is not visible in DeviceControl

• check FESA/LSA databases

fesa-class-info -p <devicename>
cd workspaceMotioncontrol
./doesLSAknow.sh <devicename>

LCDisplay does not work

• lcd software did not start. Check content of /common/export/nfsinit//60-slits-lcd:

#!/bin/sh
. /etc/functions

log 'launch lcd after FESA software to display more information on the MicroIOCs display'
log 'sleep 45 s'
sleep 45
log 'starting lcd'
daemon -n lcd -U -O daemon.debug -E daemon.err -- \
    /usr/bin/lcd_cfa533 -c /etc/slits-conf/lcd.ini

• check whether built version of 'lcd' matches the OS image libraries
• check with ldd whether all the required libraries are available on the OS, e.g. the slits driver etc
• hardware issue? internal power connection?

Error message: "Failed to kill motor"

• update of internal pmac code with PMACUtils required

Motor/Slit Moves

Motors move in one direction only, no matter the software settings or the connected motor / PDC

• check cable between motor and PDC, wrong pins might be connected

Motors don't move, no errors

• hardware reset may help (turn MicroIOC completely off using the switch in the back)
• sometimes a PMAC controller reboot may help
• screws between motor and drive loose (symptoms: motors receive commands either via FEX or Local Control GUI, but the position does not change; no errors reported)

Motors move via LocalControl, but not via DeviceControl

• step motor system is probably fine
• communication error between DeviceControl and FESA software? Network, JAPC, ...
• FESA software down
• FESA software is throwing exceptions for a reason
• Motors are stuck at outer borders

Motors don't move via DeviceControl

• no Local Control GUI instance should be connected to the stepper motor system in local control mode. Either switch to remote mode or wait 5 min until this happens automatically.

Motors don't move via DeviceControl and are in an outer position

• Motors may be in outer end position (check Status). In this case it may happen that motors are not moveable anymore in remote mode. A workaround is to switch the system to local mode and move the motor manually out of its impractical position. (observed on FRS and S-FRS systems)
• the motor configuration should be checked, especially the outer limits, the offsets, the position tolerances; perhaps a misconfiguration leads to the observed effect

Motor Pair / Slit does not move - no error message

• one (or both) motors may be in an outer end position (check Status). In this case it may happen that motors are not moveable anymore in remote mode. A workaround is to switch the system to local mode and move the motors manually out of their impractical position. (observed S-FRS systems)
• technically it is possible that the motor pair is not able to be moved to the center position / width because inner/outer limits are/will be reached. Try other settings or move the motors separately. No error is issued in this case.

Motors both in inner/outer HW/SW end position at the same time

• sure sign that no motor hardware is connected, cable not connected, connection between motor and PMAC controller interrupted, possible symptom: motor LED off on MicroIOC
• check, whether serial cables are connected according to motor configuration

Motor moves in one direction only, no matter what

• pins might be wrongly connected in cable between motor and PDC (the self-manufactured ones)
• or, there could also be a connector issue on the motor connection side
• wrong software configuration settings (limits, direction, etc.)

Motors move in opposite direction

• software configuration: pulse polarity (negative/positive)
• hardware bug: motor connections pins might be switched the wrong way (not necessarily the cable, but the motor connector, e.g. at the SFRS stepper motor system "media board")

Motor does not move fast enough with heavy load under vacuum, slips through

• check velocity settings and acceleration time, should be slower (e.g. half of set values)

Attempting to move a motor pair via FESA Explorer / DeviceControl: "FESA_10016 Invalid value: '0.0500..0003" not in range '[0; 0.005499..97]'

• software settings misconfiguration??? Check settings in LocalControl GUI; inner and outer limit settings must be defined properly and should not be mixed up!
• check: inner/outer limits, high/low limit offsets might be switched, low limit should be negative, high limit positive, position tolerances
• invalid motor positions, inability to move the motor pair???
• check polarity of low limit offset, should be negative
• exceptions are thrown in real time action, they stop subscriptions via JAPC, not via RDA

LocalControl GUI: actual position cannot be displayed on motor drive tab (-> N/A)

• software configuration: SSI encoder type selected?
• cables (axis) between MicroIOC and PDC not properly connected?

LocalControl GUI: actual and set positions don't match

• software configuration:
  • limit configuration?
  • motor counts
  • potentiometer length
  • steps per mm
  • offsets
  • position tolerances
  • slower velocity and acceleration time to avoid slips during moves of heavy loads under vacuum
• hardware
  • pin connections?
  • cable?

Subscriptions don't receive updated values of property parameters, get operations impossible

• thrown exceptions/conditions in RT action may stop the subscription notification thread

MicroIOC -related

MicroIOC cannot be turned on anymore: fuse

LEDs on MicroIOC

Shared Memory Access Issues

[root@<FEC name> ~]# /usr/bin/lcMonitor_cfa533 -c /etc/slits-conf/lcmonitor.ini&
[root@<FEC name> ~]# Exception at: lcMonitor.cpp:147, Description: Failed to create MBox driver: Exception at: src/SharedMemory.cpp:483, Description: semget failed: File exists : 17
Creating shared memory in error state.
terminate called after throwing an instance of 'MBoxNameSpace::AccessControlException'
  what():  Exception at: src/SharedMemory.cpp:483, Description: semget failed: File exists : 17

'Error reading from SSI device'

2020-03-06T07:04:19.988Z  tstci17              lcd
tstci17 lcd: Exception at: lcd.cpp:724, Description: main: error:Exception at: src/SharedMemory.cpp:34, Description: Failed to create MBox driver: Exception at: src/PmacComm.cpp:86, Description: PmacComm::doConnect: connect failed: No route to host

[root@<FEC name> ~]# ipcs
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x000004d2 32768      root       666        24648      6                       
------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00003039 32768      root       666        1         
------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages    
[root@tstci16 ~]# ipcrm -m 32768
[root@tstci16 ~]# ipcrm -s 32768

. checkAndClearSharedMemory.sh

X-Port / m64 / remote access

Logging in to the X-Port

name@asl74x: telnet <nomenclature>m64
Trying <IP>...
Connected to <nomenclature>m64.
Escape character is '^]'.
Connection menu: (select by number)
  1) microIOC                             2) Exit to CLI
  3) Log out                            
Selection = 

Further information on X-Port handling

Reset SBC/PMAC via X-Port

WR-Timing-related errors

SAFTD:ERROR | TIM_SYNC:DIAG_UNKNOWN | TIM_LOCK:DIAG_UNKNOWN

matthies@asl749 ~ $ sss fel9OI7
Enter passphrase for key '/home/bel/matthies/lnx/.ssh/id_rsa':
Creating commands: ll, la, cdf, cdi, cdt, diag, inst
can't open connection to device dev/wbm0 (system failure)
eb-info: dev/wbm0: system failure
can't open connection to device dev/wbm0 (system failure)
expr: syntax error
expr: syntax error-FEC status: SAFTD:ERROR | TIM_SYNC:DIAG_UNKNOWN | TIM_LOCK:DIAG_UNKNOWN
FEC uptime: 15 min, 1 users, load average: 0.06, 0.02, 0.03
DU running: No daemon with DU running.
FPGA specs: Not available.
SAFT build: fallout-v6.2.0 (git-hash: d95c321)
NFS link : fel9OI7 (/opt/nfsinit/fel9OI7)
OS release: GSI embedded release 7 (build 2021-10-26)-
Aborted
ScuBus Tag: Error reading tag configuration. Init links could be corrupted.

[root@fel9OI7 ~]# saft-ctl tr0 -i
terminate called after throwing an instance of 'std::runtime_error'
 what(): Cannot connect to socket. Possible reasons: all sockets busy, saftd not running, or wrong socket permissions
Aborted

FESA device-instance configuration:
NO FTRN in global device ("false")
TriggeredEC, AcquisitionUpdateEC nicht in device-configuration mappen (nichts mit Timing-Configuration, nur Timer-gesteuerte Eventkonfigurationen verwenden)

• check WR switch connection (cable connected?)
• check WR switch port (ACO-TG)
• reboot the microIOC system:

[root@fel9OI7 ~]# eb-reset dev/wbm0 fpgareset

• hardware reset might be required if softboot is not sufficient

ERROR: Cannot connect to socket. Possible reasons: all sockets busy, saftd not running, or wrong socket permissions Aborted

Creating commands: ll, la, cdf, cdi, cdt, diag, inst can't open connection to device dev/wbm0 (system failure)
eb-info: dev/wbm0: system failure
can't open connection to device dev/wbm0 (system failure)
expr: syntax error
expr: syntax error
-
FEC status: SAFTD:ERROR | TIM_SYNC:DIAG_UNKNOWN | TIM_LOCK:DIAG_UNKNOWN FEC uptime: 15 min,  1 users,  load average: 0.06, 0.02, 0.03 DU running: No daemon with DU running.
-
FPGA specs: Not available.
SAFT build: fallout-v6.2.0 (git-hash: d95c321) NFS link  : fel9017 (/opt/nfsinit/fel9017) OS release: GSI embedded release 7 (build 2021-10-26)
-
Aborted
ScuBus Tag: Error reading tag configuration. Init links could be corrupted.

[root@fel9017 ~]# saft-ctl tr0 -i
terminate called after throwing an instance of 'std::runtime_error'
  what():  Cannot connect to socket. Possible reasons: all sockets busy, saftd not running, or wrong socket permissions Aborted
[root@fel9017 ~]#

Misc

Fix OpenJDK Java 11 home on Robert Boywitt's Laptop

[rboywitt@sdanbg011 local-control]$ pwd
 /home/rboywitt/local-control
 [rboywitt@sdanbg011 local-control]$ cat run_local_control_j11.sh 
 #!/bin/bash
 
 export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-11.0.14.1.1-1.el7_9.x86_64"
 
 export PATH=$JAVA_HOME/bin:$PATH
 
 echo $JAVA_HOME
 
 echo "Launch local control GUI"
 java -jar local-control-j11_2.2.jar &
 
 [rboywitt@sdanbg011 local-control]$

Quick Testing

Anschlüsse

SSI-Gerät

Potentiometer

MicroIOC under CentOS7

Setup and Configuration for FESA3 4.0.0

• symbolic link in /common/tftp/csco/pxe/pxelinux.cfg set to microioc.el7, e.g. tstci02 -> microioc.el7
• symbolic links in /common/export/nfsinit/tstci02:

-rwxrwxr-x 1 matthies bel 1119 Jan  4 10:49 20-mount
lrwxrwxrwx 1 matthies bel   34 Jan  4 10:52 30-timing-rte -> ../global/timing-rte-RC8-balloon_0
lrwxrwxrwx 1 matthies bel   15 Jan  4 10:50 40-slits -> ../global/slits
lrwxrwxrwx 1 matthies bel   20 Jan  4 10:51 50-fesa -> ../global/fesa_64bit
-rwxrwxr-x 1 matthies bel  260 Jan  4 10:49 60-slits-lcd

• update of driver, lcd, lcMonitor and server using script ./buildAndDistributeDriverEtc.sh -d tstci02 (run in motion control workspace)
• release of updated FESA software to /common/export/fesa/local
• location of properties.ini: /common/fesadata/data//slits-conf/properties.ini

Stepper Motor System Revamped (2020)

Software Setup

Paths

Configuration

• PXE-Verzeichnis, Links für IP, OS Image

NFS links

• MicroIOCs with the CFA533 LC display also contain a different ethernet network card. The network interfaces are configured appropriately in slits_cfa533.sh .
• The script to launch the lc display software should use /usr/bin/lcd_cfa533

Hardware Changes

• Optional: integration of PCM white rabbit timing receivers (FTRN)
• PMAC reset cable has to be removed within MBOX

  • Anleitungen zum Einbau des WR Timing Receivers: microIOC_user_manual_19_FTRN_Einbau_6.21.pdf, S. 67-69

  • Entfernen des Reset-Kabels zum PMAC-Controller: microIOC-MBOX-FAIR_repair_Manual.pdf, S. 7 ff

  • Wichtig: MicroIOC -interne Flachbandkabel an PCI-104-Board fuer RS 485-Kommunikation mit PMAC-Controller via ADC/SSI nicht vertauschen!

Firmware Updates

• PMAC update using pmac-utils
• SSI update, requires spezialized programmer
• CPLD update (2020: V10), requires JTAG programmer

WR Timing

Prerequisites

• FESA configuration: "timingReceiverPresent" global configuration field has to be set to "true".
• FESA configuration: event mapping per device
• FESA configuration: device configuration: accelerator, timing/accelerator zone
• check availability:

[root@fel9005 ~]# saft-ctl tr0 -i
saftlib source version : saftlib 2.1.0 (v2.1.0): Aug 13 2019 11:29:41
saftlib build info : built by ahahn on Aug 13 2019 15:32:33 with asl743.acc.gsi.de running CentOS Linux release 7.6.1810 (Core)

[root@fel9005 ~]# saft-ctl tr0 -s
no WR lock!!!
receiver free conditions: 253, max (capacity of HW): 0(256), early threshold: 4294967296 ns, latency: 4096 ns
sinks instantiated on this host: 2
 /de/gsi/saftlib/tr0/software/_1 (minOffset: -1000000000 ns, maxOffset: 1000000000 ns)
 -- actions: 0, delayed: 0, conflict: 0, late: 0, early: 0, overflow: 0 (max signalRate: 10Hz)
 -- conditions: 3
 ---- EvtID: 1212048910621605888, mask: 0xfffffff000000000, offset: 0x000000000, active: 1, destructible: 1, owner: :1
 ---- EvtID: 1212067808477708288, mask: 0xfffffff000000000, offset: 0x000000000, active: 1, destructible: 1, owner: :1
 ---- EvtID: 1212048841902129152, mask: 0xfffffff000000000, offset: 0x000000000, active: 1, destructible: 1, owner: :1
 /de/gsi/saftlib/tr0/software/_3 (minOffset: -1000000000 ns, maxOffset: 1000000000 ns)
 -- actions: 0, delayed: 0, conflict: 0, late: 0, early: 0, overflow: 0 (max signalRate: 10Hz)
 -- conditions: 0

Inject Timing Events manually

• Set SettingTriggered -Property via fex or pdex:

user@asl74x slits-conf $ pdex YR09DSBHA SettingTriggered position=0.033 -sp2
NOMEN  = YR09DSBHA (MotorSlit_DU.yr09ci01 | Motor)
TIMDOM = CRYRING_RING (210)
|
SettingTriggered
Please report - UNKNOWN FESA ERROR: cmwrda.Rda.read: USER_ERROR: Error caused by: Field 'enabled' is missing data.. src/MotorSlit/Server/SettingTriggeredSetAction.cpp:90
device="YR09DSBHA" property="SettingTriggered" selector="FAIR.SELECTOR.P=2" requestType="SET"

• First tests: possible with the script /inject_events.sh, the script will inject a fake event with the number 104. The FESA software needs to be configured appropriately for this event.
• First tests: The selected cycle is: sequence index: 0, process index: 1, chain index: 0, timing group: 211
• Alternative script: ./opt/fesa/nfs/global/scripts/inject-event-id.sh 211 532 2 2 2 2

Troubleshooting

• enable enhanced log output: saftbus-ctl --enable-logging
• parse log output for emitted signals: saftbus-ctl --enable-logging
• disable enhanced log output: saftbus-ctl --disable-logging
• decode timing events
• auslesen, ob die Condition-Property AcceptLate gesetzt ist: saftbus-ctl --get-property de.gsi.saftlib.Condition /de/gsi/saftlib/tr0/software/_1/_1957747793 AcceptLate b

Details

Suggested boot order

Remote Reset

PMAC Controller (PMAC Utils)

Backup

upload -ip 192.6.94.5 -a /rmac.txt

Update

sendfile -ip 192.6.94.5 -vr genericSlits.pmac
cd /opt/slits_cfa533/pmac-util

sendfile -ip 192.6.94.5 -vr /usr/bin/genericSlits.pmac

# install tools to update pmac software
for i in pmac/pmacConsole pmac/sendfile pmac/upload pmac/genericSlits.pmac pmac/pmacProperties.ini; do
 install -m 755 /opt/$NAME/$ARCH/bin/$i /usr/bin/
done
for i in pmac/genericSlits.pmac pmac/pmacProperties.ini; do
 install -m 644 /opt/$NAME/$ARCH/bin/$i /usr/bin/
done

opkg install ldd

MicroIOC: Standalone System

Boot from USB Stick

• requires prepared image on an USB stick, transfer with e.g. mkusb
• BIOS settings adjustable when bootable USB stick is connected
• desired boot order:
  • 1) boot from USB stick
  • 2) boot from network (PXE)

How to clone a USB boot stick

Create image from USB stick

Write image to USB stick

Information

Linux

[root@mbox ~]# uname -a
Linux mbox 3.10.101/rt111-scu03 #1 SMP PREEMPT RT Fri Jun 7 15:36:56 CEST 2019 x86_64 GNU/Linux
[root@mbox ~]# cat /proc/version
Linux version 3.10.101-rt111-scu03 (handel@asl741) (gcc version 4.8.5 20150623) (GCC) ) #1 SMP PREEMPT RT Fri Jun 7 15:36:56 CEST 2019
[root@mbox ~]# cat /etc/os-release
GSI embedded release 7 (build 2019-08-12)

Installations

Stepper Motor Software Processes and Services

How to prepare an OS Image

Update for yocto (asl75x)

• see script ~/workspaceMotionControl3/switch_asl75x.sh (https://git.acc.gsi.de/matthies/MotorSlit-scripts/src/branch/master/cluster_switch)
• adapt PXE OS image link
• update NFS init Links
• adapt NFS init scripts (nfsvers=4; no /etc/functions; different mount points), uses files from ~/workspaceMotionControl3/asl75x and ~/workspaceMotionControl3/asl75x_init
• adapt makefiles, comment CXX ?= g++
• re-compile driver software (libslits, lcd, lcmonitor, slits-server) on asl75x (see script ~/workspaceMotionControl3/build.sh)

export LD_LIBRARY_PATH="/common/usr/embedded/yocto/fesa/current/sdk/environment-setup-core2-64-ffos-linux"
source /common/usr/embedded/yocto/fesa/current/sdk/environment-setup-core2-64-ffos-linux # bringt nichts

• TODO re-build pmac-utils, install on image
• TODO re-compile FESA software (MotorSlit /MotorSlit_DU) with FESA3 7.5.+ on asl75x
• TODO how to install other packages like ldd? opkg is not available anymore, not even apt-get
• update export software directories (see script ~/workspaceMotionControl3/copy.sh):

/common/export/slits_cfa533/x86_64/lib
/common/export/slits_cfa533/x86_64/bin
/common/fesadata/data/<microioc>

Back-Switch to asl74x-environment

• see script ~/workspaceMotionControl/switch_asl74x.sh (https://git.acc.gsi.de/matthies/MotorSlit-scripts/src/branch/master/cluster_switch)

MicroIOC under RH6 - deprecated

Mounted Directories

[root@tstci05 /]#  ps 
 854 root      10:40 ./lcMonitor
  864 root       0:00 ./server
  870 root       1:19 ./lcd
  889 root       3:59 /opt/devacc/local/tstci02/library/devman -p
  890 root       0:00 /sbin/getty 115200 /dev/ttyS0
  891 root       0:00 /sbin/getty 19200 /dev/tty1
  901 root       0:02 /usr/bin/dropbear
[root@tstci05 /]#  kill   889

[root@tstci05 /]# cd  /opt/slits/pmac/
[root@tstci05 pmac]# ./upload -a oldconfig-2014-07-08.pmac
Uploading coordinate systems settings...
Uploading motion programs...
Uploading kinematic programs...
Uploading PLC programs...
Uploading P variables...
Uploading Q variables...
Uploading M variables...
Uploading I variables...
All done!
[root@tstci05 pmac]# ./sendfile default-conf-3-3-2006.CFG
Establishing communication with PMAC...OK
Sending file to PMAC...
Done!!
[root@tstci05 pmac]# ./sendfile genericSlits.pmac
Establishing communication with PMAC...OK
Sending file to PMAC...
Done!!

[konh@asl722 lnx]$ ssh root@ke1ci01
root@ke1ci01's password: 
sh: /usr/X11R6/bin/xauth: not found
[root@ke1ci01 /]# cd /opt/pmac/
[root@ke1ci01 pmac]# ./pmacConsole 192.6.94.5
Connected to PMAC at 192.6.94.5.
$$$
STR: '$$$'
 $00 
^C
[root@ke1ci01 pmac]# 

[konh@asl722 lnx]$ cd projekt/steppermotors/localcontrol/2013-12-17/
[konh@asl722 2013-12-17]$ java -jar local-control.jar &