Initial Setup Process for a Supermicro Server

Basic System Information

• The Mainboard is called Supermicro X11SSW-F (Verify that in the Bios Setting before uploading any new firmware !!)
• Information/manual/software/firmware regarding the board can be found on the supermicro webpage of that board.

Mounting the Hardware

• Usually Someone from IN mounts the Pexaria card and connects the ethernet cables / the Pexaria LEMO .. here some details on that:

• The following two figures explain how-to install a PCIe card into a SuperMicro after the top cover has been removed.

Figure: Remove three screws (red) and pull the handle (blue).

Figure: Unplug the frame and mount PCIe modules.

Finally, reinsert the frame, close the handle, insert the three screws and reinstall the top cover.

Here a "Labeled" Pexaria card:

Please note that the Pexaria card must be registered by Timing, BEFORE it gets connected to the Timing Network !!

Management Port (IPMI)

• Name of Managenment Port always is either <hostname>i64 or <hostname>i
• If management port is not reachable after first boot, unplug + replug of RJ45 should help.
• User is "ADMIN", default PW is "ADMIN" oder "RMS.ChangeMe" ... please change that to something meaningfull in: Configuration --> Users --> ADMIN --> modify User
• For existing systems, ask the according system maintainer for the password
• Reboot/Shuitdown/Console can be used on this website: <hostname>i64.acc.gsi.de
• Only the left ethernet port seems to work
• This one is the management Port:

Figure: SuperMicro Ethernet ports.

• Uhhh, its not quite as simple; when only one Ethernet port is connected the following will happen
  • LAN Interface 'Failover' : all ports work as 'Management Port'; you can only reach the IPMI via <hostname>i64 (or <hostname>i)
  • LAN Interface 'Dedicate' : ports behave as indicated in the figure 'Management Port' above
  • thus, check (and configure) the desired behaviour via IPMI -> Configuration -> Network

Figure: SuperMicro Ethernet port config.

• You can modify or even factory reset the management software via IPMICFG-Linux.x86_64. (Installed at /common/export/fesa/local/dal004/IPMICFG_1.32.0_build.200910)

Booting via PXE (Ramdisk) and Flashing Gateware

• Attention! There is a "proadcast bug" with older Pexaria gateware (v4.0.0 and below), which potentially may lead to broadcasted garbage events on the timing network. Because of that, please emediatly flash a new gateware on it when the system is Powered On !
• Define image to boot on pxe-boot
  • source /common/usr/cscofe/scripts/accdefs # to be able to use ACO/FEC tools
  • pxe-config -l # to check if someone already configured that host
  • pxe-config -c # to configure the host (default bootimage will be 'yocto' .. select whatever you want to boot)
• reboot the system to load the correct ramdisk (e.g. via the management port)
• Flash the correct gateware to the Pexaria: https://www-acc.gsi.de/wiki/Main/CheckAndFlashTimingReceiverFirmware
• Optionally open bios (pres 'del' during boot) and make sure that "resume to latest powerstate after power loss" is set.
  • BIOS --> Advanced --> Boot Feature --> Restore on AC Power Loss --> Last State

BIOS/IPMI(BMC) Firmware Upgrade

• Tested for Board Supermicro X11SSW-F, with IPMI 3.64 --> 4.04 and BIOS 2.0c --> 2.8a
• Note that if IPMI/BIOS firmware upgrade fails, that can brick your device, so only do so if there is strong need, and be very careful
• Make sure to store all IPMI/BIOS settings, since a transition of the old settings will not be possible. Manual reconfiguration will be required
• Make sure to download the correct IPMI (BMC) and BIOS firmware for your board from the supermicro webpage of that board.
  • The concrete board ID can be checked on the 'main' page in the bios settings.
  • You current IPMI/BIOS version can be checked on the page 'system' in the IPMI webinterface
• It is IMPORTANT to first upgrade the IPMI firmware and only afterwards upgrade the BIOS firmware!
• IPMI Firmware Upgrade
  • Unzip both archives you downloaded. The IPMI firmware ships some pdf's with it. READ THEM CAREFULLY!!
  • Upload the IPMI binary via the web interface. ... it's the file with the *.bin ending.
  • During IPMI upgrade, as noted in the pdf, uncheck all checkboxes which are about reserving configuration. NOT DOING SO MIGHT BRICK YOUR IPMI.
  • Flash the new IPMI. Be patient and don't interrupt the process ! Afterwards a reboot will be triggered.
  • Default login is ADMIN/ADMIN Directly change that after you logged in
  • Configure the new IPMI settings, according to the previous configuration
• BIOS Firmware upgrade
  • In order to upgrade the BIOS firmware via the IPMI web interface, you need to have a Supermicro IPMI License-Key for that server
    • Don't generate a key on your own with this command:

      echo -n 'bmc-mac' | xxd -r -p | openssl dgst -sha1 -mac HMAC -macopt hexkey:8544E3B47ECA58F9583043F8 | awk '{print $2}' | cut -c 1-24

    • Instead, register on the supermiscro webpage and buy a license for ~30$ per system!
  • Make sure to exit BIOS before you do the upgrade
  • Upload the new BIOS firmware via the new IPMI webinterface (The biggest file after unzip ... should be ~16MB)
  • Flash the new BIOS. Be patient and don't interrupt the process ! Afterwards a reboot will be triggered.
  • If you see a black screen during boot via iKVM/HTML5 monitoring, then choose PowerControl --> Set Power Off and afterwars, PowerControl --> Set Power On.
  • Press 'del' on the virtual keyboard to enter the new Bios
  • Configure the new BIOS settings, according to the previous configuration
• If you reach this step: Congratulations, you are done!

Known Bugs

• When using a PCIE-USB Extension Card, the system only reboots when the boot is watched via IPMI
  • This bug can be prevented by disabling legacy USB support in the BIOS:

    Advanced --> USB Configuration --> Legacy USB Support

  • For more info, here is the related issue: https://gitlab.com/al.schwinn/DigitizerClass2/-/issues/136