DevaccMavenBuildHowto < Frontend

You are here: Foswiki>Frontend Web>DevaccMavenBuildHowto (20 Jan 2023, PeterKainberger)Edit Attach

Build, Release, and Rollout of Device Access Software
using Maven, SCons, and Scripts

Caution: Device Access Software development must still (Apr. 2019) be done on the acc6 cluster! (See Hinweise zum Umzug auf das acc7-Cluster.)

32bit libraries must be provided for Unilac application programs, but there's no 32bit CORBA and no Motif on acc7.
Nevertheless, FE is ready to generate 64bit and PPC software as well as M68k SE software on the acc7 cluster.
- Preconditions for 64bit and PPC software:
  - All i386 and i386-static modules are commented out in the pom.xml files.
  - Correct oracle version is used in .../framework/nameservice/server/x86_64/pom.xml and .../framework/nameservice/server/fedbupdate/x86_64/pom.xml.
  - Correct library versions are included in .../fecs/*/assembly.xml. See .../fecs/templates/libs-acc7.txt and .../fecs/templates/show-libs.sh
Generation of 64bit and PPC software using Maven 3.6.0 has been successfully testet, too. (Dec. 2018)
- Tests included mvn deploy of snapshots but not mvn release:prepare/perform.

Note that ECM and EQMs are still built and released using the well-known tools. See chapter Build, Release, and Rollout ECM and EQMs below.

Caution: Work on your Git workspace! Do not work on your SVN workspace any more, it is deprecated! All Device Access sources have been migrated from SVN to Git already. See Hinweise zum Umzug von Device Access von SVN nach Git.

Preconditions
What Build, Release, Rollout means
List of most used Commands
What to do when
- Software of a Device Access framework module has changed
- Software of an eqpiupment model has changed
- Software of an equipment driver has changed
Build and Release Device Access Framework
Build and Release Equipment Drivers
Build and Release Equipment Software
Generate XML and Python Class Files of an Equipment Model
Rollout XML and Python Class Files of an Equipment Model
Create or Add Conditions
Rollout of Conditions Handling Tools
Handle Equipment Model XSD Files
Package and Release Software for a Host
Rollout Software for a Host System
Package and Release Software for a FEC
Prepare FEC before first Rollout
Rollout Software for a FEC
Which Software Versions are Running?
Build and Test New or Upgraded Equipment Software
Build, Release, and Rollout ECM and EQMs
ECM, EQMs, and EC Directories
Releases and Rollouts for HKR-Cluster
Nightly Builds
Python Modules Built with Makefiles
Local DBS Modules Built with Makefiles
LaTeX Styles for Documentations
How a FEC Boots
Administrate Caplog
Administrate Nameserver
Administrate UfcServer
References and Links

Preconditions

Authentication is required to deploy artifacts in Nexus and to access the Subversion and Git repositories. See Authentication to learn how to manually configure your Maven password settings. See also Git Authentication for the Gitea repository. And don't forget to add your public SSH key from ~/.ssh.id_dsa.pub or ~/.ssh.id_rsa.pub in the Git user's SSH / GPG Keys tab.

A valid host key for asl335 is necessary on the development machine, e.g. on asl735, to deploy the USR XML files to asl335:/common/usr/cscofe/htdocs/eqmod-xmls. Use ssh asl735.acc.gsi.de once on your development machine to generate the host key.

All version numbers of the framework, the equipment models, the fecs, and the hosts are double figures in the form ss.dd.pp. If necessary, leading zeros must be used, like 09.08.07. Drivers are allowed to not comply with this rule.
You need a local ~/.m2/settings-global.xml to redirect some paths and you must add this to every mvn command like this:
```
mvn ... -gs ~/.m2/settings-global.xml
```

What Build, Release, Rollout means

Build: Compile and link sources, generate sources like mx_22.xml or MX_22EqMod.py.
Release: Collect all necessary files, pack them into a ZIP and upload it to the Nexus repository. Releases may be snapshot or real releases.
Rollout: Download a ZIP from the Nexus repository, unzip it in case of a host rollout or place the ZIP into a FEC directory in case of a FEC rollout, and create the necessary symbolic links.

These definitions are valid for all Device Access software except for ECM and EQMs.

Release for ECM and EQMs: may be seen as the copy of the generated software to their common directories (see "ECM Directories" and "EQM Directories" in chapter ECM, EQMs, and EC Directories).

Rollout for ECM and EQMs: may be seen as the generation and copy of an sr-file to its CPU directory (see "EC Directories" in chapter ECM, EQMs, and EC Directories) and the download to its EC. Note that an ECM is usually not rolled out stand-alone, that is without (further) EQMs.

Both, release and rollout of EC software, is done using relecm or releqms respectively.

List of most used Commands

bei Problemen mit halbfertigen releases:

mvn release:clean

Ein Gerätemodell bauen:

mvn versions:use-latest-releases -gs ~/.m2/settings-global.xml
mvn versions:commit -gs ~/.m2/settings-global.xml
git commit -a -m "use latest releases"
git push
mvn release:prepare --batch-mode -DscmCommentPrefix="use latest releases" -gs ~/.m2/settings-global.xml
mvn release:perform -gs ~/.m2/settings-global.xml

Software für einen FEC (k.......) bauen und releasen:

mvn versions:use-latest-releases -gs ~/.m2/settings-global.xml
mvn versions:commit -gs ~/.m2/settings-global.xml
git status
git commit -a -m "use latest releases"
git push
mvn release:prepare --batch-mode -DscmCommentPrefix="use latest releases" -gs ~/.m2/settings-global.xml
mvn release:perform -gs ~/.m2/settings-global.xml
mvn package -gs ~/.m2/settings-global.xml
copygup -z target k...... 10.??.??
ssh root@k...... /sbin/reboot

What to do when

Software of a Device Access framework module has changed

Build and Release Device Access Framework
Package and Release Software for a Host
Rollout Software for a Host System

If the changed Device Access framework module affects FEC software, e.g. devman or libvmedevice, then

Do above steps 1 to 3
Package and Release Software for a FEC
Rollout Software for a FEC

Software of an eqpiupment model has changed

Build and Release Equipment Software
Package and Release Software for a FEC
Rollout Software for a FEC

If the equipment model API, the <eqmod>.xml file, has changed, then

Do above steps 1 to 3
Rollout XML and Python Class Files of an Equipment Model
Inform Vitaliy (CSCOSV) about the upgraded equipment model version, e.g <eqmodversion>MX_23</eqmodversion> in mx.xml, so that he can import the XML file for JAPC.

See Generate XML and Python Class Files of an Equipment Model for infos concerning therapy properties.

Software of an equipment driver has changed

Build and Release Equipment Drivers
Build and Release Equipment Software that needs the driver
Package and Release Software for a FEC
Rollout Software for a FEC

Build and Release Device Access Framework

Don't want to read all the specific stuff? Then skip to chapter Build and Release Framework Example

Clone all Framework Modules from Gitea

Clone all framework modules from accdata to vmedevice!

Thus, devacc/framework should be cloned from the Gitea repository into the user's workspace and named e.g. framework. Note that the top level pom.xml resides in devacc/framework of the Gitea repository respectively in the framework directory of the user's workspace.

If the Gitea commit/push has been done with the Maven release:prepare and release:perform goals, all pom.xml files cloned from the repository contain the correct SNAPSHOT version. So, after clone you can immedialtly start doing e.g. an mvn install without changing the version of a single pom.xml file.

Nevertheless, if the version in a pom.xml is not appropriate, use the set or the update-child-modules goal of the Maven Versions Plugin (see below for details).

Update Artifact Versions in POM Files

In certain cases, it is necessary to adjust the version numbers in the pom.xml files after cloning/pulling from Gitea. For example if a major software update has been implemented, the version number should be updated from 10.26.02-SNAPSHOT to 10.27.00-SNAPSHOT but not to 10.26.03-SNAPSHOT.

To update the version in the parent POM at the framework directory, the parent sections of all child POMs in accdata ... vmedevice, and the section with framework dependencies in all architecture POMs use:

mvn versions:set -DallowSnapshots -DnewVersion=10.27.00-SNAPSHOT -gs ~/.m2/settings-global.xml

Be sure to update to a SNAPSHOT version, since release:prepare and release:perform will later change those snapshot versions to a real release version (10.27.00-SNAPSHOT to 10.27.00).

Note that there are modules in the framework tree which are not part of the build process and thus their POMs will not be updated with mvn versions:set. The tools milmon and vmeterm for example will be build only for x86_64. The POMs in their i386 directories remain unaffected.

To update all other (non-framework) dependencies to the latest versions, use the

mvn versions:use-latest-releases -gs ~/.m2/settings-global.xml

goal. It's always a good idea to check the versions in the pom.xml files after such a command.

Build and Install Framework

Go to the framework parent directory, e.g. framework (see chapter Clone all framework modules above), and invoke mvn install. This builds all framework modules and installs them in your local repository ~/.m2/repository/de/gsi/cs/co/fe/devacc/framework/. Installing the framework modules allows you to subsequently re-build selected submodules based on the newest framework environment in your local repository.

To re-install all modules, use mvn clean install.

A mvn compile does not install any targets in your local repository. Presently (1.Sep.2014), the build may also not succeed since there is still an unresolved inter-dependency between two framework modules. Avoid this problem by using mvn install.

A mvn deploy at the framework parent directory installs all framework modules in the central Nexus csco-snapshot repository. A mvn deploy should be done for snapshot versions only! For non-snapshot releases see the chapters below.

Release Framework

A real (non-snapshot) release is done with

mvn release:prepare [--batch-mode] [-DscmCommentPrefix="Comment for Git tag"]
mvn release:perform

This includes

the modification of versions within the POM files,
the version handling of sources in cooperation with the Gitea repository (push, tag),
as well as the version handling of artifacts in cooperation with the Nexus repository (upload of a zip-file containing headers, libraries, executables, etc.).

Note that a mvn release:[prepare|perform] can be done only at a directory where the pom.xml contains the tags <developerConnection> and <tagBase>! This means, that the whole framework, each single equipment model, and each single FEC must be released on its own. Thus, a mvn release:[prepare|perform] at the parent directories workspace/eqmodels, workspace/eqdrivers, and workspace/fecs is not possible!

Note also that only the framework parent defines a version number in framework/pom.xml. All modules from accdata to vmedevice do not define own versions in their framework/<module>/pom.xml. Instead, they inherit the version from the parent POM.

Before doing a mvn release:[prepare|perform], all necessary sources must have been committed and pushed into the Gitea repository or must be declared as ignorable.

For Git just create the file .gitignore on the project home directory and add the sources that should be ignored.

If an error has occured during mvn release:[prepare|perform], use mvn release:rollback to rollback changes already made by this release.

For more info see Maven Release Plugin.

How a framework release is done using mvn release is best described with an example, see Build and Release Framework Example below.

Special Device Access Java Handling

The Device Access Java API devacc_java depends only on the Corba IDL. Its parent is csco-parent while the parent of all other framework modules is framework-parent. Additionally, its POM may have to be updated when the Java environment changes.

So the Device Access Java API must be generated independently from all other framework modules. Use the same approach to build and release devacc_java as with the other framework modules.

Also ask Vitaliy Rapp for help on this topic. Probably it's the best idea to remove devacc_java from framework and to place the source code elsewhere at CSCOSV. See some Emails on this topic.

Build and Release Framework Example

First very briefly:

Be sure you are on an acc6 development machine!

cd ~/workspace
git clone git@git.acc.gsi.de:devacc/framework.git framework
cd framework
mvn versions:use-latest-releases   # check updated pom.xml for eqmodel SNAPSHOT versions
mvn versions:commit
emacs ...
git commit -a -m "Bug fixed"
git push
mvn release:prepare --batch-mode -DscmCommentPrefix="Bug fixed"
mvn release:perform

If it is necessary to increase the minor or major version number (see Version number rules) use

cd ~/workspace
git clone git@git.acc.gsi.de:devacc/framework.git framework
cd framework
mvn versions:set -DallowSnapshots -DnewVersion=10.27.00-SNAPSHOT
mvn versions:use-latest-releases
mvn versions:commit
emacs ...
emacs ...
git commit -a -m "New method xyz in AccDevice"
git push
mvn release:prepare --batch-mode -DscmCommentPrefix="New method xyz in AccDevice"
mvn release:perform

For devacc_java, if necessary (see Special Device Access Java Handling above), use

cd ~/workspace/framework/devacc_java
mvn versions:use-latest-releases
mvn versions:commit
cd ~/workspace/framework
git commit -a -m "Use latest releases"
git push
# TODO: noch ausarbeiten für Git & Nexus!
# cd ~/workspace/framework/devacc_java
# pom.xml mit Version ohne SNAPSHOT erzeugen
# git commit/push/tag
# mvn deploy
# pom.xml mit neuer SNAPSHOT-Version erzeugen
# git commit/push
# Ein mvn release/perform geht hier nicht!

respectively

cd ~/workspace/framework/devacc_java
mvn versions:use-latest-releases
mvn versions:commit
emacs pom.xml
cd ~/workspace/framework
git commit -a -m "Use latest releases"
git push
# TODO: noch ausarbeiten für Git & Nexus!
# cd ~/workspace/framework/devacc_java
# pom.xml mit Version ohne SNAPSHOT erzeugen
# git commit/push/tag
# mvn deploy
# pom.xml mit neuer SNAPSHOT-Version erzeugen
# git commit/push
# Ein mvn release/perform geht hier nicht!

Now some explanations for the first brief framework example above where only the 'incremental version' will be increased.

After cloning devacc/framework from Gitea the framework/pom.xml in the user's workspace contains
```
<artifactId>framework-parent</artifactId>
<version>10.26.01-SNAPSHOT</version> 
```
Go to ~/workspace/framework and update all pom.xml files using
```
mvn versions:use-latest-releases 
```
so that a Maven build uses the latest (non-framework) dependencies.
After some source code changes and commit/push of the changed files, the framework/pom.xml in user's workspace still contains the same version that it had after cloning. (If the source code changes require a major version change, see chapter Update Artifact Versions in POM Files on how to do that before committing/pushing the changed files.)

After mvn release:prepare --batch-mode and mvn release:perform, there are the following versions:

There is a new Gitea repository tag
```
10.26.01
```

The pom.xml in the Gitea repository 10.26.01 tag contains

<artifactId>framework-parent</artifactId>
<version>10.26.01</version>

There are new artifact versions in the Nexus csco repository.

/de/gsi/cs/co/fe/devacc/framework/framework-accdata-i386/10.26.01/framework-accdata-i386-10.26.01.zip
/de/gsi/cs/co/fe/devacc/framework/framework-accdata-ppc/10.26.01/framework-accdata-ppc-10.26.01.zip
/de/gsi/cs/co/fe/devacc/framework/framework-accdata-x86_64/10.26.01/framework-accdata-x86_64-10.26.01.zip
...
/de/gsi/cs/co/fe/devacc/framework/framework-vmedevice-ppc/10.26.01/framework-vmedevice-ppc-10.26.01.zip

The pom.xml in the Gitea repository master branch contains

<artifactId>framework-parent</artifactId>
<version>10.26.02-SNAPSHOT</version>

The framework/pom.xml in user's workspace contains

<artifactId>framework-parent</artifactId>
<version>10.26.02-SNAPSHOT</version>

See also chapter 2.3.1 Inconsistency after Maven release in "Hinweise zum Umzug von Device Access von SVN nach Git" to check whether inconsistencies still exist. If yes, they have to be handled!

Go to ~/workspace/framework/devacc_java and update the pom.xml file using
```
mvn versions:use-latest-releases 
```
so that the Maven build uses the latest dependencies. Among others, the latest dependency of the just released framework-corbaifc-idl module will be used. But see also Special Device Access Java Handling above!
: Ausarbeiten für Git! (Last, commit and push the changes, prepare and perform the release for the devacc_java module.) Oder Vitaliy Rapp konsultieren.

Build and Release Equipment Drivers

The method to build and release a driver is similar to the method for equipment software. So see chapter Build and Release Equipment Software for detailed info.

Usually, a Driver is not rolled out on its own. Instead it is 'linked' via a dependency to an equipment model and then packaged and rolled out for a FEC. See chapter Child POM of Equipment Software as well as Package and Release Software for a FEC and Rollout Software for a FEC for more info.

Note that drivers are allowed to deviate from the ss.dd.pp version format rule.

Build and Release Equipment Software

Only the build and release of USRs is described here. EQMs are still build and released with the well known scripts and makefiles (vmeconfig, make, geneqms, releqms, ...).

Don't want to read all the specific stuff? Then skip to chapter Build and Release Equipment Software Example

Clone Equipment Model from Gitea

Clone the devacc/eqmodel_<eqmod> from the Gitea repository and name it eqmodels/<eqmod> in your workspace.

If the Gitea commit/push has been done with the Maven release:prepare and release:perform goals, all pom.xml cloned from the Gitea repository contain the correct (SNAPSHOT) version for the <eqmod> artifact. So, if the last tagged pom.xml of a module in Gitea contains <version>10.00.02</version>, then the corresponding pom.xml cloned from the Gitea master branch contains <version>10.00.03-SNAPSHOT</version>.

Nevertheless, if the version in a pom.xml is not appropriate, e.g. since a major software update is to be implemented, use the set or the update-child-modules goal of the Maven Versions Plugin. See chapter Update Artifact Versions in POM Files which is also valid for building and releasing equipment drivers and framework software.

POMs should contain explizit versions for their dependencies! To update your POMs so that the latest versions of their dependencies are used during generation, use the mvn versions:use-latest-releases goal. Note that in nearly all cases, equipment software should not depend on snapshot versions of framework artifacts and/or equipment drivers! See chapter Update Dependency Versions in POM Files which is also valid for building and releasing equipment drivers and framework software.

If you start developing a completely new equipment model, use the template XMLs in ...devacc/eqmodels/templates as a starting point.

Parent POM

Say, you are upgrading the DGX equipment model and your working directory is ~/workspace/eqmodels/dgx. The artifact description in the parent pom.xml is then e.g.

<groupId>de.gsi.cs.co.fe.devacc.eqmodels</groupId>
<artifactId>eqmodel-dgx-parent</artifactId>
<version>10.00.03-SNAPSHOT</version>
<packaging>pom</packaging>

where the version is a snapshot version.

Since DGX runs only on a PPC-GuP, there's only one architecture module.

<modules>
  <module>ppc</module>
</modules>

See for instance the equipment model CIA which has three modules.

Child POM

The child pom.xml in ~/workspace/eqmodels/dgx/ppc defines no own version. It must use the parent version. This assures that all targets of a multi-module build, e.g. KGB for x86_64, i386, and ppc, based on the same sources have the same version.

<parent>
  <groupId>de.gsi.cs.co.fe.devacc.eqmodels</groupId>
  <artifactId>eqmodel-dgx-parent</artifactId>
  <version>10.00.03-SNAPSHOT</version>
</parent>

<artifactId>eqmodel-dgx-ppc</artifactId>
<packaging>scons</packaging>

<dependencies>
  <dependency>
    <groupId>de.gsi.cs.co.fe.devacc.framework</groupId>
    <artifactId>framework-vmedevice-ppc</artifactId>
    <version>10.26.02</version>
    <type>zip</type>
  </dependency>
  <dependency>
  ...

Usually, equipment model software, even those in development, are built against a released framework, i.e. an non-snapshot version. So dependencies on framework modules do not specify SNAPSHOT versions.

Update Dependency Versions in POM Files

Version ranges of dependencies are not recommended for releases since old builds of software will not be reproducible in the future. They also cause problems when releasing software using mvn release:prepare and mvn release:perform. Maven recognizes newer SNAPSHOT releases in this case, but must not!

So after cloning of a project and before generating it, e.g. with mvn compile, but at least before releasing it with mvn release:prepare and mvn release:perform, the corresponding POMs should be updated so that they use the latest released versions of the artifacts they depend on.

Say the last released version of framework-accdevice-ppc is 10.26.03.

If your pom.xml has two dependencies like

<dependency>
  ...
  <artifactId>framework-accdevice-ppc</artifactId>
  <version>[10.26,10.27)</version>
  ...
  <artifactId>framework-vmedevice-ppc</artifactId>
  <version>10.25.42</version>
  ...
</dependency>

performing mvn versions:use-latest-releases changes the pom.xml to

<dependency>
  ...
  <artifactId>framework-accdevice-ppc</artifactId>
  <version>10.26.03</version>
  ...
  <artifactId>framework-vmedevice-ppc</artifactId>
  <version>10.26.03</version>
  ...
</dependency>

See also the mvn versions:resolve-ranges goal.

SConscript

SCons together with SConscript does all the real work like USR code generation, compilation, linking, Pyton stubs generation for PropHelper, and equipment model XML expansion (for JAPC).

The SConscript file resides in the src directory of the equipment model. In most cases, it is standard and needs no equipment model specific modification.

Only for Equipment Models with sub-models a slight modification is necessary to extract the correct equipment model name out of the artifact name. The standard code in SConscript, e.g. for DGX, is

# Get eq-model name from artifactId: eqmodel-dgx-ppc -> dgx
#-----------------------------------------------------------
name = env["MAVEN"].artifactid.split('-')[1]

However, e.g. for ISAU, which is a sub-model of STD, the code in SConscript is

# Get eq-model name from artifactId: eqmodel-std-isau-ppc -> isau
#-----------------------------------------------------------------
name = env["MAVEN"].artifactid.split('-')[2]

When the template eqmodels/templates/SConscript[-SubEqmods] is used as a starting point for a new equipment model, see the TODOs in the source file and handle them appropriately. And don't forget to remove the TODOs when you're ready!

Doing a Build

Usually, a build is done with the latest versions of the modules (artifacts) the equipment model depends on, e.g the latest version of accdevice (artifact framework-accdevice-ppc). To update the <versions> tags of all of the project's pom.xml files to the latest dependency versions, go to the equipment model's project home directory (e.g. workspace/eqmodels/dgx but not workspace/eqmodels/dgx/ppc) and use

mvn versions:use-latest-releases

Then see, whether software could be build (compiled and linked) without errors using mvn [clean] compile or mvn [clean] install. These commands may be invoked either in ~/workspace/eqmodels/dgx for all architectures or in ~/workspace/eqmodels/dgx/ppc only for ppc. Note that some equipment models, e.g. cia and kgb, are built for more than one architecture.

For more infos see chapter Build and Test New or Upgraded Equipment Software further down.

Doing a Release

A preceding build (see above) is not necessary. Perhaps a mvn clean may be a good idea.

Be sure to use the latest versions of dependencies. Use mvn versions:use-latest-releases to update your pom.xml files. It's always a good idea to check the versions in the pom.xml files after such a command. See chapter Doing a Build above.

Requirements for a release:

All sources are committed and pushed into the Gitea repository.
The working directory is ~/workspace/eqmodels/dgx, not ~/workspace/eqmodels/dgx/ppc.

A release in the working directory is done with

mvn release:prepare [--batch-mode] [-DscmCommentPrefix="Comment for Git tag"]
mvn release:perform

The mvn release:prepare asks for a new version number interactively unless the option --batch-mode is given. In this case the rightmost version number is incremented automatically.

If an error has occured during the release process, use mvn release:rollback to rollback changes already made by this release.

For more info see Maven Release Plugin. See also mvn release:prepare -DdryRun.

Interdependent Equipment Models

A few equipment models depend on other equipment models. To use the latest software of both models, be sure to build the base equipment model first!

First MX, then MXS
First DPX, then DPB

Equipment Models with sub-models

A few equipment models are childs of parent models. FG01 is a child of FG, ISAU is a child of STD, and PZUS is a child of PZ. Those equipment models must always be released as a whole! So, if there was a change in ISAU, the whole STD path must be released. Replace dgx in the Build and Release Equipment Software Example right below with std, fg, or pz and you know what to do for those equipment models.

Build and Release Equipment Software Example

Note the two chapters Interdependent Equipment Models and Equipment Models with sub-models right above!

First very briefly:

Be sure you are on an acc6 development machine!

cd ~/workspace/eqmodels
git clone git@git.acc.gsi.de:devacc/eqmodel_dgx.git dgx
cd dgx
mvn versions:use-latest-releases
mvn versions:commit
emacs src/dgx-read-info.cc
git commit -a -m "Bug fixed"
git push
mvn release:prepare --batch-mode -DscmCommentPrefix="Bug fixed"
mvn release:perform

If it is necessary to increase the minor or major version number (see Version number rules) of dgx use

cd ~/workspace/eqmodels
git clone git@git.acc.gsi.de:devacc/eqmodel_dgx.git dgx
cd dgx
mvn versions:use-latest-releases
mvn versions:commit
emacs src/dgx-dev-def.h
emacs ...
mvn versions:set -DallowSnapshots -DnewVersion=10.01.00-SNAPSHOT
git commit -a -m "New slave data struct in DPR"
git push
mvn release:prepare --batch-mode -DscmCommentPrefix="New slave data struct in DPR"
mvn release:perform

Now some explanations for the first brief example above where only the 'incremental version' will be increased.

After cloning of eqmodel_dgx from the Git repository, the ~/workspace/eqmodels/dgx/pom.xml file may contain

<artifactId>eqmodel-dgx-parent</artifactId>
<version>10.00.04-SNAPSHOT</version>

And ~/workspace/eqmodels/dgx/ppc/pom.xml may contain

...
<dependencies>
  <dependency>
    ...
    <artifactId>framework-accdevice-ppc</artifactId>
    <version>10.26.00</version>
    ...
    <artifactId>framework-vmedevice-ppc</artifactId>
    <version>[10.26,10,27)</version>
    ...

Go to ~/workspace/eqmodels/dgx and update the pom.xml files using
```
mvn versions:use-latest-releases 
```
so that a Maven build uses the latest dependencies. Usually, most of the dependencies are defined in the architecture pom.xml files. So, if the latest release of the framework is 10.26.03, then ~/workspace/eqmodels/dgx/ppc/pom.xml contains
```
...
<dependencies>
  <dependency>
    ...
    <artifactId>framework-accdevice-ppc</artifactId>
    <version>10.26.03</version>
    ...
    <artifactId>framework-vmedevice-ppc</artifactId>
    <version>10.26.03</version>
    ...
```
afterwards. It's always a good idea to check the versions in the pom.xml files after this command.
After some source code changes and commit/push of the changed files (including ~/workspace/eqmodels/dgx/ppc/pom.xml) into the Gitea repository, the ~/workspace/eqmodels/dgx/pom.xml still contains the same version that it had after cloning. (If the source code changes require a major version change, see chapter Update Artifact Versions in POM Files on how to do that before committing/pushing the changed files.)
After mvn release:prepare --batch-mode and mvn release:perform, there are the following versions:
1. There is a new Gitea repository tag
```
10.00.04
```
  of the project devacc/eqmodel_dgx.
2. The pom.xml in the Gitea repository tag 10.00.04 contains
```
<artifactId>eqmodel-dgx-parent</artifactId>
<version>10.00.04</version> 
```
  Note that the version suffix -SNAPSHOT has gone!
3. There is a new artifact version in the Nexus csco repository.
```
/de/gsi/cs/co/fe/devacc/eqmodels/eqmodel-dgx-ppc/10.00.04/eqmodel-dgx-ppc-10.00.04.zip
```
4. The pom.xml in the Gitea repository master branch contains
```
<artifactId>eqmodel-dgx-parent</artifactId>
<version>10.00.05-SNAPSHOT</version> 
```
5. The ~/workspace/eqmodels/dgx/pom.xml in user's workspace contains
```
<artifactId>eqmodel-dgx-parent</artifactId>
<version>10.00.05-SNAPSHOT</version> 
```
6. See also chapter 2.3.1 Inconsistency after Maven release in "Hinweise zum Umzug von Device Access von SVN nach Git" to check whether inconsistencies still exist. If yes, they have to be handled!

Generate XML and Python Class Files of an Equipment Model

For further application program usage, XML and Python class files are generated and packed into the equipment model zipfile during the Maven build of the equipment model.

A generated XML file <eqmod>_<version>.xml (e.g. fg01_18.xml) contains all sources originally included via xi:include. In other words, all xi:includes are resolved.

For the prophelper tool, therapy property Python classes are appended to the generated file <EQMOD>_<version>EqMod.py (e.g. FG01_18EqMod.py) according to the tags <additional>/<defaultproperty> in <eqmod>.xml The Python sources that contain the therapy property Python classes are part of the fe-usrgen artifact and reside in its ./var/python subdirectory. The sources are

therapy-usrs.py
therapy-master-usrs.py
therapy-slave-usrs.py
therapy-usrs-helpers.py

Appending therapy Python sources instead of generating them from the XML file is a quick and dirty solution already used on the acc5 cluster. If a property description or a helper class changes in a therapy property XML file, the related therapy property Python file must be changed accordingly!
See ...utilities/code-generation/fe-usrgen/src/scons/tools/usrgen.py and ...utilities/code-generation/fe-usrgen/src/var/python/therapy-*.py.

Rollout XML and Python Class Files of an Equipment Model

The version of the property interface of an equipment model, e.g. the 20 in MD_20, changes much less frequently than its USRs version, e.g. md 10.00.03. Thus, the rollout of the XML and Python class files of an equipment model are not done using Maven. Instead, the rollout must be done explicitly using relxml or relpy respectively.

However, the release of these files is done with every Maven release!

relxml

Use relxml on the development cluster (asl735) only. Use relxml -H for a detailed description.

Accesses to the equipment model XML files via webserver must use the address https://websvcdev.acc.gsi.de/groups/cscofe/eqmod-xmls/. Accesses via https://websvcpro... will find no XML files.

Update 21. Aug. 2018: The webservers have been migrated to the acc7 clusters long ago. That means that they now access their htdocs directory on asl34x and asl74x respectively. As long as we are still developing and releasing equipment models on the acc6 cluster, relxml (on asl735) does a rollout of the equipment model XML files to all four clusters.

relpy

Use relpy on the development cluster (asl735) or on the production (HKR) cluster (asl335). Use relpy -H for a detailed help.

If relpy is run on asl735, a rollout to asl335 will be done as well. If relpy is run on asl335, no rollout will be done to asl735.

Update 21. Aug. 2018: As long as we are still developing and releasing equipment models on the acc6 cluster, relpy (on asl735) does a rollout to to all four clusters.

XML and Python Class File of DMAN

The equipment model DMAN may be handled as every other equipment model although it is part of the framework devman artifact. The scripts take care of this fact.

Create or Add Conditions

Conditions are one component of a framework release. All header files *msg.h are rolled out to /common/usr/cscofe/opt/devacc/pro/include ($incmsg), a static library libcondmsgs.a is rolled out /common/usr/cscofe/opt/devacc/pro/lib ($libmsg), and a shared library libcondmsgs.so to /common/usr/cscofe/opt/devacc/pro/lib64 ($libmsg64).

To update conditions for or add conditions to an existing facility, e.g. MX, or to create a completely new facility, it is not necessary to rollout a new framework release. The tool genmsg does the 'rollout' of the appropriate header file (e.g. mxmsg.h), the static and the shared library directly to the directories specified in $incmsg, $libmsg, and $libmsg64.

Keep in mind, that the conditions artifacts in Nexus which are rolled out with a framework release are themselves generated. Thus, they may not reflect the latest contents of the conditions database! Simply use genmsg md mx dgx pzs ... to update the header files of the specified facilities as well as the static and the shared library. The libraries will then contain the latest definitions of all facilities.

Example

Get the XML-file of the facility that should be updated from the conditions database.
```
$ conditions-loader get -facilities MX
```
Edit the XML-file MX.xml.
Update the conditions database.
```
$ conditions-loader update -file MX.xml
```
Rollout header file and libraries for EQMs.
```
$ genmsg mx
```

Add Conditions for USRs

To generate USRs that need new or updated conditions, it is necessary to update the conditions artifacts in the Nexus repository.

[

: Also conditions will move from SVN to Git! Ask Susi Jülicher. ]
Therefore, after you have updated the conditions database (see example above), checkout (or update) /cscosv/conditions/artifacts/trunk from SVN and execute ./mvn-all deploy at the top-most directory. Conditions artifacts will then be deployed into the Nexus csco-snapshot repository at /de/gsi/cs/co/sv/conditions/artifacts/conditions-art-*/1.0.0-SNAPSHOT. That's all.

Note that always 1.0.0-SNAPSHOT versions will be deployed; there are neither higher snapshot versions nor non-snapshot versions in Nexus for conditions!

Afterwards, use mvn -U ... once in your equipment model directory to get the updated artifacts from Nexus.

Further Help

See Condition Compiler and Bedienungsanleitung Condition Compiler for more information. Use genmsg -h for help.

For further questions ask the developer of the Conditions Handling Tools Stefan Krepp.

Rollout of Conditions Handling Tools

To use the condition handling tools conditions-loader, conditions-generator, and conditions-importer, they have to be rolled out ('released'). These tools are independent of Device Access versions and thus are rolled out to /common/usr/cscofe/lib/java/conditions by relcondtool. Use relcondtool -h for help and see Condition Compiler.

Handle Equipment Model XSD Files

The XML Schema Definition (XSD) files for the equipment model XMLs changes very rarely.

Actual files are located in .../workspace/utilities/code-generation/fe-usrgen/src/resources/xsd/eqmod_types.xsd. If one of these files has changed, it must be uploaded to the webserver, where it is expected from the equipment model XML files.

To upload all XSD files, simply use the Script upload-xsd.sh in the directory.

  ./upload-xsd.sh <username>

Note that the xsd-files in .../workspace/utilities/uti/ixml/xsd/ and .../workspace/utilities/uti/xml-v03/ are deprecated and obsolete.

To see what's on the webserver or to upload single files, use cadaver.

asl735$ cd ~/workspace/utilities/uti/xml-v03
asl735$ cadaver https://www.acc.gsi.de/dav/XMLSchema/xsd/v03
dav:/dav/XMLSchema/xsd/v03/> ls
Listing collection `/dav/XMLSchema/xsd/v03/': succeeded.
        eqmod.xsd                           1146  May 19  2017
        eqmod_additional.xsd                1964  May 19  2017
        eqmod_default.xsd                   1314  May 19  2017
        eqmod_default_properties.xsd         979  May 19  2017
        eqmod_header.xsd                    1699  May 19  2017
        eqmod_properties.xsd                1788  May 19  2017
        eqmod_types.xsd                    13663  May 19  2017
dav:/dav/XMLSchema/xsd/v03/> put eqmod_types.xsd
dav:/dav/XMLSchema/xsd/v03/> quit

To upload all files in the directory with cadaver use

dav:/dav/XMLSchema/xsd/v03/> mput *

Package and Release Software for a Host

This means to package all software that is needed on a host system (development and production (HKR) cluster) in a zip-file and to upload the zip-file to the Nexus repository. Needed software is

Services (executables) like nameserver and ufcserver,
Shared libraries from libaccdata.so to libusrs.so,
Python modules for Device Access,
Condition headers and libraries,
All headers and libraries needed for ECM/EQMs development.

Very briefly:

cd ~/workspace
git clone git@git.acc.gsi.de:devacc/hosts.git
cd hosts
# see note 1 below
mvn versions:use-latest-versions -DallowSnapshots=false    
git commit -a -m "Use latest versions"
git push
# see note 2 and 3 below
mvn release:prepare --batch-mode [-DignoreSnapshots=true] -DscmCommentPrefix="Use latest versions"
mvn release:perform

Note 1: Note that mvn versions:use-latest-versions is used here instead of mvn versions:use-latest-releases. Note also its optional parameter -DallowSnapshots=false. This updates all dependencies in the POM to the latest non-snapshot framework release. It's always a good idea to check the versions in the pom.xml files after this command.

Note 2: Use the additional option -DignoreSnapshots=true with mvn release:prepare as long as libraries/artifacts from SV are available as snapshots only. At the moment (Feb. 2018), these libraries/artifacts are

de.gsi.cs.co.sv.conditions.artifacts.conditions-art-*
de.gsi.cs.co.sv.opera.vms-libs.teventdefs

with version 1.0.0-SNAPSHOT.

Note 3: Be sure that two de.gsi.cs.co.sv... artifacts above are specified with a version range in the dependencies section of the hosts/<target> POMs. This is valid as long as these artifacts from SV are available as snapshots only. Example of .../workspace/hosts/x86_64/pom.xml:

    <dependency>
      <groupId>de.gsi.cs.co.sv.conditions.artifacts</groupId>
      <artifactId>conditions-art-lib64-dyn</artifactId>
      <version>[1.0.0-SNAPSHOT, 1.1.0]</version>
      <type>zip</type>
    </dependency>

Note 4: See also chapter 2.3.1 Inconsistency after Maven release in "Hinweise zum Umzug von Device Access von SVN nach Git" to check whether inconsistencies still exist. If yes, they have to be handled!

Rollout Software for a Host System

Rolling out a framework release means to download the appropriate zip-file from the Nexus repository, create the appropriate directories on the cluster (asl735, asl335; root is /common/usr/cscofe/opt/devacc/<release>), place the zip content into these directories, and create symbolic links to these directories. Releases may be production (link called pro) or development (link called dev) releases.

A production release rollout must be done on the production (HKR) cluster (asl335) and on the development cluster (asl735).

A development release rollout may be done on the development cluster (asl735) only.

A rollout is done using the script reldevacc. See reldevacc -H for further detailed help. A complete rollout for a single cluster consists of four parts:

Rollout software for x86_64 (e.g. reldevacc -a x86_64 10.26.02)
Rollout software for i386 (e.g. reldevacc -a i386 10.26.02)
Rollout Python software (e.g. reldevacc -a python 10.26.02)
Set a development (dev) or a production (pro) link (e.g. reldevacc -r pro 10.26.02)

See /common/usr/cscofe/opt/devacc/<yyy-mm-dd>.log and /common/usr/cscofe/opt/devacc/<release>/devacc-versions-<arch>.txt for the versions released. See also chapter Which Software Versions are Running?

Files needed for EC software development (ECM and EQMs) will also be rolled out.

Note: As long as there's no Maven/SCons support to differentiate between lib and lib64, reldevacc handles this shortcoming internally. This handling has to be removed when Maven/SCons supports both lib and lib64.
See a host-x86_64-<version>[-<snapshot>].zip file. As long as it contains lib/lib*.so files but not lib64/lib*.so files, there is no lib64 support from Maven/SCons.

Package and Release Software for a FEC

Don't want to read all the specific stuff? Then skip to chapter Package and Release Software for a FEC Example

Clone a FEC from Gitea

This is similar to the handling of euipment models, see chapter Build and Release Equipment Software.

Clone devacc/fec_<FEC> from the Gitea repository and name it fecs_<FEC>.

If the Gitea commit/push has been done with the Maven release:prepare and release:perform goals, all pom.xml cloned from the Gitea repository contain the correct (SNAPSHOT) version for the <FEC> artifact. So, if the last tagged pom.xml of a module in Gitea contains <version>10.00.02</version>, then the corresponding pom.xml cloned from master branch contains <version>10.00.03-SNAPSHOT</version>.

Nevertheless, if the version in a pom.xml is not appropriate, use the set or the update-child-modules goal of the Maven Versions Plugin. See chapter Update Artifact Versions in POM Files.

POMs should contain explizit versions for their dependencies! To update your POMs so that the latest versions of their dependencies are used during generation, use the mvn versions:use-latest-releases goal. Note that in nearly all cases, production FECs should not depend on snapshot versions of framework, equipment software, and/or equipment drivers artifacts! See chapter Update Dependency Versions in POM Files.

If you want to configure a completely new FEC, use the template XMLs in Git project devacc/fec_templates as a starting point.

POM and Assembly

The pom.xml contains all generated artifacts which are needed for the FEC. These are usually:

Devman and Infosnd,
the standard equipment models EC and SWPZ (PPC-GuPs only),
the specific equipment models for this FEC,
and, if necessary, additional modules (see e.g. DSM)

Drivers are normally included automatically via the dependency of the equipment model.

The assembly.xml adds additional modules (files) that are necessary on the FEC. These may be:

Libraries of omniORB and boost,
additional Linux libs (libicudata etc., non-PPC only)
additional start scripts that accompany 95_devman,
and devacc-versions.txt (see chapter Which Software Versions are Running?).

It also controls which files and directories should be included in and exluded from the zip-file which is to be generated.

assembly.xml on acc6 cluster

In most cases, the assembly.xml must not be changed. Just clone devacc/fec_templates from the Gitea repository and copy fecs/templates/assembly-acc6-ppc.xml or fecs/templates/assembly-acc6-x86_64.xml to your <FEC> directory and rename it to assembly.xml.

Versions of libraries may have changed. Use show-libs.sh on fecs/templates to see them.

assembly.xml on acc7 cluster

For acc7 use show-libs.sh and see libs-acc7.txt on fecs/templates for the right versions of omniORB, boost, and additional Linux libraries.

Release Software for a FEC

Release means to generate and upload a new zip-file to the Nexus repository.

The process is very similar to a equipment model release, so see also chapters Doing a Release and Build and Release Equipment Software Example.

Package and Release Software for a FEC Example

Be sure you are on an acc6 development machine!

cd ~/workspace/fecs
git clone git@git.acc.gsi.de:devacc/fec_k4xcg01.git
cd k4xcg01
mvn versions:use-latest-releases
mvn versions:commit
emacs pom.xml  # e.g. add/remove an equipment model
git commit -a -m "Equipment model XYZ added"
git push
mvn release:prepare --batch-mode -DscmCommentPrefix="Equipment model XYZ added"
mvn release:perform

See also chapter 2.3.1 Inconsistency after Maven release in "Hinweise zum Umzug von Device Access von SVN nach Git" to check whether inconsistencies still exist. If yes, they have to be handled!

In some cases it is necessary to tell Git to ignore some resources, e.g. the directory target. Create .gitignore on the project home directory (e.g. workspace/fecs/k4xcg01) and add the resources that should be ignored. This must be done before the commit/push commands.

Prepare FEC before first Rollout

Before the very first rollout, an nfsinit directory for the FEC must be created on an asl735 or asl335 machine. The directory is shared between asl735 and asl335 hosts.

cd /common/export/nfsinit/
mkdir <FEC>
cd <FEC>
ln -s ../global/devacc 90_devacc

Equipment models that need persistant data, e.g. stepper motors on uIOCs (DSM, DSME), must mount a directory on a server machine via NFS. This must be done by an additional script /common/export/nfsinit/<FEC>/20-mount. See for example kueci01 or kapci01.

Rollout Software for a FEC

Rollout means to download a zip-file from the Nexus repository to the appropriate FEC directory.

Use relgup <FEC> <version>

to download the zip-file from Nexus and to place it in FEC directory /common/export/devacc/cpu/<FEC> and
to define a symbolic link <FEC>.zip to the downloaded zip-file.

The FEC directory is shared between asl735 and asl335 hosts. During boot, the FEC downloads the <FEC>.zip via NFS. See relgup -H for further detailed help.

Use cdcpu <FEC> to change to FEC's CPU directory.

See the log-file <yyyy-mm-dd>.log which is written by relgup into the FEC's CPU directory.

Edit <FEC>.dbd and Update <FEC>.dbs Files

Caution: Work on your Git workspace! Do not work on your SVN workspace any more, it is deprecated!

Use dbsgen and dbscopy.

Update your workspace first before editing a dbd-file.

cd ~/workspace/localdb/dbd
git pull

And don't forget to push your edited dbd-file back to Gitea when you are ready.

git commit -a -m "Device XYZ added"
git push

dbscopy copies <FEC>.dbs to two destinations:

to the /common/export/devacc/cpu/<FEC> directory
to the remote FEC's root directory root@<FEC>:/

The first destination is used to copy the dbs file when the FEC boots. During this phase, the dbs file is copied to the FEC's RAM-disk via NFS.

The second destination is directly the RAM-disk of the FEC. The dbs file is copied with scp. So a new local database may be loaded by Devman

just by using its RESET property of the FEC,
by executing /etc/init/boot.d/95_devman restart directly on the FEC or
by performing ssh root@<FEC> '/etc/init/boot.d/95_devman restart' on a remote host, e.g. asl735,

and without the necessity to reboot the whole FEC.

Which Software Versions are Running?

A devacc-versions.txt file is generated and packaged to its artifact zip-file during the package phase for a host or a FEC artifact. It contains all packaged artifacts and their versions.

For a FEC do e.g.:

ssh root@k2xcg01 cat devacc-versions.txt

For host software do e.g.:

cd /common/usr/cscofe/opt/devacc/pro
ls -l devacc-versions-*.txt
less devacc-versions-x86_64.txt

Build and Test New or Upgraded Equipment Software

Don't want to read all the specific stuff? Then skip to chapter Build and Test Equipment Software Example

Build and Install USRs

Be sure to use the latest versions of dependencies. Use mvn versions:use-latest-releases to update your pom.xml files. See chapter Doing a Build above.

Doing a mvn [clean] install results in a file eqmodel-<eqmod>-<arch>-<ss.dd.pp>-SNAPSHOT.zip at the target directory, e.g. ~/workspace/eqmodels/dgx/ppc/target/eqmodel-dgx-ppc-10.00.03-SNAPSHOT.zip, as well as at the local ~/.m2/repository/... of the user. The latter will be used during packaging (mvn package) of the necessary software (USRs, libraries, devman) for a FEC (GuP, uIOC).

If only one developer is testing this software, it is sufficient to package the software for this FEC using the USRs to be tested from the local .m2 repository and the rest of the software (other USRs, libs, devman) from the Nexus repository csco.

If several developers are in need of this (new) software, the zip-file must be placed as a snapshot version in the Nexus repository csco-snapshot to make it available for all developers.

To place the zip-file in the Nexus csco-snapshot repository, use mvn [clean] deploy instead of mvn [clean] install.

Package the Software for a FEC

Go to the (workspace) directory of the FEC that you want to use for testing your USRs (e.g. ~/workspace/kp1cg25/ or ~/workspace/fecs/kp1cg25/) and edit the dependencies in the pom.xml accordingly. At least, the USRs to be tested must be your generated snapshot version.

<dependency>
  <groupId>de.gsi.cs.co.fe.devacc.eqmodels</groupId>
  <artifactId>eqmodel-dgx-ppc</artifactId>
  <version>10.00.03-SNAPSHOT</version>
  <type>zip</type>
</dependency>

Instead of an explizit snapshot version (10.00.03-SNAPSHOT) you may as well use a version range ([10.00-SNAPSHOT,10.01)) in pom.xml.

Package the software for the GuP using mvn [clean] package. This will generate a zip-file in the target directory, e.g. ~/workspace/[fecs/]kp1cg25/target/fec-kp1cg25-10.26.00-SNAPSHOT.zip.

Copy and Run the Software

Copy this zip-file to the GuP's CPU directory using copygup -z target kp1cg25 10.26.00. Use copygup -H for further detailed help.

See the logfile <yyyy-mm-dd>.log at the CPU directory or use unzip -l target/<zipfile> or less target/devacc-versions.txt to see which files and versions have been zipped.

Reboot the FEC. It will unzip the zip-file and run the new software. See also chapter How a FEC Boots.

Build and Test Equipment Software Example

On the equipment model directory: Implement the new code and install the equipment model zip (into your local .m2 repository).

cd ~/workspace/eqmodels/dgx
mvn versions:use-latest-releases
emacs src/myfile.cc
mvn install

Then, on the FEC directory: Edit the equipment model version in the pom.xml to use the previously installed equipment model zip and package (or install) the FEC software.

cd ~/workspace/fecs/kp1cg25
emacs pom.xml

  <dependency>
    <groupId>de.gsi.cs.co.fe.devacc.eqmodels</groupId>
    <artifactId>eqmodel-dgx-ppc</artifactId>
    <version>10.00.09-SNAPSHOT</version>    <!-- here's the new DGX test version -->
    <type>zip</type>
  </dependency> 

mvn package

Finally see what has been packaged, copy the zip to the CPU directory, and reboot the FEC.

less target/devacc-versions.txt
  The following files have been resolved:
    ...
    de.gsi.cs.co.fe.devacc.eqmodels:eqmodel-dgx-ppc:zip:10.00.09-SNAPSHOT:compile
    ...
dir target
  -rw-rw-r-- 1 kain bel 7056812 May  3 12:24 fec-kp1cg25-10.26.01-SNAPSHOT.zip
copygup -z target kp1cg25 10.26.01
ssh root@kp1cg25 /sbin/reboot

Build, Release, and Rollout ECM and EQMs

Software for equipment controllers is not build with Maven and SCons. ECM and EQMs are still built and released using the well-known tools.

For local tests:

vmeconfig
make

For production generation, release, and rollout:

genecm
relecm
geneqms
releqms
ecload

For production tools, use <tool> -H for further detailed help.

Tag EQM-Software in Git Repository

It is strongly recommended to tag EQM-software in the Gitea repository using mvn release:prepare and mvn release:perform even if there have been no changes in USR-software. This approach guarantees consistent version numbers in Gitea, Nexus, USRs (POMs) and EQMs (generation using geneqms). See Clone Equipment Model from Gitea and Build and Release Equipment Software Example.

Linking your EQMs with a locally generated ECM

For testing a new ECM version it may be desirable to do that together with (existing) EQMs. Thus, the EQMs must be linked with your locally generated ECM. How to do that is described here.

Assume, that the ECM project has been cloned from Gitea into your .../workspace/ecm directory and that you have done a make clean all at that directory to generate all ECM files needed to generate your EQMs. Assume also that the ECM version is 10.xx.

To generate your EQMs using the locally build ECM 10.xx, you need two links at your .../workspace/ecm directory

10xx -> .
current -> 10xx

and the environment variable ecm68 to point to the current link.

$ echo $ecm68
.../workspace/ecm/current

Now you can use "make clean all" at your EQMs source directory, e.g. .../workspace/eqmodels/dgx/src, to link your EQMs with the locally generated ECM. The resulting file may then be g-dgx-sissdi-eqms100007-ecm10xx.sr.

ECM, EQMs, and EC Directories

The directory structures on the acc6 cluster are different compared to the acc5 cluster. But environment variables like $incvme, $ecm68 etc. are still valid.

ECM Directories

/common/usr/cscofe/opt/devacc/sys/ecm/10/1002
                                        /current -> 1002
                                        /g-ecm-v10.sr
                                 /m68k/common
                                      /cxx
                                      /error
                                      /psos

Also the command cdsys ecm [sysversion] still works.

EQM Directories

/common/usr/cscofe/opt/devacc/eqp/bc/2014-06-18.log
                                    /g-bc-eqms100000.o
                                 /...
                                 /dti
                                 /...
                                 /xml/bc_08.xml
                                     /2014-07-29.log

The command cdeqp [eqmodel] still works.

EC Directories

/common/export/devacc/cpu/k2xcg01/ec1
                                 /ec2
                         /...         
                         /ka2cg04/ec0
                                 /...
                                 /ece

The command cdcpu [GuP|EC] still works.

Message Header Files

After each rollout of a new Device Access version, symbolic links xxx-msg.h to the new message header files xxxmsg.h must (still) be created on $incmsg to allow EQMs generation. See $utiasl/makeMsgSymlinks.sh in Tools.

Releases and Rollouts for HKR-Cluster

For software which was generated via Maven see chapter Rollout Software for a Host System.

Utilities (mostly) located in $uti, $utiasl, $utivme, and $utipy are rolled out using the Makefiles on their project directories. A make copy or make release rolls out the utility to both the asl735 and the asl335 cluster. Note that additions and adaptions on /common/usr/cscofe/bin must be done on both clusters!

Nightly Builds

Nightly builds are done using Jenkins.

Based on SVN, each of the Device Access modules eqdrivers, eqmodels, fecs, framework, hosts, utilities were build nightly. The projcet names in Jenkins were devacc-eqdrivers, devacc-eqmodels, devacc-fecs, devacc-framework, devacc-hosts, and devacc-utilities.

Nightly builds based on SVN are now disabled in Jenkins.

It is not clear yet whether a nightly build based on Git is feasible (and still necessary). In Git, there are no modules like eqmodels any more that comprise all equipment models. So, as far as I know, each Git project, like eqmodel_dgx or fec_k1xcg01, has to be defined in Jenkins on its own, which means a really great effort.

Python Modules Built with Makefiles

Some Python modules are independent of Device Access releases. These modules are still built and rolled out using Makefiles.

Module	Makefile
devscr	.../utilities/devscr/devscr/src/Makefile
eqmods	.../utilities/devscr/eqmods/src/Makefile
PropHelper	.../utilities/PropHelper/PropHelper/src/Makefile
eqmodwrappers	.../utilities/PropHelper/eqmodwrappers/src/Makefile

All Makefiles copy the sources to the appropriate directories on the asl735 as well as on the asl335 cluster. See the Makefiles for more info.

Local DBS Modules Built with Makefiles

The local DBS modules dbsgen, mxmd_coeff, and mxmd_extract are independent of Device Access releases. These modules are still built and rolled out using Makefiles.

The source files of these modules are located in project localdb/dbsgen.

The Makefile copies the modules to the $utivme directory on the asl735 cluster. See the Makefiles for more info.

Note: dbscopy[.py] is located in the project utivme.

LaTeX Styles for Documentations

A few LaTeX styles developed at ACO are or may be necessary for equipment model (gm-<eqmodel>.tex) or other documentation.

The sources, including the Makefile to copy (rollout) them to their destination, are located at .../utilities/uti/latex.

The sources are rolled out to the destination directory /common/usr/cscofe/texmf/tex/latex/gsi/ by using the Makefile.

How a FEC Boots

First, see Embedded NFS Initialisierung. The /common/export directory of an asl735 machine can be seen (NFS mounted) from a PPC-GuP.

A FEC, e.g. kp1cg25, finds during boot his directory /opt/nfsinit/kp1cg25 (which is /common/export/nfsinit/kp1cg25 on an asl735 machine) and executes the scripts in the directory in directory order. The last script to be executed is 90_devacc which is a link to ../global/devacc which in turn is a link to generic on the same directory.

The generic script is called with some predefined environment variables, e.g. $NAME contains devacc. So generic calls /opt/devacc/devacc.sh (which is /common/export/devacc/devacc.sh on an asl735 machine).

The devacc.sh script then unpacks the zip-file, copies the dbs file and finally calls all scripts named /etc/init/boot.d/9[1-9]_devman* (see below) in alphabetical order. Note that 95_devman has just been unpacked from the zip-file.

Last but not least, the 95_devman script starts the devman.

Note that you can manually restart Devman

by executing /etc/init/boot.d/95_devman restart directly on the FEC or
by performing ssh root@<FEC> '/etc/init/boot.d/95_devman restart' on a remote machine, e.g. asl735.

More possible parameters are stop, start, and status.

Note also that both start and restart do not load the zip-file from the server!

Note on 95_devman and init.sh

The original source filename of 95_devman is init.sh. It is located in ../framework/devman/src/, installed as 95_devman by ../framework/devman/src/SConscript and then packed from Maven into the ../framework/devman/<arch>/target/framework-devman-<arch>-<version>.zip file.

Note on mount for persistant data

Note on wildcard filename 9[1-9]_devman*

Slits on uIOCs need more software than just devman. To install/start this additional software in correct order, additional scripts are also necessary. So for example two more scripts, one that should be executed before devman and one after, may use the following filenames and would then be executed in the following order:

94_devman_slits
95_devman
96_devman_slits-lcd

Leading numbers must be in the range 91 to 99, 90 is not allowed! See also /common/export/devacc/devacc.sh.

Note on uIOC Serial Interfaces M-Box

Those uIOC M-Boxes need a special Linux configuration for their serial interfaces. See chapter "Troubleshooting Serial Interfaces M-Box" in MicroIOC.

Administrate Caplog

Caplog wird als Linux-Service wo/wie eingehängt? -> Christoph fragen.

Das Startscript ist asl335:/common/usr/cscofe/opt/devacc/pro/bin/caplog-cluster.sh.

Das Exe ist asl335:/common/usr/cscofe/opt/devacc/pro/bin/caplog.

Die Config-Datei ist asl335:/common/usr/cscofe/opt/devacc/pro/etc/caplog/caplog-log4j.xml.
Achtung, Alarme werden nach wie vor (Sep. 2018) via syslog geloggt, da log4cxx ein Problem mit dem 'Re-Connect' zum Server hat(te).

Alle drei Dateien sind also versions-abhängig, da sie unter .../devacc/pro/... liegen.

Geloggte Alarme angucken in Graylog: Im Eingabefeld program:caplog eingeben oder unter "Saved searches" oben rechts "Alarms (Caplog)" auswählen.

Administrate Nameserver

Der Nameserver wird als Linux-Service wo/wie eingehängt? -> Christoph fragen.

Das Startscript ist asl335:/common/usr/cscofe/opt/devacc/pro/bin/nameserver-cluster.sh.

Das Exe ist asl335:/common/usr/cscofe/opt/devacc/pro/bin/nameserver.

Die Rechtedatei ist asl335:/common/usr/cscofe/opt/devacc/etc/nameserver/right.xml. Die darin referierte Datei http://www-acc.gsi.de/XMLSchema/right.xsd wird beim mvn deploy des Nameservers dorthin kopiert (siehe .../nameservice/server/src/xml/right.xsd, .../nameservice/server/x86_64/pom.xml und .../nameservice/server/src/SConscript).

Die Persistenz-Datei ist asl335:/common/usr/cscofe/opt/devacc/var/nameserver/accessdata.txt.

Startscript und Executable sind versions-abhängig, da sie unter .../devacc/pro/... liegen. Rechte- und Persistenz-Datei sind versions-unabhängig, da sie unter .../devacc/etc/... bzw. unter .../devacc/var/... liegen.

nameservercmd and nsrvrem

The helpers nameservercmd and nsrvrem may be used on boyh clusters (asl735, asl335).

Administrate UfcServer

Der UfcServer muss in xinetd eingehängt werden. Siehe workspace/ufcserver/config/* (Achtung, die Dateien sind veraltet! Sie galten auf acc5. Wie es jetzt ist, Christoph fragen.).

Das Exe ist /common/usr/cscofe/opt/devacc/pro/bin/ufcserver.

Auf VMS den logischen Namen

"ACC_UFCSERVER_HOST" = "asl320.acc.gsi.de"
        = "asl321.acc.gsi.de"
        = "asl322.acc.gsi.de"
        = "asl323.acc.gsi.de"
        = "asl324.acc.gsi.de"
        = "asl325.acc.gsi.de"
        = "asl326.acc.gsi.de"
        = "asl327.acc.gsi.de"

entsprechend umhängen!

Wenn VMS abgeschaltet ist, wird auch der UfcServer nicht mehr benötigt!

Modul aus ...framework/pom.xml rausnehmen (oder auskommentieren).
Christoph Bescheid sagen, dass er ihn aus dem xinetd rausnimmt.

References and Links

[1]	Condition Compiler
[2]	Condition Compiler Bedienungsanleitung
[3]	Git
[4]	The Git Book
[5]	Gitea and Gitea @ GSI
[6]	Jenkins and Jenkins @ GSI
[7]	Maven
[8]	Maven Quick Reference Card
[9]	Maven Release Plugin
[10]	Maven Versions Plugin
[11]	Maven at CSCO	including authentication and useful commands
[12]	Nexus and Nexus @ GSI
[13]	SCons
[14]	Subversion and Subversion @ GSI
[15]	Subversion Guideline at CSCO	including Subversion structure

Topic revision: r187 - 20 Jan 2023, PeterKainberger

Frontend

Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback

Build, Release, and Rollout of Device Access Softwareusing Maven, SCons, and Scripts

Table of Contents

Preconditions

What Build, Release, Rollout means

List of most used Commands

What to do when

Software of a Device Access framework module has changed

Software of an eqpiupment model has changed

Software of an equipment driver has changed

Build and Release Device Access Framework

Clone all Framework Modules from Gitea

Update Artifact Versions in POM Files

Build and Install Framework

Release Framework

Special Device Access Java Handling

Build and Release Framework Example

Build and Release Equipment Drivers

Build and Release Equipment Software

Clone Equipment Model from Gitea

Parent POM

Child POM

Update Dependency Versions in POM Files

SConscript

Doing a Build

Doing a Release

Interdependent Equipment Models

Equipment Models with sub-models

Build and Release Equipment Software Example

Generate XML and Python Class Files of an Equipment Model

Rollout XML and Python Class Files of an Equipment Model

relxml

relpy

XML and Python Class File of DMAN

Create or Add Conditions

Example

Add Conditions for USRs

Further Help

Rollout of Conditions Handling Tools

Handle Equipment Model XSD Files

Package and Release Software for a Host

Rollout Software for a Host System

Package and Release Software for a FEC

Clone a FEC from Gitea

POM and Assembly

assembly.xml on acc6 cluster

assembly.xml on acc7 cluster

Release Software for a FEC

Package and Release Software for a FEC Example

Prepare FEC before first Rollout

Rollout Software for a FEC

Edit <FEC>.dbd and Update <FEC>.dbs Files

Which Software Versions are Running?

Build and Test New or Upgraded Equipment Software

Build and Install USRs

Package the Software for a FEC

Copy and Run the Software

Build and Test Equipment Software Example

Build, Release, and Rollout ECM and EQMs

Tag EQM-Software in Git Repository

Linking your EQMs with a locally generated ECM

ECM, EQMs, and EC Directories

ECM Directories

EQM Directories

EC Directories

Message Header Files

Releases and Rollouts for HKR-Cluster

Nightly Builds

Python Modules Built with Makefiles

Local DBS Modules Built with Makefiles

LaTeX Styles for Documentations

How a FEC Boots

Note on 95_devman and init.sh

Note on mount for persistant data

Note on wildcard filename 9[1-9]_devman*

Note on uIOC Serial Interfaces M-Box

Administrate Caplog

Administrate Nameserver

nameservercmd and nsrvrem

Administrate UfcServer

References and Links

Build, Release, and Rollout of Device Access Software
using Maven, SCons, and Scripts