Full PRO reactor release

This page explains how to release the full PRO stack (Java artifacts) using the cscoap-reactor with Andreas' release script.

For a step-by-step guide for how to perform a full release and rollout including database updates, please see AppHowToRelease - this page here only describes part of the full procedure!

Time schedule

When?	What?
3-4 weeks before the Release	Write Release announcement Email (see template), including information on when the Release branches should be there and how to create them.
3-5 days before the Release	Write a reminder Email, that all branches must be there, which day which time
1-2 days before Release	Create Wiki page with Release Order graph, see below
0	Release
0 + x	Rollout
0 + xx	After Rollout is performed, LSA and BSS are up and running and Modellers have imported the Physics Model, write the Release Email (see template)

Before the Release

Checkout all projects

First check that the cscoap-rector is up to date on the pro branch, e.g. no missing artifacts for the release and all obsolete artifacts are removed. Clone all projects using the release branch:

branch=pro; /common/usr/cscoap/bin/mkws/checkout.reactor --ssh --flat -p /common/scratch/cscoap/$USER/$branch/ -b $branch

You can perform the script many times, it skips already cloned repositories and prints those, that are still missing.

Create missing Release Branches yourself (optional)

The release engineers of missing projects should be forced to create a functional and prepared release branch, otherwise it is also possible to exclude these artifacts form the release process (comment out in cscoap-reactor). In this case, developers have to release manually by their own.

If you need to create the Release branch for a project, e.g. it is needed by other artifacts, you can do so by using the normal branching command to create a release branch starting from master and call checkout.reactor for the release afterwards.

Create Wiki Page with Release Graph

On our Releasepage, create a new Subpage for the upcoming Release.

Once all projects are checked out, the graph can be created using:

branch=pro
cd /common/scratch/cscoap/$USER/$branch/cscoap-reactor
./allGraphs

The graphs will reside in the target folder of the reactor. Copy the contents of the /target/ ReleaseOrderGraph.dot to the Release Wiki page. It is not yet a working graph, but some tags must be placed before it:

<span class='WYSIWYG_COLOR' style='background-color: #ffff00; color: yellow;'>______</span> = release branch is already created
<span class='WYSIWYG_COLOR' style='background-color: lightgreen; color: lightgreen;'>______</span> = project is released
<span class='WYSIWYG_COLOR' style='background-color: lightgrey; color: lightgrey;'>______</span> = project is obsolete
<graphviz type="svg" style="width:100%;height:auto" inline="true">
CONTENT OF ReleaseOrderGraph.dot HERE
</graphviz>

If the Wikipage just shows errors, please check for Umlaute / special characters in Release Engineer names. Also check that you copied the above section correctly and placed it before the graph contents.

Add expand button and reactor cluster (groups) to the release graph

See AppReleaseGraphReactorClusterTemplate

Check all projects

Every user should perform the release preparations listed here: Release preparations.

However we currently check each project, if there are dependencies with fixed versions that need to be added to the central managed bom. This can be done by using the command

grep --exclude=**/target/** --exclude-dir=.git "<version>" **/pom.xml |grep -v "18.0.0-SNAPSHOT" |grep -o ".*/pom.xml" |uniq

Performing the Release

With the reactor, many or even all projects can be released at once. However it can also be beneficial to release first lower level common-library Projects, than LSA, then applications. This can be achieved by passing the --project-list argument specifing the projects to release to the reactor release script.

Doing so, you have to ensure manually, that all needed dependencies are already released or within the listed projects, otherwise your release will fail due to missing dependencies or even might include older dependencies.

For 1..n projects, perform the release

Check the pom.xml for each project

It is necessary to check each pom.xml file before the release. Following pitfalls exist and must be corrected before the release:

• please make sure, that developers and roles are still correct, remove any Umlaute here !!!! e.g. Müller -> Mueller as otherwise the Release Order Graph will not be working
• cleanup dependencies:
  • remove direct dependencies that anyway come through others, might be a leftover from a quick bugfix
  • remove dependencies, that are commented out, especially those with -SNAPSHOT in the name as they cause problems in the release process
  • check that there are no fixed older pro-versions defined, e.g. pointing to 15.0.0 (as only -SNAPSHOTS will be increased to the latest pro release version number). Those should point to the latest master SNAPSHOT version instead.

In Email die Release preparation steps verlinken, dieses hier eher löschen.(?)

Release

cd /common/scratch/cscoap/$USER/$branch/
git clone https://git.acc.gsi.de/schaller/scripts
cd cscoap-reactor
../scripts/mvn/mvn_reactor_release -rv 18.0.0 -dv 18.0.1-SNAPSHOT

Problems during the Release and Recovery

Release cannot be performed at all e.g. due to missing dependencies or Javadoc errors

• Scenario 1: You can fix the error
  • No recovery needed. Just start the release all over again after fixing the problem (providing the --resume-from flag to the reactor release script)!
• Scenario 2: You cannot fix the error, and the project is not needed by any other project (e.g. the failing project is an application):
  • Skip the failing project and continue with the next one, providing the --resume-after flag to the reactor release script! Come back to the project later, or delegate to project expert.

Release successful, Tag screwed up

While the release seems successful, the file does not exist in Nexus (typically fails in upstream projects) and the corresponding pom.xml file of the created tag is screwed up, i.e. it contains SNAPSHOT versions. (Cause of this error is currently not known). In this case we need to manually delete the tag, reset in Git history to the point before the release and run the complete release process again.

$ tagName=cscoap-common-units-ui-18.0.0
$ git tag -d $tagName
$ git push origin --delete $tagName
$ git reset --hard HEAD~2
$ git push -f

Release successful, Tag looks correct, no file in Nexus

While the release seems successful, the file does not exist in Nexus (typically fails in upstream projects). (Cause of this error is currently not known). In this case, we can checkout the tag and just perform mvn deploy to upload it to Nexus:

$ tagName=cscoap-common-units-ui-18.0.0
$ git reset --hard $tagName
$ mvn deploy site-deploy -Prelease

Release successful, Tag correct, file in Nexus, but site deploy failed

If only the site deploy failed, you can easily repeat it by using

$ tagName=cscoap-common-units-ui-18.0.0
$ git reset --hard $tagName
$ mvn site-deploy -Prelease

However if the site deploy failed due to a lack of permissions, than check if you have done the step to rename the old site directory for the old version? If not, please do so now !!! and repeat any site deploys for downstream projects.

Release-Voraussetzungen: altes Site Verzeichnis verschieben

After the Release

Update master to next release snapshot version

• Update LSA Test Dev DB to new version. (If you miss this step, tests will fail for e.g. lsa-core-gsi when you update the snapshot version!)
  • checkout lsa-db-scripts, branch master
    
    • master_log_dev_test.xml
      • <- set to NEXT major version, e.g. for release Rx => (x+1).0
    • Commit, Push
  • Trigger execution: Jenkins: https://builder.acc.gsi.de/jenkins/job/lsa/job/lsa-test-dev/ -> trigger build (or wait until Jenkins runs it automatically)
• Change all master versions to next snapshot, e.g. if release R18 was just performed, they should be changed to 19.0.0-SNAPSHOT.
  • If you do it manually, perform a snapshot deploy of all artifacts before pushing the new versions, otherwise jenkins might fail for all non top level artifacts.If you use the release script, trust that it is ok that it pushes first.

• Update LSA Dev DB to new version.
  • checkout lsa-db-scripts, branch master
    
    • master_log_dev.xml
      • <- set to NEXT major version, e.g. for release Rx => (x+1).0
    • Commit, Push
  • Trigger execution: Ask LSA team!

Tell everyone about the release!

Afterwards, write email and announce the new versions are available on master and release is done.

Should there be a template for this e-mail?

General TODOs

• wenn von INT gebrancht, sollte das Releasescript auch zusätzlich RC-INT ersetzen können
• Release-Script muss intermediate Releases x.y mit y > 0 unterstützen (nicht nur x.0)
• Hochziehen von Versionen
  • Bei Plugins werden die Versionen irgendwie nicht richtig ersetzt (Snapshots bleiben stehen)
  • Versionsangaben in Plugin Management werden nicht korrekt hochgezogen (Snapshots bleiben stehen)
  • Versionsangaben in Profil-Dependencies werden nicht korrekt hochgezogen (Snapshots bleiben stehen)
    • Beispiel: expert-storimo, surveillance-app-fx
  • ${project.version} bei Abhängigkeiten im gleichen Reactor, sollte auch durch Release-Version ersetzt werden (bleibt stehen)
    • Beispiel: ionsource-reactor
  • Property parentversion in csco-parent-java wird nicht mit hochgezogen (alte Version bleibt stehen)
• Beim Release: Stiller Downgrade: Wenn für 18.0.0-SNAPSHOT Dependencies kein 18.0.0 gefunden wurde (s.u., zB sequencer-ui), wurde einfach mit 17.0.x released!!! Keine Fehler oä!
• Beim "updateMaster" Schritt: mvn versions:update-parent hat “neuere” Version als die von uns explizit angegebene verwendet! (angegeben: 19.0.0-SNAPSHOT, verwendet: 19.0.0-AW-SNAPSHOT (war nur testweise installiert...))

-- JuttaFitzek - 19 Nov 2020