Announcements

2020-07-13

  • maintenance week fibre channel
  • should be transparent. Note "should". We can't test it.

2020-06-29

  • maintenance week ethernet
  • update ethernet switches
  • rolling reboots of all switches

2020-06-24

We need to replace some Hardware damaged by the power outage. Replacement will happen on short notice.

Downtimes for

  • done. wsl007/wsl008 = www-acc.gsi.de
  • done. asl330-asl334 = acc6pro
  • gsi oracle database. Broken Restore pending
  • done. psl003-psl007 = wincc

2020-06-22

  • two to five days
  • Migration oracle gsi database to 19c
  • migration oracle acc database to 19c

2020-06-15

  • Maintenance week
  • Update, Patches, Firmware all systems
  • regular maintenance complete but see power outage notes

  • status power outage greencube
    • on monday 14:00 a power outage in the greencube happened
    • asl730-asl734 failed - fixed. asl730 won't be repaired
    • dbl005 failed - won't be repaired. We migrated to dbl2xx
    • nwsr04 controller failed - repaired

2020-01-14

  • Zks migration el7

2019-08-12

  • Maintenance week
  • Removal of java8 and eclipse-neon

2019-07-30

  • Maintenance acc7dev

2019-05-06

Maintenance window.

Operating system updates on all servers. Including asl74x, asl34x, fileservers, tcl1000, databases, interlock, etc.

Non userfacing systems will be updated before the maintenance window starts.

Java 11 will be rolled out.

Java 8 will be removed from all systems except acc7dev.

For anyone asking why systems are down or if systems are up again, the piggy bank is located at sb1.3.119

fertig.jpg

2019-04-16

Operating system updates including reboots and java 11 on acc7dev (asl740 to asl744) For curious people: that means 1072 software packages including eclipse

acc7dev has been updated.

2019-04-15

With the end of beamtime 2018/2019 the VMS systems (axp*, bel.gsi.de) and old domain controllers (dcw001/dcw002) will be decomissioned.

2019-03-19

git.acc.gsi.de is now in beta state. If nothing serious happens it will be declared productive post beamtime. See also Git and SshAgent

2018-10-15 to 2018-10-19

Maintenance window complete

2018-07-18 to 2018-07-19

Maintenance window complete.

2018-05-18

The new kernel did not help with the nfs problems. We reconfigured the nfs and restarted the systems.

2018-05-14

we still experience nfs problems. We try to patch the kernel. This requires a rolling restart of all acc7 servers.

2018-05-08

asl744 has been rebooted. Reason for nfs problems is unknown. Ticket with redhat is open.

system asl340 lost memory modules. System is currently unavailable. This means OxygenXML is not available

2018-04-10 to 2018-04-11

Maintenance window complete.

2018-03-06

OS upgrades on storage systems. This is transparent and no system should be affected.

2018-01-29 to 2018-02-02

Maintenance window complete

done - Oracle Bundle Patch (accdbp, accdbu, accdbt) on the server side. This affects all Database services including for example LSA. We will try to execute this with a rolling upgrade, node by node, keeping at least one node available. Database sessions require a reconnect on failover.

done - Upgrade Oracle Instantclient on el7 (acc7dev, acc7pro) to version 12.2

Meltdown patches on all user facing systems

  • done - asl730-asl734 acc6dev
  • done - asl330-asl334 acc6pro
  • done - asl740-asl744 acc7dev
  • done - asl340-asl344 acc7pro
  • done - asl102 interlock
  • done - asl103 fe monitor
  • done - tsl001 timing
  • done - dal001 dataacquisition
  • done - dal002 dataacquisition
  • done - psl003-psl005 wincc
  • done partially -tcl1000-tcl10xx thinclient
  • done - vml2x vmware based machines
  • done - vmlax vmware based machines
  • done - vmlbx ovirt based machines
  • done - zkl001-zkl002 zks
  • done - vml003 - vml004
  • done - usl604-usl606

2017-12-04 to 2017-12-05

system maintenance

the acc7 chassis has some defects, requiring dissassembly. For this reason expect that we will need to complete maintenance window and core services like acc7 and nfs won't be available for a longer time.

2017-10-16 to 2017-10-20

Oracle Upgrade to 12c (accdbp, accdbu, accdbt)

2017-07-17 to 2017-07-21

Maintenance window complete.

system maintenance including NFS migration to new hardware. All user facing systems will be down. This includes

  • acc6pro asl330-asl334
  • acc6dev asl730-asl734
  • acc6file asl430-asl432, fsl00c
  • acc7pro asl340-asl344
  • acc7dev asl740-asl744
  • webservices websvpro, websvcdev, packages, olog, www-acc, www.acc, artifcats, builder, ...
  • virtual machines
  • vmware server
  • zks
  • ...

We will stop all user facing services and start with the migration of fsl00c, this includes the home directories. Once migration of NFS is complete will continue with general operating system upgrades. Next we will need to upgrade our storage system fabric.

2017-06-29

dns alias fcmw00a will be removed

2017-03-20 to 2017-03-21

system maintenance complete

2017-01-06 to 2017-01-20

Datacenter relocation. All systems will be down.

Yes that is a time frame of two weeks. And yes you can expect issues once the systems are online again.

For anyone asking if systems are up again, the piggy bank is located at br2.2.152

fertig.jpg

Systems complete:

  • gsi oracle databases
  • gsi ords application servers (cdb)
  • zks
  • www-acc.gsi.de (wiki, subversion, bugzilla)
  • acc6 (asl330-asl334, asl730-asl734)
  • nfs server (fsl00c)
  • acc7 (asl340-asl344, asl740-asl744)
  • websites: websvcpro, webvscdev, packages, olog, ...
  • vmware machines (vmla...)
  • ovirt machines (jenkins, builder.acc.gsi.de)
  • vms (axp, bel.gsi.de)

Change of ACC uplink Network unavailable from 7:00 to 7:30

2016-11-30 oracle migration

ACC Oracle Databases will be migrated to new storage systems. Databases will be unavailable.

2016-08-29 maintenance week

maintenance complete.

Known Issues:

  • x-win32@acc6: x-win32 is incompatible with Redhat Enterise Linux 6.8.
    If you need to connect to acc6 (asl73x, asl33x) please use a workaround
    • use cygwin
    • connect to acc7 using x-win32 and connect to acc6 via ssh and x forwarding.
    • a newer x-win32 version will fix this, but is not (yet) available at gsi
    • new version of x-win32 available at softwarecenter

2016-06 jenkins

the buildserver will should be migrated to new hardware end of june?

2016-03-15 reset scuxl

all scus will be resetted

2016-03-14 maintenance week

maintenance complete.

Starting Monday 14. march to Friday 18.

Software updates and reboots of all services.

status 2016-03-15:
  • acc6 done; asl73x, asl33x, asl43x
  • webserver done, wsl00x
  • logstash done; usl30x
  • timing done; tsl001
  • usl60x done
  • vml00x done
  • vml200x done
  • wincc done; psl004
  • anything zkl related

2016-03-07 fsl00t retirement

NFS Server fsl00t will be retired.

2016-01-25 maintenance week

Starting Monday 25. Jan to Friday 29

System maintenance complete

  • fibre channel core switches relocated
  • 12 servers moved
  • over 150 cables layed
  • over 50 servers updated

2016-01-04 subversion structure

subversion will be restructured.

The repository bel will be frozen.

Commit all changes before and prepare to create new workspaces.

See also SubversionStructure

status 2016-01-04:
  • new repositories created
  • bel renamed to bel-archiv
  • bel-archiv is read only
  • some repository data migrated (including history)
  • please migrate other data (excluding history) using svn export and svn import

2015-12-07 maintenance week

Starting Monday 07. Dec to Friday 11

System maintenance. Expect major service interruptions.

Status:
  • wsl00x done -> www-acc, wiki, subversion
  • asl43x done -> NFS, home,
  • asl33x done -> acc6-PRO, artifacts
  • asl73x done -> lsa server
  • usl30x done -> logstash
  • asl102 done -> interlock
  • usl602 done -> buildserver
  • usl603 done -> virtual scu
  • tsl001 done -> timing
  • zkl00x done -> zks
  • vml00x done -> fesa build, rpmbuild, etc

pending:
  • zks terminals reboot (updates are done)

maintenance week closed.

2015-11-30 webserver migration

Starting 10:00, expecting 4h of downtime.

migration of www-acc.gsi.de and www.acc.gsi.de to new hardware.

failure of wiki, subversion, bugzilla and other webservices using these domains.

Migration is complete. https certificate and kerberos tickets are working.

2015-07-23 Upgrade Artifacts

Upgrade artifacts.acc.gsi.de. Starting 10:00 expecting 2h downtime

2015-05-04

Starting Monday 04. May to Friday 08

System maintenance. Expect major service interruptions.

System updates. Network Firmware updates. Will shutdown acc6 and acc5 cluster for a few hours.

Status update 2015-05-04: network and storage switches are patched, acc5 and acc6 are patched. Database systems will be patched on Tuesday

Status update 2015-05-05: maintenance complete

2015-04-07

Default java will switch to java 8. Typing "java" will result in a java 8 runtime.

Default eclipse version will be luna. Start with "eclipse-luna" The alias "eclipse", currently pointing to kepler, will be removed

2015-03-16

Starting Monday 16. Mar to Friday 20

System maintenance. Expect major service interruptions.

Depending on completion of electric power installation, the current plans include physical movement of storage systems and bladecenters to new racks.

Interruption will include acc5 (asl72x) acc6 (asl73x), webservers (wsl00x), fileservers (fsl00c, fsl00t), network boot, oracle databases (acc and gsi), zks, ...

Status update 2015-03-18:

Blade enclosures and storage systems have been moved. Most user facing services/machines are still powered down. We expect to restore main services (acc5 and acc6) on Thursday.

Status update 2015-03-19:
  • acc6 is up and running
  • acc5 has a hardware defect, expected to be running friday
  • other services (jenkins, logstash) are powered off
Status update 2015-03-20:
  • acc6 is up and running
  • acc5 is up and running
  • other services (jenkins, logstash) are powered off

2015-01-12

Starting Monday 12. Jan to Friday 16. System maintenance. Expect major service failures.

Current plans include physical movement of servers and blade centers. Expect hours/days of downtime for linux cluster acc5, acc6, NFS servers fsl00t, fsl00c, tftp services, etc.

Status update 2015-01-13: Software updates are mostly complete. Waiting for electric installation to finish before moving servers.

Status update 2015-01-19: Electric installation not completed. New maintenance window in march

2014-11-11

all el6 systems received an subversion upgrade to 1.7. For details see Subversion

2014-10-23

For security reasons caused by the ssl poodle bug, SSLv3 has been deactivated on our webservers. This affects the subversion connection of eclipse-indigo on the acc5 cluster. Subversion access on acc5 is only possible using the command line client.

2104-08-06

new java version 1.7.0_67 on acc6. Check your eclipse settings.
This update fixes security issues and a java webstart bug.
Starting with this release eclipse settings should be stable during upgrades.

2014-07-24

acc6-file had a failure from 16:40 to 16:50. This affected for example all home filesystems on acc6-pro and acc6-dev.

One of the cluster protocols was lacking redundancy. Configuration changes to solve this issue froze a cluster service and a reboot was required. Sorry for the interruption. Reboot of all acc6 clusters (file, pro, dev) is complete.

2014-05-26

System maintenance complete

2014-05-21

2014-05-21, 2014-05-22, 2014-05-23, 2014-05-26 system maintenance. Expect service failures. date changed maintenance now includes friday

2014-05-13

new java version 1.7.0_55 on acc6. Check your eclipse settings

2014-05-08

webserver certificates for www-acc.gsi.de and www.acc.gsi.de have been refreshed. To update subversions cache use svn info https://www-acc.gsi.de/svn/bel

2014-03-26

new java version 1.7.0_51. Check your eclipse settings.

2013-11-25 System updates.

2013-11-25 from 08:00 to estimated 2013-11-28 18:00

Affected: all machines and services

2013-08-26

2013-08-26 from 08:00 to estimated 2013-08-26 14:00

Affected: Blade chassis blc292 with the hosts asl73x, psl00x

Modification of enclosure network uplink.

2013-07-08

Systemaktualisierung.

Zeitraum 2013-07-08 ab 08:00 bis vorraussichtlich 2013-07-10 16:00

Betroffen sind von IN betreuten Server. Komplettausfall von acc6, webserver und datenbanken. Service unterbrechungen von allen weiteren IN Diensten. Cluster (acc5, acc6), Webserver (subversion, wiki), Buildsystem (maven repository, jenkins).

2013-05-22

Migration auf JDK7 auf allen Systemen.

2013-04-08

Systemaktualisierung.

Zeitraum 2013-04-08 ab 08:00 bis vorraussichtlich 2013-04-10 16:00

Betroffen sind von IN betreuten Server. Komplettausfall von acc6, webserver und datenbanken. Service unterbrechungen von allen weiteren IN Diensten. Cluster (acc5, acc6), Webserver (subversion, wiki), Buildsystem (maven repository, jenkins).

Update 2013-04-08 14:00:
Ein Firmware upgrade im SAN ist fehlgeschlagen, wir nehmen fuer heute erstmal alle Dienste hoch und werden uns morgen nochmal mit dem upgrade beschaeftigen. Mit erneuten Ausfaellen ist zu rechnen.

2013-01-14

Systemaktualisierung.

Zeitraum 2013-01-14 ab 08:00 bis vorraussichtlich 2013-01-16 16:00

Betroffen sind von IN betreuten Server. Die Cluster (acc5, acc6), Webserver (subversion, wiki), Buildsystem (maven repository, jenkins). Sowie der Netzwerkbetrieb.

2013-01-07

Aufgrund von Wartungsarbeiten an der Klimaanlage Abschaltung aller IN Server.

Zeitraum 2013-01-07 ab 16:00 bis vorraussichtlich 2013-01-08 16:00

Betroffen sind alle von IN betreuten Server. Alle Cluster (axp, acc5, acc6), Webserver (subversion, wiki), Buildsystem (maven repository, jenkins)

Es wird versucht den Netzwerkbetrieb aufrecht zu erhalten.

2012-12-11

Aufgrund von Wartungsarbeiten an der Klimaanlage Abschaltung aller IN Server.

Zeitraum 2012-12-11 ab 16:00 bis vorraussichtlich 2012-12-12 16:00

Betroffen sind alle von IN betreuten Server. Alle Cluster (axp, acc5, acc6), Webserver (subversion, wiki), Buildsystem (maven repository, jenkins)

Es wird versucht den Netzwerkbetrieb aufrecht zu erhalten.
Topic revision: r137 - 30 Jun 2020, ChristophHandel
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback