Archiving System Workshop Protocol (08.2018)

Dates: 29.08.2018 - 31.08.2018 (3 Days)

Participants: Marko Kuder (XLAB), Vito Cucek (XLAB), Vitaliy Rapp (GSI)

Location: XLAB Office in Ljubljana

Short overview of discussed topics

Day 1:

• Status at GSI and XLAB
• List of Missing Features (XLAB perspective)
• Comments and Feedback to Design Document v.1.2 (FDR)

Day 2:

• Comments and Feedback to Design Document v.1.2 (FDR) (cont.)
• Major technical issues

Day 3:

• Major technical issues (cont.)
• Priorities of pending features and other work
• Next steps
• Short discussion about the BTM Project

Outcome (for some more detail see also attached minutes from XLAB)

Status:

• The features considering the Development Cycle II are finished and are currently being reviewed by GSI
• Technical Design Document should incorporate feedback before official approval
• XLAB is working on monitor (to process events and send notifications to app) and restructuring the app for more intuitive interface
• With respect to changes and features, which implementation is required the project duration should probably be extended by several month. The exact amount will be communicated after preliminary estimation of the complete effort. Currently proposed (draft) project deadline is May 2019

Technical:

• Storage:
  • Index Structure: for the STS a time based index structure is used, where one index represents one day. For LTS each reduction algorithm is assigned to one index
    • The drawbacks of this approach must be evaluated: disk space usage, search performance
  • The usage of the document-types will be removed
  • The new way to storing the device (source) data must be evaluated in order to check the ability of the Elasticsearch DB to compress the unindexed data
  • Selector components must be indexed and be present at the top level storage, making selection on specific BPC IDs fast
• Configuration:
  • The configuration of the services should be stored inside the zookeeper component, therefore only one setting parameter (zookeeper address) should be provided to the services
• Reductor:
  • Reduction Algorithms should work on device/parameter level and not on the field level, their output however may contain multiple fields
  • Reduction Algorithms should take the selector into account and preserve it in the reduced data, making searches inside of reduced data based on BPC IDs (or other parts of the selector) possible
  • An index should be deleted if all data it contains is reduced AND the data is older then specific timestamp
• Collector:
  • Currently the Data coming from devices is split into raw- and meta-data. This split is performed by the collector nodes. To provide the proxy function for the system this split should either be the writer or the data should be combined by the Proxy-Component
  • The current way to load balance the collector node requires multiple manual steps to perform. This should be reworked, so the balancing (and fault tolerance) should be provided automatically - simmilar to the reductor service. Ideally the AS administrator should only configure the sources, while the actual load balancing is then performed by the AS itself
• Other components:
  • Some simple API Gateway should be implemented instead of Apache Reverse proxy, this API component should read the data from the zookeeper to acquire the knowledge about currently running instances. This will make a simple load balancing easily achievable

Organizational:

• A Backlog document containing the pending features and changes with their priorites will be created
• To ease the integration process at GSI shorter release cycles should be established