Diagnostic Logging System

Legacy notes, see DiagnosticLogging

The diagnostic logging services is provided by CSCOIN by running a graylog cluster to receive and store diagnostic logging messages.

Guidelines

• consult cscoin before logging to this cluster
• do not permanently send debug messages
• graylog has an automatic logfile retention

Usage

Connections

• DNS name is graylog.acc.gsi.de
• IP is 140.181.134.178
• Receivers are active for syslog (port 514), gelf (12201)

Archive Browser

login with your ACC account on https://graylog.acc.gsi.de/

Live Viewer

A prototype live viewer is available on acc7dev. The tool is called graylog

user@asl740: graylog tail --source scuxl0083 "much too late"
2018-09-20T11:30:41.305Z  scuxl0083            PowerSupplySis18_DU 
scuxl0083 PowerSupplySis18_DU: 2018.09.20 11:30:41:086590 FESA.USR.PowerSupply.RealTime.VoltageSetRt src/PowerSupply/RealTime/VoltageSetRt.cpp:606 ERROR  (delayToEvent 74.8818 / duration 11.8079)  hardware event for GS12QS2D SeqIndex: 3 BPIndex: 3 GroupID: 300 EventNumber 512 Delay to HwEvent: 83.1328ms EventExecutionTimeStamp = 1537443077998.706055ms ramp preparation ended after 86.6897ms -> much too late!

2018-09-20T11:30:41.305Z  scuxl0083            PowerSupplySis18_DU 
scuxl0083 PowerSupplySis18_DU: 2018.09.20 11:30:41:101384 FESA.USR.PowerSupply.RealTime.VoltageSetRt src/PowerSupply/RealTime/VoltageSetRt.cpp:606 ERROR  (delayToEvent 45.5637 / duration 12.2568)  hardware event for GS12QS2D SeqIndex: 3 BPIndex: 4 GroupID: 300 EventNumber 512 Delay to HwEvent: 54.0747ms EventExecutionTimeStamp = 1537443078042.604492ms ramp preparation ended after 57.8206ms -> much too late!

On first usage it will ask for a password and create an access token, stored in ~/.config/graylog.token

If the query is too broad and receives too many messages tail will terminate.

Note: this is a quick and dirty hack, it sure has bugs, it handles errors by throwing stacktraces, it drops messages if logrates are too high, it will be removed if it produces too much strain on the servers.

Syslog Emitter (c, c++, etc.)

Any process can output syslog format to graylog.acc.gsi.de:514

CMW Log Emitter

evaluation of a native gelf emitter for cmw log (used by fesa) is under way by cscosv

see https://www-acc.gsi.de/svn/cscosv/cmw/cmw-log-cpp/branches/gelf/

for the gelf library see: https://github.com/serge-14/gelflib

Java Emitter

use the cscoap common dependencies package. see: https://git.acc.gsi.de/fcc-commons/common-dependencies/src/branch/master/src/main/resources

for webservices/servlets add an additional dependency on log4j-web and specify servlet context parameter log4jConfiguration to switch logging configs.

python emitter

use python standard logging and GelfUdpHandler from the pygelf logger (https://github.com/keeprocking/pygelf)

Example

from pygelf import GelfUdpHandler
import logging

gelfHandler = GelfUdpHandler(host='graylog.acc.gsi.de', port=12201, _my_static_field = "myData")
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.addHandler(gelfHandler)
logger.info('hello gelf')

Message fields

Graylog has active input pipelines parsing syslog fields, timestamps, modify hostnames, etc.

Static fields already in use in other applications:

• _program
• _nomen
• _user_name