LoggingServicesP1

Goal

The goal is to collect to a central logging host ddm-log.uchicago.edu:

• The dq2.log file from each site running DQ2 0.3 site services.
• FTS logfiles from the central server at BNL.

The resulting logfile collection will be visible to the DDM troubleshooting console at http://ddm-log.uchicago.edu:8000/.

Background

The syslog-ng logging infrastructure is used to forward logging information to central logging hosts for troubleshooting. This version is based on:

• DQ2 0.3 logfiles, dq2.log.
• This version is based the VDT 1.7.1 distribution containing syslog-ng 2.0.4.
• More information can be found at:
  • The VDT 1.7.1 release, http://vdt.cs.wisc.edu/releases/1.7.1/
  • Syslog-ng as distributed by VDT, http://vdt.cs.wisc.edu/components/syslog-ng.html
  • The pacman package, http://vdt.cs.wisc.edu/vdt_171_cache/Syslog-ng.pacman

Pre-installation

• Make sure you have the appropriate Pacman for VDT 1.7.1. The one recommended for OSG 0.6 will do.
• See https://twiki.grid.iu.edu/twiki/bin/view/ReleaseDocumentation/PacmanInfo

Installation

• Syslog-ng in the new release of VDT 1.7.1 is located at http://vdt.cs.wisc.edu/vdt_171_cache/Syslog-ng.pacman.
• Login as root
• Get the recommended pacman; installed in /opt, setup.
• mkdir /opt/syslog-ng/ and cd to it.
• =# pacman -get http://vdt.cs.wisc.edu/vdt_171_cache:Syslog-ng=

Configuration for a DQ2 0.3 site service

• # source /opt/syslog-ng/setup.sh
• # cd /opt/syslog-ng/vdt/setup
• # ./configure_syslog_ng --server y
• # ./configure_syslog_ng --central-host ddm-log.uchicago.edu:5142
• # ./configure_syslog_ng --add-source "/var/log/dq2/dq2.log" ( On AGLT2 this file was "/tmp/dq2.log")

The above writes a configuration file in /opt/syslog-ng/syslog-ng/etc/syslog-ng.conf.

Additional configuration changes for local testing (see next item)

• Edit the syslog-ng.conf file.
• Add the following, in the appropriate sections of the file (specific directories are optional):

# Sources
source test_src { unix-stream("/tmp/syslog-ng-test_src.socket");  };

# Destinations
destination test_dst { file("/tmp/syslog-ng-test_dst.log" perm(0644) ); };

# Define what should be forwarded to the destinations
log { source(test_src); destination(test_dst); };

• Note: these changes will be lost on subsequent invocations of the VDT ./configure_syslog_ng command. So you'll have to remember to edit the config file if you do that.

At AGLT2 we were already running syslog-ng (for regular central logging). The previous incarnation of syslog-ng for DDM was named syslog-ng-ddm. Now it seems to be name syslog-ng. I resolved this on our host by renaming our service in /etc/init.d to syslog-ng.local.

Are the Local Testing Instructions Necessary?

The installation at SWT2 came with local testing enabled using:

source test_src  { unix-stream("/tmp/syslog-ng-test");  };
destination test_dest      { file("/tmp/syslog-ng-test.log"); };
log { source(test_src); destination(test_dest); };

This can be tested with:

# /opt/syslog-ng/syslog-ng/sbin/logger-ng -u /tmp/syslog-ng-test "This is a test message."

The output arrives in /tmp/syslog-ng-test.log

Starting up the service

• # vdt-control --on

enabling init service syslog-ng... ok

You should see a daemon process on your system, daemon    2990     1  0 04:12 ?        00:00:00 /opt/syslog-ng/syslog-ng/sbin/syslog-ng -f /opt/syslog-ng/syslog-ng/etc/syslog-ng.conf -p /opt/syslog-ng/syslog-ng/etc/syslog-ng.pid

• By default, a logfile for the service itself is written to: /tmp/syslog-ng.log.

Validation

• For local testing, if you added additional configuration for local testing:
  • # /opt/syslog-ng/syslog-ng/sbin/logger-ng -u /tmp/syslog-ng-test_src.socket "This is a test message."
  • You should see the message in the logfile, /tmp/syslog-ng-test_dst.log.
• For validation of dq2.log forwarding to ddm-log.uchicago.edu:
  • Email Rob (rwg@hep.uchicago.edu) that you are now forwarding.
  • The logfile will then be visible in Splunk (http://ddm-log.uchicago.edu:8000) - but only after Splunk has been configured to index it.
• Be sure to in the SiteCertificationP1 table.

Stopping the service

• This can be done with #vdt-control --off.

FTS logging

• To be done. Might be able use the syslog-ng injector tool for this (see VDT page above).

Previous twiki installation instructions

• Original testing instructions for DQ2 0.2.12, http://twiki.mwt2.org/bin/view/Logging/SyslogInstall
• Instructions based on VDT 1.7.0, see LoggingServicesP1ver1

-- RobertGardner - 25 Jun 2007