ATLAS Worker Node

Introduction

This page illustrates the software used to support Grid interactions in OSG of ATLAS pilots. There are 2 packages:

• wn-client maintained by OSG and installed by system administrators
• ATLAS-wn maintained by ATLAS software managers and installed by them (in $OSG_APP)

The remaining of this page contains installation instructions and package description. These instructions are for system administrators, ATLAS software managers and ATLAS (Panda) developers. These are not user packages. If you are looking for a client for interactive use, please install WlcgClient. Development notes about ATLAS clients are in AtlasClientsDevelopment.

NOTE that DQ2Clients included in ATLAS-wn require python 2.6: the install python 2.6 tells you how to install python 2.6; if you can use only python 2.4 check the install old ATLAS-wn-0.1.35b section below.

Installation of wn-client

The Worker Node client (wn-client) is a package required by OSG and installed in $OSG_GRID as mandated by OSG. All jobs running on some OSG site must be able to fine the content os wn-client in $OSG_GRID if that platform is supported (e.g. You'll find wn-client on all Linux hosts but there is no wn-client for Windows so you'll find none if you run on a Windows cluster)

There is a single Worker Node client in OSG, no VO customized versions, and it does not have a version number by itself. Refer to the OSG/VDT version (printed by the osg-version command) to see if you have the latest available (OSG 1.2.25 as of 11/11/2011 but check VDT News).

%NOTE% The examples listed (installation and update) are extremely simplified, more a reminder for people familiar with it than full instructions. The sections automatic update and alternatives for CA and CRLs in the Advanced topics below explain some more option. Anyway if you are installing wn-client for the first time please read the OSG twiki.

%NOTE% $OSG_GRID (set to /opt/wn in the examples) is defined in the Computing Element (CE) configuration. OSG mechanisms take care to propagate the value to the information system (BDII) and to an environment variable in the jobs runtime environment.

Install wn-client in $OSG_GRID as explained in the OSG twiki. E.g.:

mkdir /opt/wn; cd /opt/wn
pacman -get http://software.grid.iu.edu/osg-1.2:wn-client
source setup.sh
vdt-post-install
vdt-ca-manage setupca --location local --url osg
vdt-control --enable vdt-update-certs
vdt-control --enable  fetch-crl
vdt-control --on

Answer yes (or yall) to the questions during Pacman installation

To update, replace the existing wn-client (after a backup), e.g.:

mv /opt/wn /opt/old_wn
mkdir /opt/wn; cd /opt/wn
# continue as above

%NOTE% This will cause a brief unavailability of wn-client. If you prefer not to risk to interfere with regular production jobs see the alternative (longer) installation below

wn-client is all a system administrator is required to install. The ATLAS pilot will also use additional software but this one will be installed and managed by the ATLAS pilot or the ATLAS software administrator (Xin) in a subdirectory of $OSG_APP. Specifically the software will be installed in $OSG_APP/atlas_app/atlaswn/ and the package installed is ATLAS-wn ( pacman -get http://www.mwt2.org/caches/:ATLAS-wn), containing dq2-client and other ATLAS specific software auxiliary to the jobs. Anyway system administrators have not to worry about this (Just FYI).

Installation of ATLAS-wn

The package ATLAS-wn has been created in order to rationalize and group together all middleware packages used by ATLAS (US-ATLAS) and not provided by OSG. This is installed by the ATLAS software manager on all sites running ATLAS jobs in a well known subpath of $OSG_APP (currently $OSG_APP/atlas_app/atlaswn/). This package is similar to OSG's WN-Client in the sense that it contains software that must be available to all ATLAS jobs

• the path has to be mounted on all WNs and headnodes (wherever batch and fork jobs run
• the setup file must be sourced by the ATLAS pilot

but it is VO specific, it contains software controlled and needed only by ATLAS (therefore not included in OSG's wn-client).

At the moment it contains DQ2Clients? and a compatibility package.

%WARNING% $OSG_GRID (set to /opt/wn in the examples) must be defined in the environment and OSG wn-client must be installed there before starting the Pacman installation of ATLAS-wn

To install ATLAS-wn you need Pacman, e.g.:

source $OSG_APP/pacman/setup.sh
mkdir $OSG_APP/atlas_app/atlaswn; cd $OSG_APP/atlas_app/atlaswn
pacman -get http://www.mwt2.org/caches/:ATLAS-wn

To update ATLAS-wn you can do a new install and link the previous path, e.g.:

mkdir $OSG_APP/atlas_app/atlaswn_new
#install as above in $OSG_APP/atlas_app/atlaswn_new
mv $OSG_APP/atlas_app/atlaswn $OSG_APP/atlas_app/atlaswn_old
ln -s $OSG_APP/atlas_app/atlaswn_new $OSG_APP/atlas_app/atlaswn

%NOTE% As of today ATLAS-wn is a package adding features used mainly in analysis jobs and the recommended setup is not to fail the pilot if the package is not there. This allows for updates while the site is online and running production jobs.

Use of wn-client and ATLAS-wn

There is no interactive use. Pilots (or other programs) will source both wn-client and ATLAS-wn setup files to have all the required clients. Supposing that command_string contains the command you want to execute, in your program you may use something like the following Python fragment:

setup_string = "source $OSG_GRID/setup.sh"
if os.path.exists("$OSG_APP/atlas_app/atlaswn/setup.sh"):
    setup_string += "; source $OSG_APP/atlas_app/atlaswn/setup.sh"
ecode, out = commands.getstatusoutput(setup_string+'; '+command_string)

wn-client is guaranteed to exist on every OSG CE, some installations may not have ATLAS-wn so it is safer to check for it. The fragment above is a simple example, in your code you may want to add at least some exception or signals handling (try... except) and you may want to check separately for the environment variables and substitute them in the strings (e.g. using os.getenv).

Advanced topics

Here is a collection of more advanced and uncommon topics.

Automatic update of certificates and CRLs

If you want a reliable client you must update your certificates, either manually, or, better, automatically with vdt-control

• You can enable and start a service to maintain updated your certificates and CRLs:

  vdt-control --enable vdt-update-certs
  vdt-control --on vdt-update-certs
  

• Now you can use vdt-control to control the services running on the machine (e.g. to list, start and stop the services):

  vdt-control --list
  vdt-control --on
  vdt-control --off
  

Alternatives for CA certificates and CRLs

vdt-ca-manage allows to pick the desired CA certificates and install them in an arbitrary location, private or shared with other grid softwares. A complete reference on available certificate sets and where to put the certificates, including an arbitrary location, is available on the VDT site. The default installation above is self contained. This section covers two other common options to install the OSG CA certificates. You can use one of these instead of the CA installation in the instructions above.

Install in /etc/grid-security/certificates

You must install as root to use this system directory:

source /opt/wlcg-client/setup.sh
vdt-ca-manage setupca --location root --url osg
vdt-post-install

This installation by default enables the periodic update of certificates and CRLs.

Link an existing CA installation

You can use an existing CA installation (owned by an other package) on the host or mounted in a shared directory (e.g. via NSF). If several grid softwares share the same CA installation it is important that one is the owner and runs the services to update it and all the other only link to it (without changing the content). Let's assume that $CACERTDIR contains the path of the certificate directory, e.g. /your/cacertdir or /etc/grid-security/certificates. To link to the existing CA installation:

source /opt/wlcg-client/setup.sh
ln -s $CACERTDIR $VDT_LOCATION/globus/TRUSTED_CA
ln -s $CACERTDIR $VDT_LOCATION/globus/share/certificates

If you don't make the links above your wlcg-client may work only partially.

Install and setup Python 2.6

The default python from SL5 is python 2.4 whereas dq2 requires 2.5 or greater. We suggest python 2.6 (and not 2.5 or 2.7) because it is available from EPEL. Site administrators are recommended to install python 2.6 from EPEL (preferred). You need to be root. Add the EPEL repository (if you don't use it already). Note that this will alter your system yum repositories!

rpm -Uvh http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

Install python26:

yum install python26

And link atlasosgcompat/bin/python to pythonloader, so that it will become the default python in the users PATH once they source the ATLAS-wn setup (e.g. supposing your ATLAS-wn installed in $OSG_APP/atlas_app/atlaswn/atlasosgcompat):

ln -s $OSG_APP/atlas_app/atlaswn/atlasosgcompat/bin/pythonloader $OSG_APP/atlas_app/atlaswn/atlasosgcompat/atlasosgcompat/bin/python

pythonloader is a script executing your python scripts with the first python (executable named python or python26) with version <=2.5 present in your PATH.

NOTE: If installing python26 from EPEL is not an option, then click here for instructions for possible alternatives that include a Pacman installation of Python 2.6.

Install ATLAS-wn-0.1.35 to use python 2.4

ATLAS-wn includes DQ2Clients? , the ATLAS data management clients. Since version 0.1.36 DQ2Clients? require python 2.5 or greater even if the system python in SL5 is python 2.4. We really recommend to install and use python26 (see above) because DQ2Clients? 0.1.35 is old and buggy, but if you cannot do otherwise, there is a version of ATLAS-wn that includes DQ2Clients? 0.1.35 and hence will work with python 2.4. Just use ATLAS-wn-0.1.35b instead of ATLAS-wn in the install/update commands above (e.g. =pacman -get http://www.mwt2.org/caches/:ATLAS-wn-0.1.35b=).

Problems by mixing 32bit and 64bit software

All packages in ATLAS-wn are not system dependent and work fine on 64 and 32bit platforms (the MySQL binding included in DQ2Clients that were only 32bit have been removed before August 2009). All packages in wn-client exist in both 32 and 64bit version and OSG/VDT will choose the correct one for your system at installation time matching the platform (OS) in the installation host. In mixed environments:

• A 64 bit OS can support both 64 and 32 bit applications and libraries
• A 32 bit OS supports only 32 bit applications and libraries
• Python compiled libraries (e.g. LFC binding included in wlcg-client) have to match the interpreter. E.g. 64bit Python requires 64bit bindings, 32bit Python requires 32bit bindings (therefore 32bit wn-client).

The LFC libraries are an exception. Since OSG 1.2.21 LFC libraries are selected at run time and a 64bit installation will work correctly with python 2.4/5/6 32 or 64 bit. So don't worry about this is you have all nodes with 64 bit OS and ATLAS sw is 32 bit. It will work correctly with a regular installation on a 64 bit node.

If one of the above is violated you get errors often difficult to identify. The simple solution for mixed environments (32/64bit OSes with a single shared wlcg-client installation) is to install everything in 32bit version:

• You need to install 32bit compatibility libraries on 64bit OSes (e.g. Here is a list for SLC4)
• During each Pacman installation add the option -pretend-arch i686 (e.g. pacman -pretend-arch i686 -get wn-client) to force a 32bit installation. Pacman is actually caching this setting, so it is important you use -pretend-arch i686 the first time you install something in that directory (Pacman cache). Here is VDT documentation about -pretend-arch i686.
• NOTE: the environment variable: export VDT_PRETEND_32=1 (that was set before the VDT installation) is not working anymore.
• If you need Python 32 bit you can install on your own or use: pacman -get http://www.mwt2.org/caches/:Python32bit. The Python delivered is Python 2.4 (C API 1012 required by _lfc.so) and it is compiled on a SL4.7. The Pacman installation can be in its own directory or on top of wn-client (recommended if you installed it 32 bit and you will need Python 32 bit only for it). If you wonder, no -pretend-arch i686 is needed because this package is only 32 bit, anyway it is fine if you added the option.

More cautious update

Panda pilots can override the default wn-client ($OSG_GRID). If you prefer not to risk to interfere with regular production jobs you can pick an alternative install directory but in this case you'll have to communicate the new directory to whoever is testing the pilots (Panda developers and the Panda shift team) and let them know to use for the test jobs that directory (the one where you installed wn-client_dev). They will will override OSG_GRID and have the pilot source the setup file in that directory.

-- MarcoMambelli - 07 Oct 2008