ATLAS Clients Development

Introduction

This page is a collection of notes about the development of ATLAS clients described in:

• wlcg-client: interactive user client
• ATLAS-wn: client for batch (Grid) jobs, used together with OSG wn-client

Other documents explain the configuration and use of the clients with other ATLAS packages:

• DQ2ClientsInOSG: how to use DQ2Clients in OSG

Temporary documents for the update to OSG 1.1.25:

• UpdateOSG (for site administrators)
• UpdateOSGFull (for the ATLAS software manager)

End-users and system administrators should refer to the documents above. This document is a collection of explanations, historical and development notes for developers and maintainers of the ATLAS clients.

Known issues / Fixed issues

Setup of ATLAS releases removing Expat from the library path

Expat is an XML parser used by LFC, DQ2Clients, Athena and other programs. ATLAS releases (at least versions 14 to 16.6) during their setup change the LD_LIBRARY_PATH environment variable removing all paths including a directory named "expat", specifically $VDT_LOCATION/expat/lib and $VDT_LOCATION/lib32addon/expat/lib that contain the 64 and 32 bit versions of the expat libraries. This happens if Athena setup is executed after the Grid (wlcg-client or wn-client) setup (that includes those paths in LD_LIBRARY_PATH). Normally (e.g. in Panda pilots and Pathena) ATLAS setup is executed last, after all Grid setups.

This causes the LFC commands (requiring the expat library) to fail. It was affecting jobs using TAGs or back-navigation and showing as a security error in LFC commands.

ATLAS-OSG-compat included in ATLAS-wn from v0.14 on fixes this problem and is also a placeholder for future compatibility fixes.

Both wlcg-client (from v1.0) and ATLAS-wn (from v0.14) solve the issue with the following workaround performed during the Pacman installation:

1. create a symbolic link to the expat directories in a atlasosgcompat directory
   • keepexpat -> $VDT_LOCATION/expat
   • keepexpat32 -> $VDT_LOCATION/lib32addon/expat
2. add the new directories (links) to LD_LIBRARY_PATH

This way when the other paths are removed, the new ones (atlasosgcompat/keepexpat/lib and atlasosgcompat/keepexpat32/lib

This workaround allows not to worry about the order of execution of the setup files. This has been tested successfully with grid commands, DQ2 commands and Athena jobs.

Alternative solutions could be:

• compile expat statically with the LFC binding
• install expat libraries in the system path (so that they can be found without being in LD_LIBRARY_PATH)

32 and 64 bit

The following 2 issues both regard intermixing of 32 and 64 bit OSes and applications. There may be subtle errors coming from mixing 32bit and 64bit software (e.g 'library not found' errors).

A node it is considered being 32 bit if it runs a 32bit OS; 64 bit if it runs a 64 bit OS.

On a 32bit OS you can run only 32bit software and libraries. A 64bit OS with compatibility libraries installed can run (beside 64bit software) also 32bit software without problems:

• python
• wn-client
• ATLAS releases

If a 64bit application is run (e.g. python 64bit) it will require 64bit libraries. If a 32bit application is run (e.g. python 32bit) it will require 32bit libraries.

Please do not mix the 2 following issues:

1. Mixed environment with 32 and 64 bit nodes: provide a shared installation, e.g. via NFS, of wlcg-client or wn-client, to run in a cluster with some 32bit nodes and some 64bit nodes
2. Running python 32 bit or 64 bit nodes: being able to use LFC commands in a 64bit node after setting up Athena (and its 32bit python)

Mixed environment with 32 and 64 bit nodes

All packages in ATLAS-wn are not system dependent and work fine on 64 and 32bit platforms (the MySQL binding included in DQ2Clients that were only 32bit have been removed before August 2009). All packages in WLCG-Client and WN-Client exist in both 32 and 64bit version and VDT will choose the correct one for your system at installation time. VDT will also install automatically the 32bit version of the LFC python bindings (and all the 32bit libraries needed by them) every time the 64bit version of the LFC python bindings is installed. This was required by ATLAS.

Confusion may arise If a cluster has shared installation directories and mixed Worker Nodes (32 and 64 bit OSes), then all the grid middleware should be 32bit in order to run on both. In this case you'll have to force VDT against its default and perform a 32bit installation.

Normally VDT chooses the right package for your platform and you should let VDT do it. It will install 32bit packages on 32bit OS, 64bit packages on 64bit OS.

Other errors may arise if you upgraded your system to 64bit and have older 32bit software or libraries that you don't need any more. In that case just remove the 32bit version and let the 64bit one be in your path.

Full 32bit installation

If you have a mixed environment (some 32 and some 64 bit machines) it may be convenient. Avoid this solution if you have a uniform environment (all 32 bit or all 64 bit machines).

• You need to install compatibility libraries, at least the ones suggested in https://twiki.cern.ch/twiki/bin/view/Atlas/RPMCompatSLC5
• Before any VDT installation set the environment variable: export VDT_PRETEND_32=1
• Then install the package with Pacman (e.g. wn-client)
• Most likely you need also Python 32 bit: you can install on your own or use pacman -get http://www.mwt2.org/caches/:Python32bit26 and install it on top of the wn-client Pacman cache (it could be a different one but you'd have to add it to the default environment). The Python delivered is Python 2.6 (C API 1013 supported by _lfc.so) and it is compiled on a SL5.5 32 bit.

Since the VDT installation includes Python C extensions (e.g. LFC client), a 32 bit installation requires a 32 bit Python.

Running python 32 bit or 64 bit nodes

Even on a full 64 bit system ATLAS mostly installs and uses the 32 bit version of Athena. When you setup Athena, it frequently setup its own python interpreter, frequently python 2.6, sometime 2.4 or 2.5, always 32 bit.

Therefore, starting VDT 2.0.0p28, the package containing the LFC python bindings, LFC-Interfaces, contains the bindings for python 2.4/5/6 and on 64bit version of it VDT will also install automatically the 32bit version of the bindings and all the 32bit libraries needed by them (supporting python 2.4/5/6, both 32 and 64 bit) As requested in these ATLAS requirements.

Note that this is done automatically. No special installation flag or extra package is needed.

Install python 2.6

The default python from SL5 is python 2.4 whereas dq2 requires 2.5 or greater. We suggest python 2.6 (and not 2.5 or 2.7) because it is available from EPEL. Here we present 2 alternatives:

• Install python 2.6 from EPEL (preferred)
• Install python 2.6 using Pacman

Install python 2.6 from EPEL

You need to be root. Add the EPEL repository. Note that this will alter your system yum repositories!

rpm -Uvh http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

Install python26:

yum install python26

Change the DQ2Clients package to use python26. Installing python26 from EPEL will not make it your default Python. DQ2Clients uses the default python in your environment (/usr/bin/env python). In order to have DQ2Clients using it you have at least these 2 options:

• add python26 to the PATH using pythonloader (will be hiding the default system one) (preferred)
• add python26 to the PATH as python (will be hiding the default system one)
• change the shebang (hashbang) line in all DQ2Clients executable scripts

Add pythonloader as python in the path

The directory atlasosgcompat/bin part of both the ATLAS-wn and the wlcg-client packages is added to the PATH when the setup file for ATLAS-wn or wlcg-client is sourced. Link atlasosgcompat/bin/python to pythonloader, so that it will become the default python in the users PATH once they source the ATLAS-wn or wlcg-client setup:

ln -s $VDT_LOCATION/atlasosgcompat/bin/pythonloader $VDT_LOCATION/atlasosgcompat/bin/python

This script checks your current python, the one on top of the PATH. If you have already python>= 2.5 does nothing and uses that one to run your python script. If not, it checks in possible places where python 2.5 or greater could be (all the python or python26 executables in the path) and uses the first one it encounters (instead of the one on top of the PATH) to run your python script. You don't choose exactly which version to run, id depends on the PATH content and order. This solution is the least invasive. It is not changing external packages and it is not replacing your custom python that may have been added to the PATH (e.g. by Athena setup).

Add python26 as python in the path

The directory atlasosgcompat/bin part of both the ATLAS-wn and the wlcg-client packages is added to the PATH when the setup file for ATLAS-wn or wlcg-client is sourced. Adding a link named python will hide/replace the system python, e.g. for ATLAS-wn:

cd /osg/app/atlas_app/atlaswn/atlasosgcompat/bin
ln -s /usr/bin/python26 python

and for wlcg-client:

source /share/wlcg-client/setup.sh
ln -s /usr/bin/python26 $VDT_LOCATION/atlasosgcompat/bin/python

This solution is not changing packages different from ATLAS-OSG-compat but it will change the default python once the setup file is sourced and DQ2Clients my end up using a different python if the PATH is changed afterwards (this is not a problem as long as it is any Python 2.5/6 32bit or 64bit).

Change the shebang line in DQ2Clients scripts

All dq2 scripts start with /usr/bin/env python and the default system python in the path is python 2.4 (RHEL5 based OS). To change them to use python26 do the equivalent of:

cd /osg/app/atlas_app/atlaswn/DQ2Clients/opt/dq2/bin/
for i in * ; do sed -i "1s|#!/usr/bin/env python|#!/usr/bin/python26|" $i; done

This solution is changing an external package (DQ2Clients not developed by us) and needs to be reapplied each time that package is updated/installed, but it will not change the default python and DQ2Clients my always use /usr/bin/python26.

Install python 2.6 with Pacman

Choose the correct package:

• Python32bit26 if you have a 32 bit OS (can work also on 64 bit OS if you need it for a test)
• Python64bit26 if you have a 64 bit OS and prefer 64 bit python (recommended for 64 bit OS)

Go in the same directory where you installed ATLAS-wn or wlcg-client and install it, e.g.:

pacman -get http://www.mwt2.org/caches/:Python64bit26

Note, you will have to do it in both the directory of ATLAS-wn and the one of wlcg-client if you have both.

This will add python 2.6 as default python once the setup file is sourced, so it is equivalent to the solution adding python26 to the PATH: it is not changing DQ2Clients package but it will change the default python once the setup file is sourced and DQ2Clients my end up using a different python if the PATH is changed afterwards (this is not a problem as long as it is any Python 2.5/6 32bit or 64bit).

Avoid ExitCode=1 in voms-proxy-info

A lot of ATLAS software (DQ2Clients, pathena, ...) uses voms-proxy-info to check users credentials. The exit code of "voms-proxy-info -e" has to be 0 in order to pass the test. Historically "voms-proxy-info -e" changed its default behavior between updates. The current one requires cryptographic verification of the issuer of the certificate extensions. This is not a control necessary in a client forwarding the certificate. The old behavior can be obtained setting VOMS_PROXY_INFO_DONT_VERIFY_AC to 'true'. This is done for you in wlcg-client (in the setup). The same is done in ATLAS-wn (again in the setup). This behavior is different from the default VDT installation (OSG-Client or wn-client).

There are 3 other alternatives to have a successful proxy verification:

• use the command line option --dont-verify-ac (e.g. voms-proxy-info --all --dont-verify-ac)
• install the VOMS server certificates in /etc/grid-security/vomsdir
• (new) install the VOMS LSC files

For more information you can check:

• http://vdt.cs.wisc.edu/releases/1.10.1/release-a.html
• http://crt.cs.wisc.edu/Ticket/Display.html?id=3493
• https://twiki.cern.ch/twiki/bin/view/LCG/VOMSLSCfileConfiguration

Naming and content of the package

The client packages used in US-ATLAS include:

• wlcg-client: interactive user client
• ATLAS-wn: client for batch (Grid) jobs, used together with OSG wn-client. It includes also the DQ2Clients? package

Other packages:

• ATLAS-OSG-compat: it is included in ATLAS-wn and contains the ATLAS customizations similar to the ones in wlcg-client

Starting v1.0, wlcg-client is based on OSG 1.2 and VDT 2.0.0. Current wlcg-client 1.7 is a subset of OSG/VDT client with some ATLAS customizations. In the past it has been sometime a superset of OSG client, sometime a smaller package and some other classes packages, wlcg-addonclient and wlcg-client-lite, had similar functions:

• wlcg-client-lite: was released together with wlcg-client v1.5 and 1.6, smaller (about 340MB), installed on ATLAS Tier 3 using ATLAS-Canada framework, it is sufficient to use DQ2 Clients and Panda. It includes commands to handle Grid certificates and proxies, move files and use LFC
• wlcg-client-addon: wlcg-client v1.4 was a smaller version of wlcg-client, it became wlcg-client-lite v1.5. wlcg-client-addon included the remaining packages to make the full superset of VDT/OSG client
• wlcg-client: until v1.7 (and excluding v1.4) wlcg-client was a superset of VDT/OSG clients. It was bigger (about 1.1GB) and contained also packages like Pegasus and GlobusWS

For a complete list of content in your rlease use the command vdt-release.

Release Notes

Example of release notes that can be used as template by the developers

wlcg-client

wlcg-client and wlcg-client-lite 1.6

Date: Tue, 27 Apr 2010 18:31:16 -0500 (CDT)                                                                                                               
From: Marco Mambelli                                                                                                              
To: usatlas-grid-l@lists.bnl.gov, hn-Tier3Support@bnl.gov                                                                                              
Cc: "Rik Yoshida (ANL)" , Doug Benjamin , Asoka De Silva ,                                
    Rob Gardner                                                                                                                     
Subject: wlcg-client 1.6 released                                                                                                                         
                                                                                                                                                          
Dear all,                                                                                                                                                 
WLCG-Client 1.6 is available.                                                                                                                             
It includes the new EDG-GridFTP-Client and gLite-FTS-Client.                                                                                              
                                                                                                                                                          
wlcg-client had a name change to avoid confusion with pre 1.4 versions:                                                                                   
 * wlcg-client 1.6 is back to be the full client with all the job submission tools (1.1 GB).                                                              
 * wlcg-client-lite 1.6 is the smaller client sufficient to use with DQ2 clients and Panda, to use on Tier 3 (about 340MB, without CondorG, WS-GRAM       
and other packages).                                                                                                                                      
                                                                                                                                                          
Both are based on the latest VDT 2.0 and OSG 1.2. For a list of all supported platforms see:                                                              
http://vdt.cs.wisc.edu/vdt/releases/2.0.0/requirements.html                                                                                               
                                                                                                                                                          
Installation instructions are on the usatlas twiki:                                                                                                       
http://www.usatlas.bnl.gov/twiki/bin/view/Admins/WlcgClient                                                                                               
                                                                                                                                                          
     pacman -get http://www.mwt2.org/caches/:wlcg-client                                                                                                  
or                                                                                                                                                        
     pacman -get http://www.mwt2.org/caches/:wlcg-client-lite                                                                                             
                                                                                                                                                          
Thank you for any feedback about the package or the documentation.                                                                                        
                                                                                                                                                          
The update is recommended for all installations (new and existing).                                                                                       
There are some bug fixes and EDG-GridFTP-Client and gLite-FTS-Client (required by "DQ2 on steroids") have been added.                                     
                                                                                                                                                          
You should install:                                                                                                                                       
  * wlcg-client-lite : if you will only use DQ2 clients and pathena/prun.                                                                                 
  * wlcg-client : if you want a complete user client (includes everything                                                                                 
   in wlcg-client-lite and more. It is a superset of OSG client,                                                                                          
   customized for ATLAS)                                                                                                                                  
                                                                                                                                                          
Best regards,                                                                                                                                             
Marco Mambelli                                                                                                                                            

References

Installation documents:

• WlcgClient
• AtlasWorkerNode
• DQ2ClientsInOSG
• UpdateOSG