Working with ATLAS offline software at U.S. ATLAS Tier 1 Center

The related topics:

• Atlas Software Releases Installation and How To Use Them
• Code Repositories
• BNL Mirror of ATLAS Pacman Cache
• Data Storage
• Information And Help Resources

Accounts, file systems, computers

Please refer to the Tier I Computing Facility (ACF) help pages for details on accounts, file systems, and ACF computers. Here is a short summary.

• In order to obtain a new computer account at the ACF a prospective user has to read and agree to the Brookhaven National Laboratory computer user agreement. Your supervisor will be asked to confirm the request for a new account. If he sends the confirmation message simultaneously with your request then the account will be probably set faster.
• The computers at ACF are behind firewall. For access, first you need to ssh to the gateway machine atlasgw.bnl.gov. Then you can log into interactive Linux farm nodes, acas00N.usatlas.bnl.gov farm nodes (N=0-8). There is also the general purpose interactive Sun workstation at atlas00.usatlas.bnl.gov with larger selection of programs (e.g. netscape, xv, xfig that are not installed on acas machines). However ATLAS software is not available on Sun OS.
• Many US ATLAS users have CERN computing accounts. Read here how to get them (waiting time is one week or more).
• After login the user is in his NFS home directory with the path stored in $HOME variable. Home directories are backed up but of rather limited size (500 MB is standard). There is a scratch (not backed-up) NFS system at /usatlas/scratch/<username> for temporary file storage. When the scratch area becomes full the older and larger files are automatically purged. ACF has also established AFS cell, /afs/usatlas.bnl.gov, for sharing files with remote institutions. Users have personal AFS areas at /afs/usatlas.bnl.gov/users/<username> and all ATLAS software is installed on AFS.

Login Environment

The U.S. ATLAS accounts get automatically the standard login files in their home directories, such as .bashrc and .cshrc, with the following contents:

if ( -r /usr/local/lib/hepix/shells/hep/central_env.[c]sh ) then
   source /usr/local/lib/hepix/shells/hep/central_env.[c]sh
endif

This installs initial (very basic) U.S. ATLAS environment by sourcing a file on the central file server. A user can modify the .bashrc and .cshrc files as desired. The rather complex ATLAS software environment is managed by CMT release tool. It can not be set at login because of high dependence on the particularities of the concrete ATLAS software releases and use cases. After the login a user needs to proceed with the CMT (ATLAS release management tool) setup. CMT then customizes the ATLAS software environment, e.g. installs proper paths to compilers and external packages in $PATH and $LD_LIBRARY_PATH.

Organization of ATLAS software

ATLAS software is organized in large software releases (more than 1000 packages). New stable development releases of atlas software are produced in three weeks intervals (there are also nightly builds for checking the latest submissions, but beginners are not encouraged to use them). The names of stable ATLAS releases are X.Y.Z, where X,Y,Z=0-9. The "production" ATLAS releases have Y=0, all other releases are "development". ATLAS software releases depend on many external packages (about 70). The main OO framework, ATHENA, is based on the GAUDI architecture that is technically also an external package.

At the U.S. ATLAS Tier I Computing Facility the software for releases 13.0.0 and higher as well as nightly releases are downloaded from pacman distribution kits at /afs/usatlas.bnl.gov/software/downloads.

The software organization for releases preceding 13.0.0 mirrors the CERN ATLAS site. The major areas are

• Stable "monolithic" releases: /afs/usatlas.bnl.gov/software/dist
• Project oriented releases (incl. nightlies): /afs/usatlas.bnl.gov/software/builds
• External software for exclusive ATLAS needs: /afs/usatlas.bnl.gov/offline/external
• CERN software used by ATLAS: /afs/usatlas.bnl.gov/cernsw
• Atlas production releases installed from the distribution kits (maintained by OSG group): /usatlas/projects/OSG/atlas_app/atlas_rel

An ATLAS software release consists of an install area ( InstallArea ) and directories of packages. An install area contains references (soft links) to binaries, includes and share files dispersed in the release tree. ATLAS releases are managed by CMT release tool. The packages are organized according to CMT principles and can be simple packages (with no subpackages) or containers of simple packages and/or subcontainers. A CMT package consists of the version directory(ies) containing areas with sources, binaries, and cmt directory with CMT configuration (usually stored in single requirements file). Two remarks are worth to mention:

• The version directories are optional (but currently present in the standard ATLAS releases). If they are not present then the version information can be read from the cmt/version.cmt file
• Theoretically CMT allows multiple versions of one package in the release, however in ATLAS releases all packages have single version

Work model

CMT managerial principles and mechanisms are used in all stages of work (setting environment, compilation, linking, running) and they are evolving with ATLAS software. This means that the particular ATLAS releases must be handled with specific version of CMT. In addition the procedures for setting working environment vary for different releases. The helper scripts for automatic environment tuning have been developed for U.S. ATLAS users. The basic model of ATLAS software development include the following steps:

• Setting general environment that includes: CMT configuration variables describing the CMT version, settings for producing binary objects, and access paths to ATLAS software
• Creating work release with one or more ATLAS packages. This could be the code that a user is going to develop or packages that help to connect to the software release and acquire settings from the packages of this release. U.S. ATLAS users can conveniently check out ATLAS packages to their work areas from the local mirror of ATLAS CVS repository. Checkouts should be performed with cmt co command because it creates the correct directory hierarchy.
• Setting work environment is performed by sourcing setup.(c)sh script (generated by cmt co ) in cmt directory of a package. As well in this directory CMT generates Makefile, so the compilation and linking is performed with standard (g)make. Sourcing setup.(c)sh provides also a run time environment for jobs associated with the package.
• ATLAS OO framework, ATHENA, provides Python scripting service and it is recommended to use the athena.py script to drive ATHENA jobs. This script executes Python configuration scripts indicated in the command line and starts the athena application that loads components dynamically at run time. The Python configuration (or jobOptions) scripts store the specifications of the components (ATHENA algorithms and services) and job parameters. Their standard location in the ATLAS software release is the share directories of packages (which are also referenced in InstallArea). In conclusion, all ATLAS jobs (simulation, calibration, reconstruction...) are performed with standard script
  athena.py <package name/specific jobOption>
  in the run time environment installed by CMT mechanisms.

Relevant links

• U.S. ATLAS Physics, Software and Computing top page
• Getting Started page from ATLAS Offline Computing site at CERN
• The ATLAS Workbook - information for new users of ATLAS software on wiki
• Additional information for ATLAS software users (complementary to ATLAS Workbook). This page contains many useful recipes and workarounds.
• The Getting Started page of the U.S. ATLAS Tier I Computing Facility.
• Physics Analysis Tutorial with ATLAS software How To recipes
• More links from U.S. ATLAS Software Infrastructure

-- AlexUndrus - 21 Sep 2006 -- AlexUndrus - 28 Dec 2005