ATLAS Software Releases and How To Use Them

ATLAS software releases at BNL

ATLAS software releases are compiled on a Scientific Linux platform. Two compiler options are available:

• tag=dbg: debug, unoptimized
• tag=opt: optimized (-O2)

There are stable and nightly ATLAS software releases. Stable releases are created approximately every three weeks and reflect certain stages of ATLAS software development. The important releases used in data production are kept for a long time, others are deleted in several months. Before publication of release 13.0.0 the code was directly checked out from ATLAS CVS repository and compiled at BNL site. Releases 13.0.0 and higher as well as ATLAS nightly releases are installed from ATLAS distribution kits with Pacman tool. However, all the releases in OSG area at BNL are installed from kit distribution.

Tools and services for ATLAS releases

Code repository

U.S. ATLAS Tier I Center maintains the read-only mirror of ATLAS CVS repository at CVSROOT=/afs/usatlas.bnl.gov/software/cvs. Follow this link for details.

LXR

The LXR cross-referencing tool allows to jump easily to the declaration of any global identifier.

The package and project browser

This browser shows the packages versions of ATLAS releases and provides links to CVS, LXR, and Doxygen documentation

Tag Collector

Tag Collector is a database of packages versions with a powerful web interface

Work with ATLAS releases downloaded from kits

1. The ATLAS releases 12 and lower downloaded from kits are available for SLC3 only and are installed in /usatlas/OSG/atlas_app/atlas_rel (and also in /opt/usatlas/kit_rel/OSG_rel -- note that these are separate installations).
2. ATLAS releases 13.0.0 and higher are downloaded from kits and located in /afs/usatlas/software/downloads/releases (and a shorter symbolic link /opt/usatlas/kit_rel).
3. ATLAS nightly releases are available in /afs/usatlas/software/downloads/nightlies (bugfix and development branches).

They are self-sufficient and include all necessary setup scripts. See for details:

• Installing Atlas Software (on ATLAS Wiki)
• Beginning users should look at the Computing Workbook to get a basic idea on how to setup their account.

There are two different setups for the kit releases: one using the requirements coming with the kit, the other using user's private requirements which does not work for releases ≤11.2.0.

Recipe-1 of setup for kit releases (>11.2.0 only)

As stated in the Computing Workbook, you can use your own requirements file to setup the environment:

• Prepare your requirements files in $HOME/cmthome directory with the following basic content:

       #---------------------------------------------------------------------
       set CMTSITE STANDALONE
       macro PROJ_BASE_RELEASE    "latest" \
                  rel_0      "rel_0" \
                  rel_1      "rel_1" \
                  rel_2      "rel_2" \
                  rel_3      "rel_3" \
                  rel_4      "rel_4" \
                  rel_5      "rel_5" \
                  rel_6      "rel_6"
  
       macro PROJ_SUBDIR    "$(PROJ_BASE_RELEASE)" \
                bugfix      "bugfix/$(PROJ_BASE_RELEASE)" \
                dev         "dev/$(PROJ_BASE_RELEASE)"
  
       set SITEROOT /opt/usatlas/kit_rel/${PROJ_SUBDIR}
       macro ATLAS_DIST_AREA ${SITEROOT}
       macro ATLAS_TEST_AREA ${HOME}/testarea/${PROJ_BASE_RELEASE}
  
       # macro ATLAS_GROUP_AREA "/afs/usatlas/project/analysis/EventViewGroupArea/Numbered/13.0.40.1
       # apply_tag groupArea
  
       apply_tag oneTest 
       apply_tag setupCMT
       apply_tag setup
       apply_tag noCVSROOT
       use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)
  
       #---------------------------------------------------------------------
       

• If you are running pathena, you also need a line like set PATHENA_GRID_SETUP_SH /afs/usatlas/lcg/current/etc/profile.d/grid_env.sh
  For more details, go to pathena setup at BNL
• Go to ~/cmthome/ directory and run "cmt config" to create files such as setup.[c]sh under ~/cmthome/:

       cd $HOME/cmthome
       source /afs/usatlas/cernsw/contrib/CMT/v1r19/mgr/setup.[c]sh
       cmt config
       

Once you set up ~/cmthome/ as described in the above two steps, normally you need NOT do it again. However, you can later add additional CMT macros and patterns in the requirements file. Then for each release or a new terminal you execute the following (12.0.6, 13.010 and bugfix nightly rel_1 for examples):

1. cd ~/testarea/12.0.6
   or cd ~/testarea/13.0.10
   or cd ~/testarea/rel_1 for bugfix nightly rel_1.
2. source ~/cmthome/setup.[c]sh -tag=12.0.6,gcc323,slc4
   or source ~/cmthome/setup.[c]sh -tag=13.0.10
   or source ~/cmthome/setup.[c]sh -tag=bugfix,rel_1 for bugfix nightly rel_1.
3. Now you should be able to see an alias athena to athena.py .
4. If you need additional packages, then use "cmt co" to check them out, and/or compile them.

The tag "gcc323" and "slc4" are necessary for kit releases <13.0.0. Please note that the tag AtlasPoint1 is required for releases 13.0.25.* .

Recipe-2 of setup for kit releases (>11.2.0)

Instead of creating a private requirements file, you can use the requirements file coming with the kit as the following setup:

1. source /path_to_kit/cmtsite/setup.sh [or .csh] -tag=release_number[,projectName]
2. export CMTPATH=${working_directory}:${CMTPATH} for zsh/bash/ksh;
   setenv CMTPATH ${working_directory}:${CMTPATH} for tcsh
   • You can check out packages and compile them under ${working_directory}.
3. If you need add GroupArea, you can do:
   export CMTPATH=${ATLAS_GROUP_AREA}:${CMTPATH} for zsh/bash/ksh;
   setenv CMTPATH ${ATLAS_GROUP_AREA}:${CMTPATH} for tcsh
4. source /path_to_kit/AtlasProduction/release_number/AtlasProductionRunTime/cmt/setup.sh [or .csh]

The first command performs CMT, ATLAS release management tool, setup, where projectName is optional and it default project is AtlasOffline. The last command sets full run-time environment. The middle one is needed if you like to add your own package on top of the release. For release 13.0.10, path_to_kit is /afs/usatlas/software/downloads/releases/13.0.10 or /opt/usatlas/kit_rel/13.0.10.

If no additional package is required, the above setup can be simpler. That is, you can skip the middle step and combine the other two commands:

1. source /path_to_kit/cmtsite/setup.sh [or .csh] -tag=release_number,setup[,projectName]

Please note:

• The projectName must be specified as AtlasProduction for production patched releases,
  and as AtlasPoint1 for 13.0.25 patched releases (designed for M4 data) such as 13.0.25.4.
• Distribution software is under development and Kits are not created successfully for all nightlies
• Nightlies installation status is reflected at BNL nightlies global page. This "global" page shows the list of nightly jobs running at BNL. The jobs responsible for downloads of nightly releases have titles "bugfix mirror" and "dev mirror"
• Status of running reconstruction job with bugfix nightlies at BNL can be found at here.
• ATLAS software releases used in the data production are downloaded in /usatlas/OSG/atlas_app/atlas_rel (that is on NFS) by OSG group
• For more information about releases please refer to ATLAS Offline Release Page

Recipe-2 of setup for kit releases (≤11.2.0)

You cannot set up releases ≤11.2.0 with a private requirements file. You have to follow the setup below (11.0.5 for an example):

1. source /path_to_kit/setup.sh [or .csh]
2. export CMTPATH=${working_directory}:${CMTPATH} for zsh/bash/ksh;
   setenv CMTPATH ${working_directory}:${CMTPATH} for tcsh
3. source /opt/usatlas/kit_rel/11.0.5/dist/11.0.5/Control/AthenaRunTime/*/cmt/setup.sh
   If you want, then check out packages and compile them under ${working_directory}.

Work with ATLAS AFS releases (<13.0.0)

AFS releases are available at BNL for those preceding 13.0.0.

Location of AFS releases (<13.0.0)

Older ATLAS releases (before 13.0.0) was directly checked out from ATLAS CVS repository and compiled at the U.S. ATLAS Tier I Computing Facility locally. The software organization for these releases mirrors the CERN ATLAS site, however there are some differences summarized in the table below:

CERN Location	Analogous U.S. ATLAS Location
/afs/cern.ch/atlas/software/dist	/afs/usatlas/software/dist
/afs/cern.ch/atlas/software/builds	/afs/usatlas/software/builds
/afs/cern.ch/atlas/offline/external	/afs/usatlas/offline/external
/afs/cern.ch/sw/packages	/afs/usatlas/cernsw/packages
/afs/cern.ch/sw/contrib	/afs/usatlas/cernsw/contrib
/afs/cern.ch/sw/lcg/app	/afs/usatlas/cernsw/lcg/app

Very old "monolithic" ATLAS releases (≤11.2.0) are located in /afs/usatlas/software/dist. These obsolete releases do not have packages assigned to different software projects.

The"project oriented" ATLAS releases (≥11.3.0 and <13.0.0) are installed in /afs/usatlas/software/builds. The ATLAS software is split into 10 projects (AtlasCore, DetCommon, AtlasConditions, AtlasEvent, AtlasReconstruction, AtlasSimulation, AtlasTrigger, AtlasAnalysis, AtlasProduction, AtlasOffline) after release 11.2.0. AtlasProduction and AtlasOffline release numbers continue the "monolithic" sequence (11.3.0, 11.4.0 etc.) while the major versions of other projects are reset to "1". For example, AtlasOffline release 11.3.0 uses releases 1.3.0 of other projects. However, starting release 13.0.0 the rule is changed again and all projects get the same release numbers. For details refer to ATLAS projects wiki page.

Recipe-1 of setup for AFS releases (<13.0.0)

Beginning users should look at the Computing Workbook to get a basic idea on how to setup their account. A similar setup at BNL would be:

• Prepare your requirements files in $HOME/cmthome directory with the following basic content:

       #---------------------------------------------------------------------
       set CMTSITE BNL
       set SITEROOT /afs/usatlas
       macro ATLAS_DIST_AREA /afs/usatlas/software/dist
       macro ATLAS_TEST_AREA ${HOME}/testarea
       apply_tag setup
       apply_tag oneTest
       use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)
       #---------------------------------------------------------------------
       

• Go to ~/cmthome/ directory and run "cmt config" to create files such as setup.[c]sh under ~/cmthome/:

       cd $HOME/cmthome
       source /afs/usatlas/cernsw/contrib/CMT/v1r19/mgr/setup.[c]sh
       cmt config
       

Once you set up ~/cmthome/ as described in the above two steps, you need NOT do it again. However, you can later add additional CMT macros and patterns in the requirements file. Then for each release or a new terminal you execute the following (12.0.6 for an example):

1. cd ~/testarea/12.0.6
2. source ~/cmthome/setup.[c]sh -tag=12.0.6,gcc323 . Tag gcc323 is necessary to run on acasxxxx machines which are running SLC4.
3. Now you should be able to see an alias athena to athena.py .
4. If you need additional packages, then use "cmt co" to check them out, and/or compile them.

Recipe-2 of setup for AFS releases (<13.0.0)

Atlas Workbook provides the CMT setup recipes. In fact the CMT setup defines about 4 environmental variables but significant problems and work time loss can occur if these variables are not perfectly correct. Therefore the special script setup_cmt_usatlas.[c]sh is provided for the trouble free CMT setup for releases preceding 13.0.0 at BNL, which takes care of automatic setting of compiler gcc, CMT version, etc. The recipe is the following:

• optional step: create the login requirements file in your home directory. Alternatively, the requirements file can reside in the directory defined in variable REQUIREMENTS_DIR. Without the login requirements file the script setup_cmt_usatlas.[c]sh sets the standard CMT environment for BNL. A user can specify additional CMT macros and patterns in the login requirements file.
• create your working area and change (cd) to it.
• source /afs/usatlas/scripts/setup_cmt_usatlas.[c]sh  <base release name> <(--dbg or --opt)> <(--gcc323 or --gcc34)>
  • for project oriented releases the default project is AtlasOffline, other projects can be indicated with option -p:
    source /afs/usatlas/scripts/setup_cmt_usatlas.[c]sh -p <project> <release> <(--dbg or --opt)>
  • the base release name should be standard release name (e.g. 12.1.0), or "latest". The script tries to correct for the project release numbering convention. So
    1. source /afs/usatlas/scripts/setup_cmt_usatlas.[c]sh 2.0.3 --opt sets the environment for AtlasOffline 12.0.3 release
    2. source /afs/usatlas/scripts/setup_cmt_usatlas.[c]sh -p AtlasOffline 12.0.3 --opt - same as 1.
    3. source /afs/usatlas/scripts/setup_cmt_usatlas.[c]sh 11.0.3 --opt sets the environment for the monolithic 11.0.3 release
  • options --gcc323 and --gcc34 allow to choose the compiler on slc4 machines. The default is gcc323 for 12.2.0 release or older releases, otherwise gcc34 is chosen

The similar script, /afs/cern.ch/atlas/scripts/setup_cmt_cern.[c]sh, can be used for CMT setup at CERN (on lxplus machines).

Note about running ATLAS jobs on RHIC machines. ATLAS stable releases (installed on AFS) can be used on RHIC machines providing /opt/usatlas/lib is added to $LD_LIBRARY_PATH. This insures that libshift.so can be found when loading some libraries of CERN origin. The script /afs/usatlas/scripts/setup_cmt_usatlas.[c]sh does this addition automatically. If a different setup recipe is used then the following line in the login requirements file helps:

path_append LD_LIBRARY_PATH "/opt/usatlas/lib"

After the CMT setup, a user can proceed with the checkout of ATLAS packages (with cmt co command). The work environment for the package is installed with command source setup.[c]sh run from the cmt directory of a package. The subsequent build of the package (with gmake) establishes the InstallArea of the work release with binaries, include, and share files. Script setup.sh also sets the run time environment for athena jobs associated with the package providing that the requirements file is properly configured. In particular the package should have a dependency on an ATLAS release package with run time settings (such as Control/AthenaRunTime, AtlasCoreRunTime, AtlasConditionsRunTime, etc.) and declare its jobOptions files. The procedure for run-time environment settings continues to evolve. Please refer to ATLAS Software Development and ATLAS Wiki pages for the latest updates.

CVSROOT setting in kit setup

By default, after kit setup CVSROOT will be changed to ":gserver:isscvs.cern.ch:/local/reps/atlas".

There is a mirror of the ATLAS cvs repository available at BNL: "/afs/usatlas.bnl.gov/software/cvs". Two defined commands, bnl_cvs and bnl_cmt, in place of cvs and cmt respectively, can make use of this local mirror for cvs checkout.

Please be aware that the CVS repository at BNL is just a mirror of ATLAS CVS repository, and this mirror is available for checkouts only. The import and commit operations should be performed with CERN ATLAS CVS repository. Please visit here for details.

Relevant links

• ATLAS Offline Release Page
• The package and project browser shows the packages versions of ATLAS releases and provides links to CVS, LXR, and Doxygen documentation
• Beginniners should consult the Computing Workbook and the Physics Analysis Workbook for an introduction to ATLAS computing, physics topics, analysis recipes, etc. These two workbooks are actively maintained.
  Older guides/tutorials are available at,
  • Physics Analysis Tutorial with ATLAS software How To recipes
  • ASK tool tutorial with event generation, simulation, digitization and reconstruction recipes
• Additional information for ATLAS software users (complementary to ATLAS Workbooks). This page contains many useful recipes and workarounds.
• List of stable releases at U.S. ATLAS Tier I Center
• Status of nightlies downloads at BNL
• Status of nightlies at CERN
• Nightly Control System (NICOS) web page
• CMT documentation

-- AlexUndrus - 14 Feb 2007