r29 - 18 Aug 2011 - 12:04:03 - RobertGardnerYou are here: TWiki >  Admins Web > TestingCVMFS


Important Note

This page is now obsolete; the current notes on this topics an be found on the ATLAS CVMFS twiki page.


From software week and previous workshops, clear interest in CVMFS as a way of delivering software to facilities. In US ATLAS we've already had three generations of a software installation system, can this provide a more reliable and easier to manage system?


  • Install at the Tier 1 and one or more Tier 2s
  • Possibly evaluate within the ITB (reconfigure Panda queue to use new cache area rather than pre-installed releases)
  • Validate with test jobs
  • Stress testing
  • Evaluate and report performance relative to a conventionally installed release
  • Measure hit rate (perhaps via Squid)

Using CVMFS with Panda

  • The path to the releases must end in atlas_app/atlas_rel. If it does not, panda will append it. Therefore, you will need to create a symlink to the releases directory. At MWT2 we used /share/atlas_app/atlas_rel as a symlink to /cvmfs/atlas.cern.ch/software/manageTier3SW/ATLASLocalRootBase/Athena/i686_slc5_gcc43_opt. Then make the following changes to schedconfig:
    'appdir' : '/opt/atlas_app/atlas_rel'
    'environ' : 'VO_ATLAS_SW_DIR=/opt/atlas_app/atlas_rel APP=/osg/app TMP=/scratch DATA=/osg/data'

Installation notes for release 0.2.68-1

Note: these notes are cut-n-pasted from the 0.2.61-1 notes below and edited based upon the experience at AGLT2. Also the configuration details below are setup to use BNL's SQUID as the "CVMFS Server". For testing within USATLAS this should be fine. Sometime in May we should be able to update these configuration details to match the "production" configuration intended for ATLAS.

Before installing the cvmfs RPMs:

  • Make sure autofs is installed or do
    yum install autofs
  • Optional: create cvmfs user and fuse group accounts
  • Increase open file limits for the cvmfs user
    • Create /etc/security/limits.d/cvmfs.conf with the lines:
            cvmfs soft nofile 32768
            cvmfs hard nofile 32768
  • Install fuse rpm via yum:
    yum install fuse
  • Add this line to /etc/auto.master:
    /cvmfs /etc/auto.cvmfs
  • Add this line (if missing) to /etc/fuse.conf:
  • Set up the cvmfs repo:
    cd /etc/yum.repos.d/
    wget http://cvmrepo.web.cern.ch/cvmrepo/yum/cernvm.repo
    cd  /etc/pki/rpm-gpg/
    wget http://cvmrepo.web.cern.ch/cvmrepo/yum/RPM-GPG-KEY-CernVM 
  • Install via yum: yum install cvmfs cvmfs-init-scripts cvmfs-keys
  • Create a soft-link (needed till final production system is ready): ln -s /cvmfs/atlas.cern.ch /opt/atlas
  • /etc/init.d/autofs restart
  • If yum auto-updates are enabled, add cvmfs to the exclude list in yum.conf

Create Custom CVMFS Files

Next you need to create some cvmfs files


  • Create /etc/cvmfs/default.local with content (NOTE: EDIT the squid INFO for your site!!! The example below is for AGLT2 which has 3 real SQUIDs and a round-robin DNS entry):
  • NOTE: At AGLT2 we choose to host the cache under /tmp/cache/. Other sites may choose different locations. Make sure that the cvmfs user owns the directory: chown cvmfs.cvmfs /tmp/cache/cvmfs2. The CVMFS_CACHE_BASE variable controls the location. Set it appropriately for your site. Also, if you use /tmp, you may want to verify that any tools which automatically cleanup /tmp (like tmpwatch) are configured to leave the cache area alone.
  • NOTE2: A simpler CVMFS_HTTP_PROXY setting for a site with only 1 local SQUID would look like:
  • NOTE3: You can enable load balancing between Squid nodes by using a pipe ("|") separator, e.g.:


  • Create /etc/cvmfs/config.d/atlas.cern.ch.local with content:
  • NOTE: The above controls the amount of space (in MB) you want to allocate for atlas.cern.ch CVMFS use...recommendation is 25-50GB. This needs to be set even if using a stand alone
partition. Otherwise when the partition fills up cvmfs will not easily flush its cache automatically.


  • Create /etc/cvmfs/domain.d/cern.ch.local with content:
  • NOTE: The CVMFS_SERVER_URL controls the "source" location for CVMFS. The above configuration uses BNL's replica, followed by CERN's CVMFS server and RAL's replica as backups.
  • To test the "new" ATLAS namespace at BNL, in the BNL server definition in `CVMFS_SERVER_URL` above, change `@org@` to `atlas-newns`.

Run It

Now we are ready to try it:
  • Run:
    /etc/init.d/cvmfs start
    ls /cvmfs/atlas.cern.ch
  • Note that running ls /cvmfs/ does not work.
  • Verify that the cvmfs and autofs services are chkconfig'ed on
  • cvmfs_config showconfig
  • cvmfs_config chksetup

Documentation and other links:

Using the CVMFS Setup

See the following page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/ATLASLocalRootBase

At AGLT2 we created a simple script that users can source to allow access to this setup in CVMFS:

cat /usr/local/bin/setup/cvmfs_atlas.sh
# Setup to use ATLAS software from CVMFS
export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/software/manageTier3SW/ATLASLocalRootBase
source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
Or the csh variant:
cat /usr/local/bin/setup/cvmfs_atlas.csh
# Setup to use ATLAS software from CVMFS
setenv ATLAS_LOCAL_ROOT_BASE /cvmfs/atlas.cern.ch/software/manageTier3SW/ATLASLocalRootBase
source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.csh

Note: I am not sure this is how it is supposed to be used but I was able to get things to work using the above setup followed by asetup 16.6.4 and then running a full ATLAS chain from event generation to reconstruction.

Puppet Module

There is a puppet module from MWT2 which installs all the appropriate packages from the CERN repo, and applies the configuration from the above installation notes. This includes a manifests/init.pp as well as a few files and templates which are used by the main manifest. The only real changes needed to use this in production is to switch out the squid servers referenced in this module to use your own squid servers. This has been updated to reflect the change from using the BNL squid cache to using a local squid cache and then DIRECT.

Installation notes for release 0.2.61-1

Before installing the RPM:

  • Add this line to /etc/auto.master:
    /cvmfs /etc/auto.cvmfs
  • Add this line to /etc/fuse.conf ( create if neccessary):
  • Set up the cvmfs repo:
    cd /etc/yum.repos.d/
    wget http://cvmrepo.web.cern.ch/cvmrepo/yum/cernvm.repo
    cd  /etc/pki/rpm-gpg/
    wget http://cvmrepo.web.cern.ch/cvmrepo/yum/RPM-GPG-KEY-CernVM 
  • run yum install cvmfs cvmfs-init-scripts
  • /etc/init.d/autofs restart
  • Create /etc/cvmfs/default.local with content:
  • Run:
    /etc/init.d/cvmfs start
    ls /cvmfs/atlas.cern.ch
  • Note that running ls /cvmfs/ does not work.


  • cvmfs_config showconfig
  • cvmfs_config chksetup

Installation notes for release 0.2.55

  • Install using yum
    yum  install cvmfs.x86_64 cvmfs-init-scripts.noarch fuse.x86_64 autofs.x86_64
    mkdir /etc/cvmfs/local.d
    echo this into /etc/cvmfs/local.d/default.conf:

  • Note: CVMFS_HTTP_PROXY should be set to your own squid server if you have one.
  • Restart
    restart cvmfs /etc/init.d/cvmfs restart 
If you dont do this autofs may not start, the startup script ignores it so use restart.

  • To activate autofs mounts and see cvmfs working:
    [root@test local.d]# ls /opt/atlas
    ChangeLog  conditions  database  panda  software  tag  test

  • Inspect releases installed into cvmfs:
    root@test local.d]# ls /opt/atlas/software
    14.2.25  15.2.0  15.5.2  15.6.5               i686-slc4-gcc34-opt  sw
    14.5.0   15.3.0  15.5.3  15.6.7               i686_slc5_gcc43_opt  tag
    14.5.1   15.3.1  15.5.4  15.6.8               i686-slc5-gcc43-opt  tier3
    14.5.2   15.4.0  15.6.0  ChangeLog            manageTier3SW        vp1patches
    15.0.0   15.4.1  15.6.1  ChangeLog~           preloadcache
    15.0.1   15.5.0  15.6.3  gcc                  runVP1.sh
    15.1.0   15.5.1  15.6.4  i686_slc4_gcc34_opt  setupScripts

Planned Cernvmfs (CVMFS) directory structure

Notes from Doug Benjamin (15-Feb-2011).

Prior to the transition of the of cernvmfs servers to CERN-IT the planned configuration of the directories (currect as of 15-Feb-2011) will be:


After the switch over:


For the sake of argument let us define another environment variable:

VO_ATLAS_CVMFS_DIR="/opt/atlas/repo"    - Initially

VO_ATLAS_CVMFS_DIR="/cvmfs/atlas.cern.ch/repo"  - after switch to CERN-IT

Athena software will be installed in: $VO_ATLAS_SW_DIR/software/.

As of Today (15-Feb-2011) for extra software we would have:


Note - other software can be added as needed.

$VO_ATLAS_CVMFS_DIR/conditions (for the conditions db files) 
$VO_ATLAS_SW_DIR/database/ for DB Releases     
$VO_ATLAS_SW_DIR/nightlies  for the Nightly releases 
Note- we have not stable configuration for the nightlies yet.

Initially the directories would be:


The main repository is: $VO_ATLAS_CVMFS_DIR. The nested repositories are:



-- RobertGardner - 03 Aug 2010

  • cvmfs.tar.bz2: Small change to init.pp proxy settings to reflect updated instructions.

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


bz2 cvmfs.tar.bz2 (2.2K) | JohnDeStefano, 13 May 2011 - 10:30 | More small changes to init.pp proxy settings to reflect further updated instructions.
Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback