TestingCVMFS

Important Note

This page is now obsolete; the current notes on this topics an be found on the ATLAS CVMFS twiki page.

Background

From software week and previous workshops, clear interest in CVMFS as a way of delivering software to facilities. In US ATLAS we've already had three generations of a software installation system, can this provide a more reliable and easier to manage system?

• Deployments: SiteCertificationP16
• CVMFS support: cvmfs-talk@cern.ch

Goals:

• Install at the Tier 1 and one or more Tier 2s
• Possibly evaluate within the ITB (reconfigure Panda queue to use new cache area rather than pre-installed releases)
• Validate with test jobs
• Stress testing
• Evaluate and report performance relative to a conventionally installed release
• Measure hit rate (perhaps via Squid)

Using CVMFS with Panda

• The path to the releases must end in atlas_app/atlas_rel. If it does not, panda will append it. Therefore, you will need to create a symlink to the releases directory. At MWT2 we used /share/atlas_app/atlas_rel as a symlink to /cvmfs/atlas.cern.ch/software/manageTier3SW/ATLASLocalRootBase/Athena/i686_slc5_gcc43_opt. Then make the following changes to schedconfig:

  'appdir' : '/opt/atlas_app/atlas_rel'
  'environ' : 'VO_ATLAS_SW_DIR=/opt/atlas_app/atlas_rel APP=/osg/app TMP=/scratch DATA=/osg/data'

Installation notes for release 0.2.68-1

Note: these notes are cut-n-pasted from the 0.2.61-1 notes below and edited based upon the experience at AGLT2. Also the configuration details below are setup to use BNL's SQUID as the "CVMFS Server". For testing within USATLAS this should be fine. Sometime in May we should be able to update these configuration details to match the "production" configuration intended for ATLAS.

Before installing the cvmfs RPMs:

• Make sure autofs is installed or do

  yum install autofs

• Optional: create cvmfs user and fuse group accounts
• Increase open file limits for the cvmfs user
  • Create /etc/security/limits.d/cvmfs.conf with the lines:

            cvmfs soft nofile 32768
            cvmfs hard nofile 32768

• Install fuse rpm via yum:

  yum install fuse

• Add this line to /etc/auto.master:

  /cvmfs /etc/auto.cvmfs

• Add this line (if missing) to /etc/fuse.conf:

  user_allow_other

• Set up the cvmfs repo:

  cd /etc/yum.repos.d/
  wget http://cvmrepo.web.cern.ch/cvmrepo/yum/cernvm.repo
  
  cd  /etc/pki/rpm-gpg/
  wget http://cvmrepo.web.cern.ch/cvmrepo/yum/RPM-GPG-KEY-CernVM 

• Install via yum: yum install cvmfs cvmfs-init-scripts cvmfs-keys
• Create a soft-link (needed till final production system is ready): ln -s /cvmfs/atlas.cern.ch /opt/atlas
• /etc/init.d/autofs restart
• If yum auto-updates are enabled, add cvmfs to the exclude list in yum.conf

Create Custom CVMFS Files

Next you need to create some cvmfs files

/etc/cvmfs/default.local

• Create /etc/cvmfs/default.local with content (NOTE: EDIT the squid INFO for your site!!! The example below is for AGLT2 which has 3 real SQUIDs and a round-robin DNS entry):

CVMFS_REPOSITORIES=atlas
CVMFS_CACHE_BASE=/tmp/cache/cvmfs2
CVMFS_HTTP_PROXY="http://<your_squid_roundrobin>:3128;http://<real_squid1>:3128;http://<real_squid2>:3128;http://<real_squid3>:3128;DIRECT"

• NOTE: At AGLT2 we choose to host the cache under /tmp/cache/. Other sites may choose different locations. Make sure that the cvmfs user owns the directory: chown cvmfs.cvmfs /tmp/cache/cvmfs2. The CVMFS_CACHE_BASE variable controls the location. Set it appropriately for your site. Also, if you use /tmp, you may want to verify that any tools which automatically cleanup /tmp (like tmpwatch) are configured to leave the cache area alone.

• NOTE2: A simpler CVMFS_HTTP_PROXY setting for a site with only 1 local SQUID would look like:

CVMFS_HTTP_PROXY="http://<your_squid_dns>:3128;DIRECT"

• NOTE3: You can enable load balancing between Squid nodes by using a pipe ("|") separator, e.g.:

CVMFS_HTTP_PROXY="http://<your_squid_1>:3128|http://<your_squid_2>:3128|http://<your_squid_3>:3128;DIRECT"

/etc/cvmfs/config.d/atlas.cern.ch.local

• Create /etc/cvmfs/config.d/atlas.cern.ch.local with content:

CVMFS_QUOTA_LIMIT=25000

• NOTE: The above controls the amount of space (in MB) you want to allocate for atlas.cern.ch CVMFS use...recommendation is 25-50GB. This needs to be set even if using a stand alone

partition. Otherwise when the partition fills up cvmfs will not easily flush its cache automatically.

/etc/cvmfs/domain.d/cern.ch.local

• Create /etc/cvmfs/domain.d/cern.ch.local with content:

CVMFS_SERVER_URL="http://cvmfs.racf.bnl.gov:8000/opt/@org@;http://cvmfs-stratum-one.cern.ch:8000/opt/@org@;http://cernvmfs.gridpp.rl.ac.uk:8000/opt/@org@"

• NOTE: The CVMFS_SERVER_URL controls the "source" location for CVMFS. The above configuration uses BNL's replica, followed by CERN's CVMFS server and RAL's replica as backups.
• To test the "new" ATLAS namespace at BNL, in the BNL server definition in `CVMFS_SERVER_URL` above, change `@org@` to `atlas-newns`.

Run It

Now we are ready to try it:

• Run:

  /etc/init.d/cvmfs start
  ls /cvmfs/atlas.cern.ch

• Note that running ls /cvmfs/ does not work.
• Verify that the cvmfs and autofs services are chkconfig'ed on

Debugging

• cvmfs_config showconfig
• cvmfs_config chksetup

Documentation and other links:

• See 14:00-15:30 session on CVMFS at https://indico.cern.ch/conferenceDisplay.py?confId=119169 (lots of talks there).
• Specific talk at https://indico.cern.ch/getFile.py/access?contribId=17&sessionId=14&resId=0&materialId=slides&confId=119169
• Some CVMFS test results at Tier 1 BNL

Using the CVMFS Setup

See the following page: https://twiki.atlas-canada.ca/bin/view/AtlasCanada/ATLASLocalRootBase

At AGLT2 we created a simple script that users can source to allow access to this setup in CVMFS:

cat /usr/local/bin/setup/cvmfs_atlas.sh
#!/bin/bash
#
# Setup to use ATLAS software from CVMFS
#
export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/software/manageTier3SW/ATLASLocalRootBase
source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh

Or the csh variant:

cat /usr/local/bin/setup/cvmfs_atlas.csh
#!/bin/csh
#
# Setup to use ATLAS software from CVMFS
#
setenv ATLAS_LOCAL_ROOT_BASE /cvmfs/atlas.cern.ch/software/manageTier3SW/ATLASLocalRootBase
source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.csh

Note: I am not sure this is how it is supposed to be used but I was able to get things to work using the above setup followed by asetup 16.6.4 and then running a full ATLAS chain from event generation to reconstruction.

Puppet Module

There is a puppet module from MWT2 which installs all the appropriate packages from the CERN repo, and applies the configuration from the above installation notes. This includes a manifests/init.pp as well as a few files and templates which are used by the main manifest. The only real changes needed to use this in production is to switch out the squid servers referenced in this module to use your own squid servers. This has been updated to reflect the change from using the BNL squid cache to using a local squid cache and then DIRECT.

• cvmfs.tar.bz2: Puppet Module for CVMFS 0.2.68

Installation notes for release 0.2.61-1

Before installing the RPM:

• Add this line to /etc/auto.master:

  /cvmfs /etc/auto.cvmfs

• Add this line to /etc/fuse.conf ( create if neccessary):

  user_allow_other

• Set up the cvmfs repo:

  cd /etc/yum.repos.d/
  wget http://cvmrepo.web.cern.ch/cvmrepo/yum/cernvm.repo
  
  cd  /etc/pki/rpm-gpg/
  wget http://cvmrepo.web.cern.ch/cvmrepo/yum/RPM-GPG-KEY-CernVM 

• run yum install cvmfs cvmfs-init-scripts
• /etc/init.d/autofs restart
• Create /etc/cvmfs/default.local with content:

  CVMFS_REPOSITORIES=atlas
  CVMFS_CACHE_BASE=/var/cache/cvmfs2
  CVMFS_HTTP_PROXY="http://iut2-grid1.iu.edu:3128;http://cernvm.lbl.gov:3128"

• Run:

  /etc/init.d/cvmfs start
  ls /cvmfs/atlas.cern.ch

• Note that running ls /cvmfs/ does not work.

Debugging

• cvmfs_config showconfig
• cvmfs_config chksetup

Installation notes for release 0.2.55

• Install using yum

  yum  install cvmfs.x86_64 cvmfs-init-scripts.noarch fuse.x86_64 autofs.x86_64
  mkdir /etc/cvmfs/local.d
  echo this into /etc/cvmfs/local.d/default.conf:
  CVMFS_USER=cvmfs
  CVMFS_NFILES=32768
  CVMFS_QUOTA_LIMIT=-1
  CVMFS_SERVER_URL=http://cernvm-webfs.cern.ch/opt/@org@
  CVMFS_OPTIONS=allow_other,remount_sleep=10,entry_timeout=10,attr_timeout=10,timeout=10,negative_timeout=10
  CVMFS_REPOSITORIES=atlas
  CVMFS_CACHE_DIR=/var/cache/cvmfs/atlas
  CVMFS_HTTP_PROXY="http://cernvm.lbl.gov:3128"
  CVMFS_DEBUGLOG=/var/log/cvmfs.log
  

• Note: CVMFS_HTTP_PROXY should be set to your own squid server if you have one.
• Restart

  restart cvmfs /etc/init.d/cvmfs restart 
  

If you dont do this autofs may not start, the startup script ignores it so use restart.

• To activate autofs mounts and see cvmfs working:

  [root@test local.d]# ls /opt/atlas
  ChangeLog  conditions  database  panda  software  tag  test
  

• Inspect releases installed into cvmfs:

  root@test local.d]# ls /opt/atlas/software
  14.2.25  15.2.0  15.5.2  15.6.5               i686-slc4-gcc34-opt  sw
  14.5.0   15.3.0  15.5.3  15.6.7               i686_slc5_gcc43_opt  tag
  14.5.1   15.3.1  15.5.4  15.6.8               i686-slc5-gcc43-opt  tier3
  14.5.2   15.4.0  15.6.0  ChangeLog            manageTier3SW        vp1patches
  15.0.0   15.4.1  15.6.1  ChangeLog~           preloadcache
  15.0.1   15.5.0  15.6.3  gcc                  runVP1.sh
  15.1.0   15.5.1  15.6.4  i686_slc4_gcc34_opt  setupScripts
  

Planned Cernvmfs (CVMFS) directory structure

Notes from Doug Benjamin (15-Feb-2011).

Prior to the transition of the of cernvmfs servers to CERN-IT the planned configuration of the directories (currect as of 15-Feb-2011) will be:

VO_ATLAS_SW_DIR="/opt/atlas/repo/sw" 

After the switch over:

VO_ATLAS_SW_DIR=/cvmfs/atlas.cern.ch/repo/sw" 

For the sake of argument let us define another environment variable:

VO_ATLAS_CVMFS_DIR="/opt/atlas/repo"    - Initially

VO_ATLAS_CVMFS_DIR="/cvmfs/atlas.cern.ch/repo"  - after switch to CERN-IT

Athena software will be installed in: $VO_ATLAS_SW_DIR/software/.

As of Today (15-Feb-2011) for extra software we would have:

 $VO_ATLAS_SW_DIR/cctools 
 $VO_ATLAS_SW_DIR/ddm 

Note - other software can be added as needed.

$VO_ATLAS_CVMFS_DIR/ATLASLocalRootBase 
$VO_ATLAS_CVMFS_DIR/conditions (for the conditions db files) 
$VO_ATLAS_SW_DIR/database/ for DB Releases     
$VO_ATLAS_SW_DIR/nightlies  for the Nightly releases 
Note- we have not stable configuration for the nightlies yet.

Initially the directories would be:

/opt/atlas/repo/ATLASLocalRootBase
/opt/atlas/repo/conditions
/opt/atlas/repo/sw/database/
/opt/atlas/repo/sw/software/
/opt/atlas/repo/sw/cctools
/opt/atlas/repo/sw/ddm
/opt/atlas/repo/sw/nightlies

The main repository is: $VO_ATLAS_CVMFS_DIR. The nested repositories are:

$VO_ATLAS_CVMFS_DIR/ATLASLocalRootBase 
$VO_ATLAS_CVMFS_DIR/conditions
$VO_ATLAS_SW_DIR/database/
$VO_ATLAS_SW_DIR/software/
$VO_ATLAS_SW_DIR/nightlies

References

• The basic install instructions from Doug for Tier 3's are here: https://twiki.cern.ch/twiki/bin/view/Atlas/Tier3CVMFS2SLC5
• CVMFS session at April software week, see https://indico.cern.ch/conferenceDisplay.py?confId=119169#2011-04-06
• ATLAS CVMFS twiki page

-- RobertGardner - 03 Aug 2010

• cvmfs.tar.bz2: Small change to init.pp proxy settings to reflect updated instructions.