r4 - 05 Dec 2013 - 12:14:24 - IlijaVukoticYou are here: TWiki >  Admins Web > RucioRename

RucioRename


There are two methods that can be used to complete this task. This is part of SiteCertificationP27 of the IntegrationProgram.

Method 1: Script from Hiro

Setup

Execute

Here is an example:

Support


Method 2: Script from Ilija

Introduction

This script can be used to rename ATLAS data to Rucio paths. Requirements:

  • The user must have full rights to update LFC entries.
  • You need to do:
      export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
      source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
      localSetupDQ2Client
    
  • You need a valid grid proxy with role=production
  • Download and run the program: RucioRename.py

The Program

You need to give it a file listing all of datasets that should exist at a space token. You can obtain lists in this way:

dq2-list-dataset-site2 -N 01/01/2012 -D 01/01/2013 MWT2_UC_LOCALGROUPDISK > MWT2_UC_LOCALGROUPDISKfrom01_01_2012untill01_01_2013.dss ) 

Description:

  • The program will get list of files for each dataset.
  • For each file it will contact LFC to get list of replicas of that file.
  • The replicas corresponding to the site will be checked for existence.
  • The replica that does not exist will be removed from the LFC.
  • If the replica exists and file is already in Rucio format, it will do nothing.
  • If the replica is not in the Rucio file format, it will be renamed and the new replica registered in LFC and old record deleted.
  • An SQLite database is created having the same name as an input file but with added extension ".db" In the file, there is a table containing:
    dataset name | total files | files missing | files in Old gLFN | files in New gLNF | files renamed | replicas added| replicas deleted 
  • Ideally the last three figures should be the same, and files missing to be zero. If not, those datasets should be checked for consistency.

Setup

  1. Set surlBase, port, site_prefix, storage_token
  2. if your site does not support POSIX, you will have to change these commands: os.path.exists(pfn), os.path.exists(destDir), os.makedirs(destDir), and os.rename(pfn, destPfn) to your site's equivalents.

Execute

Here is an example:

python RucioRename.py MWT2_UC_LOCALGROUPDISKfrom01_01_2012untill01_01_2013.dss

Support


-- RobertGardner - 25 Nov 2013

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


Attachments


txt RucioRename.py.txt (5.4K) | RobertGardner, 25 Nov 2013 - 11:14 |
 
Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback