StorageValidation

Introduction

Consistency checking

Atlas-wide tools

• https://www.usatlas.bnl.gov/twiki/bin/view/Admins/AtlasLFCUtilities
  • integrity_check.py: can check a single dataset

US-developed tools

• "Complete Consistency Check" - ccc.py - more thorough checking of all datasets
• Compares DQ2 catalogs, LFC entries, PNFS namespace, and contents of dCache pools.
• Does not make any changes to filesystem or databases, but prints a list of "ghosts" and "orphans".
• Additional scripts exist for dealing with ghost/orphan files.
• Link for ccc.py: http://repo.mwt2.org/viewvc/admin-scripts/lfc/ccc.py?view=markup
• Sample output: http://www.mwt2.org/sys/ccc
• To be done:
  • Automated running via cron or similar
  • HTML output
  • version for non-dCache sites:
    • ccc.py reads output of pnfsDump, so all that is needed is a way to dump the storage namespace to a text file.

Site cleanup:

US-developed tools

• Updated version of cleanse.py which uses LFC instead of LRC, renamed dq2site-cleanse.py
• Has been modified to also clean up panda mover datasets which are not registered as DQ2 replicas.
• dq2site-cleanse.py at: http://repo.mwt2.org/viewvc/admin-scripts/lfc/dq2site-cleanse.py?view=markup
• Some stability problems due to segfaults in LFC module - varies by site.
• To be done:
  • diagnose/fix LFC stability problems
  • pure-Python version that does not use _lfc.so (access LFC via SQL).

-- RobertGardner - 14 Jan 2009 -- CharlesWaldman - 14 Jan 2009