MinutesDataManageApr14

Introduction

• Previous meetings and background : IntegrationProgram
• Coordinates: Tuesdays, Noon Central
  • (309) 946-5300, Access code: 735188; Dial *6 to mute/un-mute.

Attending

• Meeting attendees: Charles, Shawn, Patrick, Rob, Saul, Bob, Hiro, Wensheng, Pedro, Armen, Jim S., Torre, Jose, Alexei, Wei
• Apologies:John
• Guests: None

Topics for this week

• Reprocessing status - Kaushik
  • Going at average rate of 6k jobs finished per day
  • Still 30k jobs left to do
• Storage cleanup - All
  • All Tier 2's low on space, Tier 1 cleanup also needed
  • Stephane started central cleanup of aborted tasks at SWT2 and SLACT2
    • Seeing high rate of LFC dying at SWT2 since central deletion started (half dozen time in 3-4 days)
    • Seen ~4TB cleaned up, no LFC crashes, but problem with Bestman crashes (may be unrelated - probably firewall related)
    • Many LFC entries deleted, but files not deleted, but file deletion is a delayed process by 1-2 days (will provide statistics next week of how many files left behind on storage after 1 week)
  • Problem noticed by Patrick - central deletion leaves metadata in LFC
  • Will try other 3 Tier 2's, and then BNL next
  • After that, will delete 'obsolete' datasets too
  • Need plan to cleanup old deleted datasets http://panda.cern.ch:25880/server/pandamon/query/?mode=listAbortedDatasetsTXT
    • Charles and Hiro will work on a script
  • ESD deletion - check with users, reduce number of copies (Armen)
• DQ2 adler32 plugin - Hiro
  • Passive mode in both site services at BNL
  • Flagged problem with some files due to network card
  • Look into making plugin active
  • NET2, AGLT2, SLACT2 are also interested in trying this
  • Wei is following up with Simone about FTS checksum validation
• SCRATCHDISK deployment
  • BNL - done, AGLT2 - 9TB (45TB total, done, see http://head02.aglt2.org/dcache_pool.html or http://gate02.grid.umich.edu/aglt2se_tot.shtml; Need ToA entry added), MWT2 - 10 TB (created space already, create token in next few days), NET2 - 10 TB (by next week), SLACT2, SWT2 - later, after cleanup
  • What should the protection settings be for this space? (Who can write?) - consensus is usatlas1.
  • AGLT2 has setup space-tokens setup for the new SCRATCHDISK area for '/atas/Role=production', '/atlas/usatlas/Role=production', 'usatlas3','usatlas4'. See:

• Hot issues
  • "File exists" problem. See https://rt-racf.bnl.gov/rt/Ticket/Display.html?id=12638. Almost all AGLT2 SRM failures are just "file exists" errors. Where are these coming from? On MCDISK and DATADISK - so they are from DQ2 SS. Could be due to dq2_recreate after SS restart.
• AOB
  • Major failure at BNL last week, some corrupted files from that, Hiro cleaning up
  • AGLT2 seeing files coming from TRIUMF, FZK. LYON, INFN, NDGF... to MCDISK (follow up with Alexei, Hiro)

-- KaushikDe - 14 Apr 2009