r3 - 28 Feb 2012 - 13:47:03 - KaushikDeYou are here: TWiki >  Admins Web > MinutesDataManageFeb28

MinutesDataManageFeb28

Introduction

Minutes of the US ATLAS Data Management meeting, February 28, 2012
  • Previous meetings and background : IntegrationProgram
  • Coordinates: Tuesdays, Noon Central
    • (605) 475-6150, Access code: 735188; Dial *6 to mute/un-mute.

Attending

  • Meeting attendees: Armen, Shawn, Rob, Saul, Dave, Hiro, Wei, Patrick, John
  • Apologies:
  • Guests:

Topics for this week

  • Storage status, central deletion - Armen
    • Overall OK. Keep eye on AGLT2 SCRATCHDISK. Shawn - will add more.
    • OU reporting still incorrect - Patrick will follow up.
    • Some consistency problems at WISC. Armen following up with them. They will remove PRODDISK & USERDISK immediately. DATADISK will be removed in a few months, after data migration.
    • Deletion service having problems with some files. Many of these seem fine in srm? Armen managed to delete them manually. Danila has no explanation.
      • Action item - choose one clear case where file is present but Victor cannot delete it. Ask Danila explicitly to debug this case.
    • In one case of deletion errors they have % sign. Wei - XRootD treats % as CGI string.
      • Action item - find pandaID of job which created this dataset.
  • USERDISK cleanup - Hiro
    • Will send new round of emails on March 1st.
  • Discussion of US storage policy - Armen

Policy for Space Token Implementation at USATLAS Sites
========================================================

DATADISK: if site has active central analysis queue, and is getting data by PD2P data distribution

USERDISK: if site has active central analysis queue

SCRATCHDISK: if site has active central analysis queue

PRODDISK: if site has active central production queue

GROUPDISK: size defined by ATLAS, space managed by physics/performance group disk managers 

LOCALGROUPDISK: unpledged resource, space managed by local group 

HOTDISK: for conditions data and DBReleases distribution (in the future this will no longer be needed if CVMFS is used for DB releases)


Space Token Size Recommendations at USATLAS Sites (in % of the total storage) 
===============================================================================

All US sites use flexible space token partitions in order to maximize use of storage (free space is allocated proportionately 
among many tokens, leaving some minimum free space as shown below).

USERDISK:  10% (cleaned by US facilities)

SCRATCHDISK:  1.5% (cleaned by ADC)

PRODDISK:  3% (needs regular cleaning by site)

GROUPDISK:  size defined by ATLAS (~20%); if free space is more than 150TB(T1)/100TB(T2), some of that free space 
can be temporarily used in DATADISK, as long as the free space in GROUPDISK is kept at minimum level of 150TB(T1)/100TB(T2)

LOCALGROUPDISK:  size defined by local group (should be ideally all space beyond pledge, minimum 50TB)

HOTDISK:  3TB

DATADISK:  remaining space, should be >50%   

  • Hot issues
    • Kaushik will follow up on the bad tasks with missing datasets.
  • AOB


-- KaushikDe - 28 Feb 2012

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


Attachments

 
Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback