MinutesJune4

Introduction

• Previous meetings and background : IntegrationProgram
• Coordinates: Wednesdays, 1:00pm Eastern
  • Phone: (605) 475-6000, Access code: 735188; Dial *6 to mute/un-mute.

Attending

• Meeting attendees: Michael, Charles, Rob, Shawn, Sarah, Rich, Justin, John, Horst, Karthik, Bob, Mark, Kaushik, Nurcan, Wei, Xin, Hiro, Fred, John Hover
• Apologies: none
• Guests: none

Integration program update (Rob, Michael)

• IntegrationProgram for Phase 5 (April 1 - June 30, 2008: FY08Q3)
  • AtlasReleases
  • DQ2SiteServices
  • OSGservices
  • StorageServices
  • MonitoringServices
  • FileCatalog
  • LoadTestsP5
  • SiteCertificationP5
• Overarching near term goals for Phase 5:
  • Full and effective participation FDR-2 exercises
  • Complete the benchmarks of 200 MB/s sustained disk-to-disk throughput to all Tier2s
  • SRM v2.2 functionality for all ATLAS sites
  • SAM availability reporting to WLCG (May)
• Upcoming meetings:
  • ATLAS workshop of the Americas in Vancouver, June 16-18
  • OSG Users Meeting at BNL, June 16-17
  • FDR2 Analysis Jamboree at BNL: June 24-27
  • FDR2 Analysis Jamboree at ANL: week of July 14
• Follow-up from Ann Arbor meeting: AnnArborNotesMay2008

Operations: Production (Kaushik)

• Out of jobs once again. However there is a new sample to fill the queues, but there problems with DQ2 callbacks. Expect scout jobs to finish quickly, then there would be 50K jobs.
• Overall, 5M single particle events - short jobs - being defined now.
• Unhappy about lack of planning and information dissemination.
• Validation samples keep coming in - low quantity, but important jobs.
• Re-processing: 3 issues discovered: Panda mover timeouts optimized for 2GB file jobs. dbrelease file is 4 GB - causing timeouts.. Tadashi increased. Will add modification to adjust to filesize. LCG-Utils was using lcg-ls (signed integer used, bug). Had max value of 3 GB. Newer release fixes the problem. Xin to check new OSG worker-node client LCG-Utils w/ Wensheng. Bamboo starvation and site specific scheduling.
• Rod asked to define a task for reprocessing at Tier2 sites.
• Michael suggests we summarize the experience and make available to ATLAS.
• What about Oracle database access? The BNL oracle instance will be used. What is the fall back? Use Triumf's instance. This is written into the transformation.
• Reprocessing job: 2 GB input file; 4.5 GB dbrelease file. Immediately after untarring.

Shifts (Mark)

• Welcoming Marco back into the mix - thanks.
• Not many jobs.. but expect to increase shortly.
• AGLT2 - small bug in site mover - Paul fixed.
• Autopilot ready for MWT2 - end of the week expect.
• 22441 - large number of failed jobs, don't worry.
• Heavy Ion jobs - 'looping job killed by pilot' - was this issue resolved? Kaushik believes HI tasks redefined by Pavel increased to 200K. For dedicated sites, keep queues with very long wall-time limits.

Analysis queues, FDR analysis (Nurcan)

• Expecting increase in activity w/ FDR2 data - w/ release 14.
• Have requested a validation data sample to be replicated. Done. Will do validation.
• Helping with tutorial at Vancouver workshop. Working with Akira.
• Will run DPD maker package.

Operations: DDM (Hiro)

• Monitors:
  • http://www.usatlas.bnl.gov/dq2/monitor
  • http://www.usatlas.bnl.gov/dq2/monitor/dq2Pings
• LRC modifications for user dataset deletion: needed for Hiro's dq2Pings.
  • See SiteCertificationP5
  • Follow-up:
    • BU, MWT2, AGLT ready to test
    • Will need the change the default - it has Hiro's DN as a special DN.
    • UTA - Patrick will test.
• CCRC data
  • All sites need to clean the files - see Hiro's instructions.
• FDR2 data - no subscriptions made yet. Not going smoothly. (Even for calibration data - long latencies.)

RSV probe updates

• Upgrade takes about 30 minutes. Sarah will circulate instructions.
• Fred: a couple of sites are not passing the CE monitoring. AGLT2 (failing one test) and BU_ATLAS_Tier2 (failing everything - will look into it).
• Tomasz is working on RSV-Nagios probe.

WLCG accounting

• http://t2.unl.edu/gratia/wlcg_reporting
• Action item - all sites need to re-run configure-osg with sub-cluster information
• Information is picked up overnight.

SRM v2 and Space Tokens

• Follow-up:
• OU - will not update to SRM v2.2 until new storage arrives.
• Which roles should space tokens support. Role usatlas production vs atlas production. Two roles? And the mappings are different. Is there only one binding between the attribute and the space token?
• Note - jobs are being defined w/ space tokens.
• Enable multiple roles in the certificate?
• Unify all production with the simple atlas production.

Pilot upgrade for space tokens

• AGLT2 has a site setup; SE - ATLASMCDISK, and an SE path. In contact with Paul. SE prod path.
• ATLASENDUSER disk also included. (At AGLT2, put 18 TB)

Unified LHC client (Marco)

• Available for testing, see: WlcgClient
• Issue that has come up is the new set of "dash".

LFC status (John)

• Almost up and running - off an Oracle cluster backend.
• Then Hiro will test an LRC migration script.
• Then will decide on an exact migration path. Lazy migration.
• Don't expect any issues with Panda integration.
• We need to comet to a formal decision about our own deployment model. If we stick to our existing model, we should prepare arguments, and the converse. Fault tolerance and scalability issues. Suggestion is to revisit.
• Revisit in 3 weeks

Next procurements

• Standing agenda item, see CapacitySummary.
• ATLAS meeting on benchmarks: http://indico.cern.ch/conferenceDisplay.py?confId=34293
• Looking at the pledges - we're short by about 20% in summary. But at specific sites there are severe shortcomings. And 2008-2009 we need to double 1.5 to 2.5 PB, a significant growth.
• 5 - 6.3 MSI2K? a minor step, while storage by factor of 2.
• Kaushik should give guidance about what production and analysis will need.
• Jim has given requirements as well. Will start from those numbers.
• Deployed by September 15
• Implies need to go out for bids in July.
• Action item next week for rough guidance.
• Want to be finished with this by end of June - technology and how much
• Need specifications from Internet2 for network monitoring hosts. Encourage Rich to get these specifications by next Rich.

OSG 1.0

• Expect release next week - some early testing at MWT2 in advance of release.

Throughput initiative - status (Shawn)

• Monalisa and configuration links:
  • http://gridtest01.racf.bnl.gov:8081/display?page=network - https://svn.usatlas.bnl.gov/svn/usatlas-int/load-test/Control/conf/network_m2m
  • http://gridtest01.racf.bnl.gov:8081/display?page=gridftp - https://svn.usatlas.bnl.gov/svn/usatlas-int/load-test/Control/conf/gridftp_d2d
• Met this past Monday - waiting on upgraded doors at BNL by end of the month. Expect 1 Gpbs to all Tier2s in parallel. 200/400 individually.
• Hiro's tests can be made on-demand.
• Jay: to include tests from multiple doors.
• Most of the old doors will be replaced.

Nagios monitoring subcommittee (Dantong)

• WT2 and SWT2 will be reporting available space.
• Tomasz organizing a meeting to test globus-job-run.

Release installation via Pacballs (Xin)

• Follow-up
• Progress - this morning to discuss this. Fred - hoping this week to have first set of pacballs installed in DQ2. Will test with some older releases on some test machines.
• Need official naming scheme.
• Get installed with a special Panda pilot job using the software role. Expect performance to improve.
• Expect a couple of weeks of testing.
• Goal to bring into production by end of the month.

Site news and issues (all sites)

• T1: lots of activities last week regarding FDR preparation, mixing jobs, and a group studying triggers. Busy deploying storage and network infrastructure (foundry core, 2 force 10s) for connection to 10G thumpers. Expect farm extension this week 3M SI2K? .
• AGLT2: Busy with FDR2 calibration work. NFS-lock problems with SQL-lite databases. Site issue? Need to follow-up with ATLAS on this file locking.
• NET2: All is well.
• MWT2: All is well.
• SWT2 (UTA): All is well.
• SWT2 (OU): Got replacement server for gatekeeper, to be installed. 10G switch to be connected still.
• WT2: All is well.

AOB

• Updated instructions for LRC data deletion.

-- RobertGardner - 27 May 2008

• Panglia for the week: