MinutesOct8

Introduction

• Previous meetings and background : IntegrationProgram
• Coordinates: Wednesdays, 1:00pm Eastern
  • new phone (309) 946-5300, Access code: 735188; Dial *6 to mute/un-mute.

Attending

• Meeting attendees: Michael, Rob, Charles, Rich, Patrick, Suvendra / Harvard, Bob, Shawn, Saul, Marco, John/BU, Sarah, Neng, Wen, Tom, Wensheng, Fred, Horst, Karthik, Fred, Hiro, Xin, Kaushik, Nurcan, Mark
• Apologies: none
• Guests: none

Integration program update (Rob, Michael)

• See IntegrationPhase6
• Quarterly report - this is needed for the agency.

Next procurements

• Follow-up from reported status last week:
  • AGLT2
    • PO's got re-quoted.
    • Hope for equipment.
  • SWT2
    • Still attempting to get a better deal from Dell.
    • OU: expecting EPSCOR funding. 105 TB quote. Will send the pricing matrix.
  • MWT2
    • Have POs in purchasing.
  • NET2
    • Final quotes - working its way through BU purchasing. 384 TB raw. $437/TB. DS3200.
  • WT2 - not buying this round.
  • Tier2: BNL is also speaking with Dell and DDN for storage procurement. DDN offering a 1 PB system for evaluation. 9900 series.

Operations overview: Production (Kaushik)

• Run out of jobs again. Borut contacted.
• New Condor-G monitoring proving effective.

Shifters (Marco)

• AGLT2 - switching to LFC
• Problems with Panda server - high load caused by running as root.
• Pilot wrapper version - fixed by Paul.

Cosmic data distribution (Kaushik)

• Waiting to see what comes to the US, then will provide subscriptions for the US.
• RAW data - has resumed, according to MOU share.
• Reprocessed beam data is available. ESD's went to 4 T2; the remainder went to all T2's (AOD, TAG, CBNT)

PRODDISK migration

• No issues at AGLT2
• At UC, made the transition; will wait for IU.
• SWT2 - how to best use xrootd's internal mover to avoid SRM for transfers to/from the SE from compute nodes. Prefer to do both for read/write.
• SLAC - reading you don't need a space tokens, from writing. Will try this week.
• BU - using Unix - Posix w/ gpfs.

Analysis queues, FDR analysis (Nurcan)

• Background, https://twiki.cern.ch/twiki/bin/view/Atlas/PathenaAnalysisQueuesUScloud
• No major updates this week. Working on distributed analysis shift organization. Next week Akira will be on shift.

Operations: DDM (Hiro)

• Question for NE - problems getting files back from Harvard. Saul is following this. Should be working now.
• All else is well basically.
• Kaushik raises an issue of missing 12.0.6 AOD files at BNL? Question is whether these were deleted by the facility - by whom? Entries from the LRC also disappeared. More investigation needed. Whats possible from dq2-client? dq2-delete-replicas?

LFC migration

• SubCommitteeLFC, see meeting notes LFCMeetOct8

RSV and WLCG SAM (Fred, Karthik)

• See https://twiki.grid.iu.edu/twiki/bin/view/Operations/RsvSAMGridView for links to SAM and Gridview reporting consoles.
• For scheduling downtimes, the OIM system: https://oim.grid.iu.edu/
• No news - working fine. Our numbers this month have looked good.
• What about the new yellow probes that indicate a certificate is expiring within a week.

xrootd, Bestman, etc in OSG

• Call from Alain
• UTA - on a test cluster, will install xrootd as part
• OU, BU - will look a the Bestman part
• SLAC - needs a test environment
• BNL - will help with the Fermilab setup. But also interested in exercises with VDT.
• UWISC? (Tier 3)
• Also would like to get a new site from Tier 3.
• OSG has really picked up the slack here, we need to take advantage of it. What obstacles do "young" sites run into?

Site news and issues (all sites)

• T1: 5 new grid ftp doors, and arrival of Cosmic data. Have observed > 500 MB/s over a couple of hours. Looked like a cap? Found that BNL was switched to a secondary link in LHC net. Now switched back to primary. Good rates now observed. Will be adding two 10G links from Esnet.
• AGLT2: in the middle of LFC migration. DQ2 data moving fine. Rearranged analysis queues 24 for instant, 140 for overflow.
• NET2: No big news - setting up the HU queue. Analysis queue working, will be sending in jobs. BNL firewall issues.
• MWT2: Conversion to PRODDISK at UC complete. A problem with the pnfs database backend - found and fixed on Tuesday.
• SWT2 (UTA): nothing new.
• SWT2 (OU): nothing new.
• WT2: conditions database issue - Rod provided a package that uses Frontier - a sort of squid. Working. Will look for a caching effect. Looks interesting. Will finish and then begin working on LFC. Michael: database task force performance meeting (Sasha V). Needs a launch pad - in front of conditions database.

Carryover issues (any updates?)

Release installation via Pacballs + DMM (Xin, Fred)

• Xin has tested scripts; passed to Tadashi to convert to Panda job.
• Fred - has requested that BNL receive pacball datasets, as a permanent request.
• Stan will coordinate pacball generation, and they should appear at BNL shortly thereafter.
• Need to setup account sm2 account.
• Job definitions interface.
• Xin will follow-up with Tadashi.
• 14.2.23 - just released - Fred will monitor the transfers to BNL.

Throughput initiative - status (Shawn)

• Meeting held this week. Jay put up an iperf server at BNL on a 10G.
• Asking sites to review site configuration and tunings, site by site.
• Perhaps tune the new nodes, given more memory.
• Perfsonar v2 is available. Most sites have their boxes installed and ready.
• Next step: configure a mesh scheduling to test among sites. Establish normalcy. Complementary to high throughput testing.
• Rich - will suggest working with John Bigaro at BNL to setup a mesh, and then work through the Tier 2s.

Tier3

• There is a separate subcommittee formed to redefine the whitepaper (Oct 1). Placeholder to follow developments.

Revised WLCG pledges

• Need the planned pledge amounts. This has been completed and sent .

AOB

• Welcome Suvendra from Harvard faculty and sciences. BU will be providing the first point of contact. Also Ben Smith - working for John.

-- RobertGardner - 07 Oct 2008