r2 - 08 Oct 2008 - 14:49:00 - RobertGardnerYou are here: TWiki >  Admins Web > MinutesOct8



Minutes of the Facilities Integration Program meeting, Oct 8, 2008
  • Previous meetings and background : IntegrationProgram
  • Coordinates: Wednesdays, 1:00pm Eastern
    • new phone (309) 946-5300, Access code: 735188; Dial *6 to mute/un-mute.


  • Meeting attendees: Michael, Rob, Charles, Rich, Patrick, Suvendra / Harvard, Bob, Shawn, Saul, Marco, John/BU, Sarah, Neng, Wen, Tom, Wensheng, Fred, Horst, Karthik, Fred, Hiro, Xin, Kaushik, Nurcan, Mark
  • Apologies: none
  • Guests: none

Integration program update (Rob, Michael)

Next procurements

  • Follow-up from reported status last week:
    • AGLT2
      • PO's got re-quoted.
      • Hope for equipment.
    • SWT2
      • Still attempting to get a better deal from Dell.
      • OU: expecting EPSCOR funding. 105 TB quote. Will send the pricing matrix.
    • MWT2
      • Have POs in purchasing.
    • NET2
      • Final quotes - working its way through BU purchasing. 384 TB raw. $437/TB. DS3200.
    • WT2 - not buying this round.
    • Tier2: BNL is also speaking with Dell and DDN for storage procurement. DDN offering a 1 PB system for evaluation. 9900 series.

Operations overview: Production (Kaushik)

  • Run out of jobs again. Borut contacted.
  • New Condor-G monitoring proving effective.

Shifters (Marco)

  • AGLT2 - switching to LFC
  • Problems with Panda server - high load caused by running as root.
  • Pilot wrapper version - fixed by Paul.

Cosmic data distribution (Kaushik)

  • Waiting to see what comes to the US, then will provide subscriptions for the US.
  • RAW data - has resumed, according to MOU share.
  • Reprocessed beam data is available. ESD's went to 4 T2; the remainder went to all T2's (AOD, TAG, CBNT)

PRODDISK migration

  • No issues at AGLT2
  • At UC, made the transition; will wait for IU.
  • SWT2 - how to best use xrootd's internal mover to avoid SRM for transfers to/from the SE from compute nodes. Prefer to do both for read/write.
  • SLAC - reading you don't need a space tokens, from writing. Will try this week.
  • BU - using Unix - Posix w/ gpfs.

Analysis queues, FDR analysis (Nurcan)

Operations: DDM (Hiro)

  • Question for NE - problems getting files back from Harvard. Saul is following this. Should be working now.
  • All else is well basically.
  • Kaushik raises an issue of missing 12.0.6 AOD files at BNL? Question is whether these were deleted by the facility - by whom? Entries from the LRC also disappeared. More investigation needed. Whats possible from dq2-client? dq2-delete-replicas?

LFC migration

RSV and WLCG SAM (Fred, Karthik)

xrootd, Bestman, etc in OSG

  • Call from Alain
  • UTA - on a test cluster, will install xrootd as part
  • OU, BU - will look a the Bestman part
  • SLAC - needs a test environment
  • BNL - will help with the Fermilab setup. But also interested in exercises with VDT.
  • UWISC? (Tier 3)
  • Also would like to get a new site from Tier 3.
  • OSG has really picked up the slack here, we need to take advantage of it. What obstacles do "young" sites run into?

Site news and issues (all sites)

  • T1: 5 new grid ftp doors, and arrival of Cosmic data. Have observed > 500 MB/s over a couple of hours. Looked like a cap? Found that BNL was switched to a secondary link in LHC net. Now switched back to primary. Good rates now observed. Will be adding two 10G links from Esnet.
  • AGLT2: in the middle of LFC migration. DQ2 data moving fine. Rearranged analysis queues 24 for instant, 140 for overflow.
  • NET2: No big news - setting up the HU queue. Analysis queue working, will be sending in jobs. BNL firewall issues.
  • MWT2: Conversion to PRODDISK at UC complete. A problem with the pnfs database backend - found and fixed on Tuesday.
  • SWT2 (UTA): nothing new.
  • SWT2 (OU): nothing new.
  • WT2: conditions database issue - Rod provided a package that uses Frontier - a sort of squid. Working. Will look for a caching effect. Looks interesting. Will finish and then begin working on LFC. Michael: database task force performance meeting (Sasha V). Needs a launch pad - in front of conditions database.

Carryover issues (any updates?)

Release installation via Pacballs + DMM (Xin, Fred)

  • Xin has tested scripts; passed to Tadashi to convert to Panda job.
  • Fred - has requested that BNL receive pacball datasets, as a permanent request.
  • Stan will coordinate pacball generation, and they should appear at BNL shortly thereafter.
  • Need to setup account sm2 account.
  • Job definitions interface.
  • Xin will follow-up with Tadashi.
  • 14.2.23 - just released - Fred will monitor the transfers to BNL.

Throughput initiative - status (Shawn)

  • Meeting held this week. Jay put up an iperf server at BNL on a 10G.
  • Asking sites to review site configuration and tunings, site by site.
  • Perhaps tune the new nodes, given more memory.
  • Perfsonar v2 is available. Most sites have their boxes installed and ready.
  • Next step: configure a mesh scheduling to test among sites. Establish normalcy. Complementary to high throughput testing.
  • Rich - will suggest working with John Bigaro at BNL to setup a mesh, and then work through the Tier 2s.


  • There is a separate subcommittee formed to redefine the whitepaper (Oct 1). Placeholder to follow developments.

Revised WLCG pledges

  • Need the planned pledge amounts. This has been completed and sent DONE.


  • Welcome Suvendra from Harvard faculty and sciences. BU will be providing the first point of contact. Also Ben Smith - working for John.

-- RobertGardner - 07 Oct 2008

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback