r3 - 27 Jul 2010 - 11:58:59 - RichardMountYou are here: TWiki >  AtlasSoftware Web > Minutes9Jul2010

Minutes9Jul2010 - RAC core team minutes, July 9, 2010

Core Team Participation

Richard Mount, Kaushik De, Michael Ernst, Jim Cochran

Report from the CREM (Richard)

The agenda item for the CREM meeting was "Dynamic Data Distribution". As Hans had feared, many of the key people were at the WLCG workshop in London and did not join the meeting, so the agenda item was largely downgraded to discussing the opinions of people who were not present. Overall there seemed to be enthusiasm for expanding the PD2P experiment and for a careful relaxation of the computing model to blur cloud boundaries.

Operations Issues (Kaushik)

A relatively quiet week. Since the implementation of PD2P, Tier 2 storage is now filling up much more slowly. As of Tuesday, PD2P had issued around 900 subscriptions, but only 4 or 5 were to multiple sites. The hottest dataset was a data ESD - 3000 files, 130k accesses. It is still very early to try to read too much into the statistics.

The planned reduction of AOD distribution was not yet in effect.

Graeme at CERN who gave a talk at the London Workshop found that 500 out of 900 were only accessed once, while 40 out of the 900 had been accessed more than 1000 times. Graeme's study is consistent with Kaushik's observations. There are very few "hot' datasets. This is good and bad. It is more efficient than the old distribution, but raises the question "why replicate things used only once?" We could consider delaying replication until more than one person wants the data, but since the existing PD2P implementation has removed the pressure on disk space, there is no immediate need to make changes.

Kaushik noted that distribution to SWT2 was stressed under the old system, but is now network idle.

Michael - looks promising but have observed an imbalance in the last couple of days (too large a fraction of jobs at T1). 11k jobs in short queue, MWT2 not utilized, AGLT2 fluctuating a lot.

Kaushik, the problem at MWT2 is their non-standard naming scheme for their SpaceTokens. This has fooled the subscription system.

Michael - we also need to assess job throughput, hopefully we can show the effect of increased throughput due to PD2P.

Preparatory discussion of issues to be addressed at the Software and Computing Week (All)

There is growing enthusiasm for merging some ATLAS clouds. Some Europeans had floated the idea of a US-Canada merge.

The apparent success of PD2P may strengthen rumors in the wider LHC world, that ATLAS uses resources inefficiently. Richard suggested that ATLAS would need to be ready with resource requirements based on an efficient implementation of its computing model by the time of the next RRB. All agreed that this issue had to be addressed carefully as an ATLAS rather than US ATLAS effort.


Richard promised to publicize Kevin's "Additional Production" web page real soon now.

Action Items

  1. 5/7/2010: Richard, Create a web page summarizing the dataset distribution targets in the US (Called into question by the abandonment of distribution targets in favor of usage-driven data distribution)
  2. 4/9/2010: Kevin, Create first version of a Twiki guiding US physicists on requesting Additional Production. (Completed)
  3. 3/26/2010: All but especially core team, Find time during the ADC Workshop next week to identify a point-of-contact for Valid US Regional Production (on hold pending a better definition of what this task should be following successful completion of some Exotics Additional Production).

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback