r3 - 25 Feb 2011 - 18:32:44 - RichardMountYou are here: TWiki >  AtlasSoftware Web > Minutes10Dec2010

Minutes10Dec2010 RAC Minutes, December 10, 2010

Members (*=present, #=apologies)

*Richard Mount, Kevin Black begin_of_the_skype_highlighting     end_of_the_skype_highlighting (Physics Forum Chair), *Jim Cochran (Analysis Support Manager), Alexei Klimentov (ATLAS ADC), Ian Hinchliffe (Physics Advisor), Rik Yoshida (Tier3 Coordinator), *Michael Ernst (U.S. Facilities Manager), Rob Gardner (Integration Coordinator), #Kaushik De, [apologies stuck in faculty meeting] (U.S. Operations Manager), Armen Vartapetian (U.S. Operations Deputy)

Ex-Officio: Torre Wenaus, Stephane Willocq, Mike Tuts, Howard Gordon

Approval or Correction of Miinutes

All previous draft minutes were approved

Operations Report (Armen)

CPU Usage

The Geant4 component of the simulation production is about 90% complete and reconstruction is in progress. Some MC09 is also running - needed for some of the papers that should be published soon. In general, production is stable (discounting the recent Panda database issues that caused some jobs to be killed). There are about 10k production jobs running.

The ~150M 8TEV events planned to be simulated real soon now should take about one month (Simulation production typically gets 7.5 Mevts per day).

Storage Issues

BNL is now stable following the reprocessing. There have been some tense moments, but there is now about 1PB free at BNL. There are immediate plans to merge UserDisk and GroupDisk, and the longer term trend is to get rid of space tokens.

The storage issues shifted to the T2s towards the end of the week - modifications to PD2P produced many unexpected secondary copies. The worst effect was at MWT2, likely because they have the best network connection and could be filled up at 2.5 to 3 TB/hour. Nevertheless, disaster was averted by accelerating deletions and adding some storage.

In general, central deletion is almost managing to catch up (but some manual deletions at SLAC and NET2 will be needed this weekend).

Additional Production

The production requested by Ian Hinchliffe has been done.


Michael noted that following the completion of reprocessing, BNL had reverted to offering 2,500 cores for production and 2,500 for analysis.

Richard asked if it was easy to get statistics on the turnround time for analysis jobs. Michael said that using the existing Panda Monitoring to get this information was difficult and tedious. Alden had some scripts, but we needed development work to make these statistics readily available.

Michael also noted some imbalance in the treatment of analysis by the various clouds. In the US, analysis was strongly supported and given priority access to a substantial fraction of T1 and T2 resources. Some other clouds preferred to focus more on production. This situation would be unlikely to change without strong pressure from the physics analysis community.

Keeping Facilities Gainfully Employed

Richard noted that ATLAS did not have a systematic approach to maintaining a queue of lower priority production activities (typically adding statistics to existing official simulation productions). This had become clear in discussions (in the CREM and privately) about the production to be performed over the end-of-year holidays. It seemed that that a sufficient amount of high-priority official production would be probably available, but the idea of running lower priority simulations was not universally accepted. Space (where to put the output) was one issue, but there were solutions. For example, RAC members suggested that only the AOD be stored for the lower priority simulations.

Richard was encouraged to discuss the issue informally with members of the physics management.



Action Items

  1. 9/24/2020: Kaushik: Organize the nature and timing of the effort to create an automated mechanism to give US-requested additional production priority access to US non-pledged resources. On hold - manual system works just fine for now

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback