r10 - 23 Aug 2006 - 11:20:08 - JerryGieraltowskiYou are here: TWiki >  AtlasSoftware Web > ArgonneASCComputingResources

Argonne Analysis Support Center

Computing Resources


This TWIKI page is based on the previous *non-TWIKI* version of ASC Computing Resources maintained by Tom LeCompte.

Present Hardware/ATLAS Releases

atlas16: Two dual-core Opteron 280 processors, SLC 305
atlas17: Two dual-core Opteron 280 processors, SLC 305

As promised, Release 11.0.41 has been removed from atlas16 & atlas17. You should use 11.0.5 instead.

Release 12.0.0 has been installed and is available for use. Because the trigger group is still developing code that was supposed to go into 12.0.0, there will be a 12.0.1, 12.0.2 etc. branch on a fairly fast schedule. I intend to keep two 12.0.x releases - the newest one, and the previous best one. Once 12.0.x stabilizes, I will drop the prior release 12 candidate.

The plan is to keep 11.0.5 until a stable release 13 is out.

Policies and Procedures

Argonne ASC Computing Resources: Policies and Procedures

User Guide

Argonne ASC Computing Resources: User Guide

Managing Tier 3 Expectations

The overall goal is to build a system of computers designed for interactive ATLAS use. The intent is that this will be the primary computing farm for ANL HEP scientific staff, that it be open to all of US ATLAS, and it will serve as a "gateway" machine to the Midwest Tier 2: for example, user code may be developed, tested and debugged here, and when ready to run on a medium or large dataset, a "batch job" would be submitted from the Tier 3 that would run on the Midwest Tier 2. In principle, a job should be launchable from the ANL Tier 3 to any ATLAS site in the world, but we anticipate a special relationship with the Midwest Tier 2. The central ATLAS computing model supports large shared ntupled datasets. The Tier 3 will support moderately sized collections of ntuples used by small analysis groups.
  • Environment
    • By Spring:The overall environment will match the lxplus environment as closely as possible - the idea is that followiing instructions for lxplus should get you the same result here. This includes editors, utilities, and when possible, naming standards. There will be only minimal support for email (.forward files and sending mail when a job finishes, for example) and probably none for personal web pages. "Golden releases" will be supported. For the moment, this means 11.0.5 as soon as it is released. We will not support every single release, nor will we support nightlies. Access to the CVS repository is through CERN. It will be possible to submit jobs worldwide, but it will be simplest for the Midwest Tier 2. We will be running 32-bit Scientific Linux 3.
    • By Fall: We will still follow the strategy of "golden releases", but there will probably be more of them. Perhaps every developer's release for the present release, and a golden release for the previous two releases. There will be some sort of local access to the source code, at least for golden releases.If all goes according to schedule, we will running 64-bit Scientific Linux. It may be that we are still running 32-bit Athena, but even in that configuration it will help with our memory constraints. It will be easier to submit jobs worldwide. Based on lxplus use need about 1 GB of disk space per user, plus whatever space needed for data.
    • Today: 11.0.5 is supported. Access to the CVS repository is through CERN and kerberos. It works, but is slow. We are running 32-bit Scientific Linux 3.
  • Accounts
    • By Spring: We badge people who want to use the Tier 3 and get them accounts by hand.
    • By Fall: We need to set up something with lower overhead.
    • Today: If you want an account, email me.
  • Computers - In general, we expect to have two kinds of systems operational. "Production", which will be where the bulk of the activity is, and "Test", which will be a smaller computer or cluster of computers where new configurations are tested. Anyone with an account should have access to both systems. (i.e. we won't be restricting who can help shake down the Test configuration to ANL personnel)
    • Today: We have 2 desktop computers running SLC3, and configured to support multiple users. Configuration is:
Dual Opteron 280 processors 
4 GB of memory per computer, expandable to 8 GB 
200 GB (x2) of RAID storage 
Dual Gigabit Ethernet 
SLC 305
The ATLAS recommendation is 2 GB of memory per core (hence the 8 GB). It's presently limited to 3 GB because of 32-bit Linux addressing issues. Running RecExCommon? requires a memory footprint of 1.3 GB.
    • By Fall: We should add ~1 TB of network storage, a dedicated GbE? backbone between the computers, and GbE? to the site boundary. Some additional number of computers (2?) will be added.
  • Network
    • Today: We have a 100 Mbps connection to the outside world.
    • By Fall: We will have a 1000 Mbps GbE? connection to the outside world.
  • Event Data Access
    • By Spring: A small amount of data will be local - probably a handful of files from each DC2/Rome datatset. This should be enough to debug code. To get more data, you need to run the job batch on the Tier 2.
    • By Fall: I envision more data, but I don't have a clear picture of the expected use patterns, so I don't know what will be local, or even how much of it there will be. By this time we shoud have some sort of catalog installed so that if someone wants a file that is not physically present, there is a straightforward way of locating, replicating and cataloging it.
    • Today: If you want a file, you need to find it yourself, scp it to atlas16 or 17 and run pool_insertFileToCatalog.

  • FAQ
    1. Will we support laptops? - I would expect we would publish the instructions on how to set up a Tier 3 just like ours, so universities can set up their own clusters if they want. Certainly one can plug a laptop into our network (wired or wireless) and open a sesssion on the Tier 3.
    2. Scratch space? Backup policy? -The backup policy is listed in the Tier 3 Policies document. There is no scratch space per se (besides /tmp), although there is a shared data area that anyone can write to on /data.
    3. You're putting together all this computing. Can we run batch jobs on it? - The goal is interactive use, so any batch use should not interfere with that. It might be possible to use this as a compute element for the Midwest Tier 2 and run some batch jobs on it, but we need to be careful that this doesn't interfere with our primary mission of interactive use. Note that the total computing power is less than 5% of the Midwest Tier 2.

System Administrator Guide

Argonne ASC Computing Resources: System Administrator Guide
Major Updates:
In order to edit any of the ArgonneASC pages you must have a valid Registration. If you edit this page, please include your WikiName and date of update.
Main.JerryGieraltowski - set link for SysAdmin back to server at ANL

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback