Using the BNL Xrootd/PROOF Farm
What is PROOF?
The Parallel ROOT Facility, PROOF, is an extension of ROOT allowing transparent analysis of large sets of ROOT files in parallel on clusters of computers or multi-core machines. There is an official PROOF website
and a PROOF forum
in Root Talk, as well as an ATLAS hypernews forum
What can I use PROOF for?
PROOF can be used to analyze any ROOT ntuple, including tertiary DPDs and AODs (using AthenaROOTAccess
). It's fast
, allowing you to analyze the events in a
in parallel over many CPUs.
What is the BNL Xrootd/PROOF Farm?
A new Xrootd/PROOF farm was set up at BNL in summer 2010.
It is a part of the ATLAS Computing Facility (ACF) at BNL and is located inside the ACF T1 firewall perimeter.
As such it is only accessible from ACF’s interactive and batch nodes.
Though ssh tunnel through the ACF gateway should work too.
Right now it can be accessed by any user with ATLAS credentials at ACF.
We may restrict access to the farm in the future, if necessary.
The farm redirector node is xrd.usatlas.bnl.gov
Currently the farm has 10 data server nodes and 3 "data vault" nodes.
All these nodes have Xeon 5560 8-core CPUs running at 2.8 GHz, with 24 GB of RAM and 1Gb NIC.
Data servers have ~2 TB of local disk space (4 SATA disks with 500GB each in RAID0 configuration).
Data vaults have ~12 TB each (6 SATA disks with 2TB each in RAID0 configuration).
The PROOF farm master node is acas1010.usatlas.bnl.gov
). It consists of 12 slave nodes (acas1001-1009, acas0784, acas0786, acas0787), with a total of 96 CPUs.
For OS level monitoring we use ACF-wide set up based on Ganglia.
One can see Xrootd/PROOF farm Ganglia pages for the first server
and second server
Getting Started with PROOF at BNL
To use the PROOF farm, you need to have access to BNL's interactive nodes. If you do not have an account on acas, you can apply here
If you would like to try to run PROOF locally instead of at BNL, follow the xproofd setup instructions
on the ROOT website (NOTE: in order to run
, you need to setup ROOT).
Setting up ROOT at BNL
Shell scripts are provided in setting up ROOT at BNL, that is:
source /afs/usatlas/scripts/root_set-slc5.sh 5.27.04
source /afs/usatlas/scripts/root_set-slc5.csh 5.27.04
You can switch to another ROOT version using the above scripts. It will take care of removing the old one from the environmental variable and adding the new one.
Getting started packages
Two examples are provided at BNL: one in C++
, the other in python
. They just fill out one histogram and write out a new TTree for those events passing cuts.
* Example-1 in C++
: under the directory
, there are 3 relevant source files and one Readme file:
* Example-1 in python
: under the directory
, there are 2 relevant source files and one Readme file:
Just copy each example to your directory and follow the instruction in 00Readme.txt
to try it. It should work for all cases: Proof, Proof-Lite or non-Proof ROOT. Try all cases and find out the performance difference.
Adding clist to a TChain or TDSet
For each dataset in xrootd, a clist file is created under directory ~xrdadmin/xrd_copied dataset
. You can make use of the clist to add the list of files into your TChain or TDSet, for example:
TFileCollection* fileColl = new TFileCollection("fileColl");
TChain* chain = new TChain("qcd");
// or for TDSet
TDSet* dset = new TDSet("dset","qcd");
Run a Very Simple Example
In this example, you will open a PROOF session, add a file to a
a simple quantity.
TProof *p = TProof::Open("acas1010");
TChain *ch = new TChain("FullRec0");
TH1F *htemp = p->GetOutputList()->At(3);
Due to a bug in current ROOT version (5.27.04), the canvas will disappear after the merging is done. A fix is already in the ROOT head version. However, you can always get the output histogram from the output list in TProof.
The output should look something like this:
Reset Your PROOF Session
Sometimes your PROOF session will get corrupted and TProof::Open() will hang. Reset your session to restore functionality:
or for a hard reset,
Finding Available Data
Name space convention for the Xrootd servers is "root://xrd.usatlas.bnl.gov//data/datasetname/filename".
Short form "root://xrd//data/datasetname/filename should be also acceptable.
We generate, at copy time, for every copied dataset, a list of files in a dataset, with fully qualified names for the "xrd.usatlas.bnl.gov" farm.
These file lists can be found at: ~xrdadmin/xrd_copied dataset directory that is accessible from ACF interactive nodes.
The files are named in the following format "datasetname.clist" and each file corresponds to one dataset.
As we already mentioned, each such file contains a list of fully qualified file names belonging to that dataset and can be used as input for your analysis.
We already tested and will be using so called PQ2 tools (more information about PQ2 tools ca be found at main PROOF page at CERN) for data registration and data discovery on the farm.
Full availability of the PQ2 tools will be announced when appropriate.
Currently we are replicating the WWD3PD? from the SM EW subgroup, and QCD d3pd from Jet/EtMiss group (as well as some slimmed version), and some tau performance group d3pd.
If you need to copy your data to Xrootd - contact Hong Ma (firstname.lastname@example.org
) or Sergey Panitkin (email@example.com
Additional Useful Links
- From the BNL Jamboree, June 2008: Sergey's talk, Shuwei's talk
- *Introduction to PROOF farm at BNL and Analysis Tutorial is here.
- Updated! PROOF in Atlas talk at CHEP09 Conference is here. Corresponding paper can be found below in attachments.
- Updated! Talk about use of SSDs and I/O related issues in PROOF is here. Corresponding paper can be found below in attachments.
- Datasets on the xrootd server, DatasetsOnXRD
- 25 May 2013
, 16 Feb 2010 - 14:37 | CHEP09 paper on use of SSDs in PROOF environment and related I/O issues
, 16 Feb 2010 - 14:34 | CHEP09 talk about Distributed analysis with PROOF in Atlas
, 18 Jul 2007 - 11:10 | Basic commands to setup proof, add many files to a dataset, and make a simple plot, run a TSelector
(389.8K) | Main.serp, 01 Aug 2007 - 18:44 | Xrootd monotoring talk from Edgar and Ofer