Using AthenaROOTAccess with PROOF

Run AthenaROOTAccess (ARA) in PROOF

If you do not know how to use ARA to do AOD analysis, you should visit here first.

The following describes how to run ARA job in PROOF, to take the advantage of parallel processing PROOF provided. It is still in test stage and the instruction is not simple enough. We will simplify this instruction and make improvement after we do more tests and understand it better. However, you are encouraged to do your tests and your experience is highly welcome.

Environment setup on your client side

In order to run ARA, you need setup the environment of ATLAS release. The following instructions take release 14.1.0 as an example, which should also apply to other releases. First set up release 14.1.0 plus AthenaROOTAccess? groupArea:

    source /opt/usatlas/kit_rel/14.1.0/cmtsite/setup.sh -tag=14.1.0
    GroupArea=/afs/usatlas/project/analysis/ARA_GroupArea/ARA_Tags_14.1.0
    export CMTPATH=$GroupArea:$CMTPATH
    source /opt/usatlas/kit_rel/14.1.0/AtlasOffline/14.1.0/AtlasOfflineRunTime/cmt/setup.sh

After the release setup, please run the following to save 4 environment variables into a file which will be used later:

    proof=proof-setenv-14.1.0.sh
    echo "export ROOTSYS=$ROOTSYS" > $proof
    echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH" >> $proof
    echo "export PYTHONPATH=$PYTHONPATH" >> $proof
    PYTHONHOME=`which python | sed 's%/bin/python$%%'`
    echo "export PYTHONHOME=$PYTHONHOME" >> $proof

Please notice that the ROOT version for ARA is bound to the one coming with the release, and other ROOT version cannot be used.

Preparation of ARA analysis code

Creation of a TSelector relevant to the AOD ntuple.

As you may know (if not, please read this first), transient TTree is created on the fly using AthenaROOTAccess? . So you have to create a transient TTree first before running TTree::MakeSelector().

So the first step is to open one of your AOD ntuples and create a relevant transient TTree:

   $ root -l
   root [0] TPython::Exec("import PyCintex,AthenaROOTAccess.transientTree");
   root [1] TChain* chain = new TChain("CollectionTree");
   root [2] chain->Add(""/usatlas/groups/top/hma/fdr2/fdr08_run2.0052280.physics_Egamma.merge.AOD.o3_f8_m10/fdr08_run2.0052280.physics_Egamma.merge.AOD.o3_f8_m10._0001.1");
   root [2] TPython::Bind(chain,"myTree");
   root [3] TPython::LoadMacro("/usatlas/workarea/yesw2000/root/Proof/14.1.0/Examples/open-tree.py");
   root [4] TTree* tree_trans = gROOT->Get("CollectionTree_trans");
   root [5] tree_trans->MakeSelector("mySelector");

The TSelector files, "mySelector.h" and "mySelector.C", will be created.

The file open-tree.py in the above example reads:

global myTree
print myTree.GetName()

branchNames = {}
tt = AthenaROOTAccess.transientTree.makeTree(myTree,  branchNames = branchNames)
print tt

Modification of your TSelector files

You need do the following changes to your "mySelector.h":

1. Add a member m_TreeTran to your class mySelector with
   TTree* m_TreeTran;
and add members for pointer to histograms you want to create, something like:
   TH1F *h1_Gam_phi, *h1_mu_n, *h1_mu_m2mu;
2. In the Init function, add the following statements:

     TPython::Bind(tree,"myTree");

     TString str_TreeTran(tree->GetName());
     str_TreeTran += "_trans";

     TObject* obj = gROOT->FindObject(str_TreeTran.Data());
     if (obj) obj->Delete();

     TPython::LoadMacro("/usatlas/workarea/yesw2000/root/Proof/14.1.0/Examples/open-tree.py");
     m_TreeTran = (TTree*)gDirectory->Get(str_TreeTran.Data());

Next you need edit "mySelector.C":

1. In SlaveBegin function, define your histograms

   h1_Gam_phi = new TH1F("h1_Gam_phi","#phi(#gamma)", 100, -3.1416, 3.1416);
   fOutput->Add(h1_Gam_phi);

   h1_mu_n  = new TH1F("h1_mu_n","N(#mu)",10,0,10);
   fOutput->Add(h1_mu_n);

   h1_mu_m2mu  = new TH1F("h1_mu_m2mu","M(#mu#mu)",100,0,200.);
   fOutput->Add(h1_mu_m2mu);

1. In Process function, read the tree or branches and do your analysis

   // b_PhotonAODCollection->GetEntry(entry);
   // b_StreamingEventInfo->GetEntry(entry); 
   // b_MuidMuonCollection->GetEntry(entry); 
   m_TreeTran->GetEntry(entry);

   ... put your analysis code and fill histograms here ...

1. In Terminate, you can present the results graphically or save the results to file. For example,

   fOuput->Print();
   TFile* file = new TFile("my-results.root","RECREATE");
   fOutput->Write();
   file->Close();
 

Preparation on proof worker nodes before job running

Because the environment set up on your client side can not be propagated to proof work nodes where to run your AthenaROOTAccess? job in parallel, you have to tell the proof how to set up the environment required for ARA, which is done by the following 2nd ROOT statement (under "root [1]"):

   $ root -l
   root [0] TPython::LoadMacro("DQ2IF.py");
   root [1] TProof::AddEnvVar("PROOF_INITCMD","source /usatlas/workarea/yesw2000/root/Proof/14.1.0/Examples/proof-setenv-14.1.0.sh");
   root [2] TProof* p = TProof::Open("acas0420");
   root [3] p->Exec("TPython::Exec(\"import PyCintex,AthenaROOTAccess.transientTree\")");

• The 1st line loads a python script to look up xroot files in LRC.
• On 2nd line the 2nd argument specifies the shell script to set the environment variables ROOTSYS, LD_LIBRARY_PATH and PYTHONPATH.
  Just copy these variables value you get from your release setup.
• The 3rd line above (under "root [2]") makes connection to the proof server.
• The 4th line loads the necessary libraries and modules.

Now it is time to define a TDSet or TChain and process it:

   root [4] TChain* chain = new TChain("CollectionTree");
   root [5] TPython::Bind(chain,"myChain");
   root [6] DQ2IF dq2;
   root [7] dq2.getFiles("fdr08_run2.0052283.physics_Egamma.merge.AOD.o3_f8_m10", "myChain");
   root [8] chain->SetProof(kTRUE,kTRUE);
   root [9] chain->Process("mySelector.C");

Sample files

You can find examples and instructions in a package ARA-PROOF_BNL, which can be checked out via cvs co groups/PROOF/ARA-PROOF_BNL.

About This Site

Please note that this site is a content mirror of the BNL USATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your BNL USATLAS account.

Attachments