r1 - 10 Jul 2008 - 15:43:51 - StephanieMajewskiYou are here: TWiki >  AtlasSoftware Web > ProofTestBedUsingDQIF

Using DQIF


Using DQ2IF? to add files in a DQ2 dataset to a TDSet or TChain

For now it is not possible to browse the data in the xrootd server, but this is high on the to-do list. Also, we eventually would like to have the output of pathena jobs automatically put on the xrootd server (probably via a DQ2 subscription treating the xrootd server as a storage element). Tadashi has already demonstrated this is possible.

Once the datasets are resident on the xrootd server, we can get the file names via DQ2 in ROOT as Tadashi has previously shown. Tadashi has also modified DQ2IF.py to directly add files to a TDSet or TChain using DQ2IF.getFiles.

Copy the DQ2IF.py file to your local directory (included in PhysicsAnalysis/DistributedAnalysis/PandaTools):

cp /usatlas/u/maeno/offline/12.0.6/PhysicsAnalysis/DistributedAnalysis/PandaTools/python/DQ2IF.py .

DQ2IF? can override the path to the physical files as stored in the catalog and take a user-defined path instead. It also supports some pattern matching, so that you can add only the files with "AANT0" in their name (it will also exclude the log files). Here's an example using DQ2IF? to add all the files in a DQ2 dataset that have "AANT0" in their name prepending the xrootd path explicitly:

TDSet *d = new TDSet("TTree","FullRec0");
//TChain *d = new TChain("FullRec0");  // also works with TChain, but PROOF won't be used.

TPython::LoadMacro("DQ2IF.py");
DQ2IF dq2;
TPython::Bind(d,"myTDSet");
dq2.getFiles("user.TARRADEFabien.trig1_misal1_mc12.008331.PythiaVBFH120tautaulh.Athena_12.0.6.GroupArea_12.0.6.6.Jamboree_II-HightPtView-00-00-30.AAN","myTDSet" ,"root://acas0420.usatlas.bnl.gov//data/cache/HPTV/","AANT0");

This module should work with pyROOT too. So once DPDs are registered in DQ2 and LRC, one can easily use the info in PROOF sessions.

Tadashi's example to move pathena output directly to xrd storagelement

  • One can specify destination SE in pathena using --destSE. Eg. pathena --destSE SLACXRD. Then output will be copied to SLAC xrootd SE via subscription.


Major updates:
-- TWikiAdminGroup - 18 Jan 2018

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


Attachments

 
Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback