Tutorial of GoodRunLists and LumiCalc at BNL

---

Introduction

• This is a simplified BNL-specific tutorial modified from GoodRunsListsTutorial and CoolLumiCalcTutorial at CERN. However, it should also work for kit at any site after replacing the path of kit location. Please refer to the tutorials for more examples.

• Described in the tutorial below is the creation of good-run-list configuration files and calculation of integrated luminosity. Good run lists are written in XML format (developed by TAG group), and carry meta-data on queries used to make them. They are generated by the 'GoodRunsLists' package. In this tutorial we also discuss the creation, reading, manipulation, and writing of good run lists, and the application of good-run-lists in both athena and non-athena environments, and calculation of integrated luminosity.

Preparation of setting up Athena and required package(s)

In the examples below, we will perform the tutorial with Athena release 15.5.1.

Setting up Athena

The location of release 15.5.1 at BNL is /opt/usatlas/kit_rel/15.5.1.

1. Make a working directory:

mkdir /usatlas/scratch/$USER/15.5.1
cd /usatlas/scratch/$USER/15.5.1
export TestArea=$PWD

2. Now you can set up release 15.5.1

cd $TestArea
source /opt/usatlas/kit_rel/15.5.1/cmtsite/setup.sh -tag=15.5.1
export CMTPATH=$PWD:${CMTPATH}
source /opt/usatlas/kit_rel/15.5.1/AtlasOffline/15.5.1/AtlasOfflineRunTime/cmt/setup.sh

Later we will create a shell script to include everything we need source for a new session.

Setting up dblookup.xml

The DBRelease settings will use sqlite200 by default. In order to use oracle instead on command line, we need modify dblookup.xml.

1. Make a directory XMLConfig:

cd $TestArea
mkdir XMLConfig
cp $CORAL_DBLOOKUP_PATH/dblookup.xml XMLConfig/

2. Next modify XMLConfig/dblookup.xml. We see 2 occurrences of _TRIGGER"> in that file.
The first occurrence and the 2 following lines:

<logicalservice name="COOLONL_TRIGGER">
<service name="sqlite_file:sqlite200/ALLP200.db" accessMode="read" />
<service name="sqlite_file:/afs/usatlas.bnl.gov/software/downloads/releases/15.5.1/DBRelease/7.4.1/sqlite200/ALLP200.db" accessMode="read" />

The second occurrence and the 2 following lines:

<logicalservice name="COOLOFL_TRIGGER">
<service name="sqlite_file:sqlite200/ALLP200.db" accessMode="read" />
<service name="sqlite_file:/afs/usatlas.bnl.gov/software/downloads/releases/15.5.1/DBRelease/7.4.1/sqlite200/ALLP200.db" accessMode="read" />

Just remove the 2nd following line, i.e., the lines containing sqlite_file:/afs/usatlas.bnl.gov.

In the same two blocks (COOLOFL_TRIGGER and COOLONL_TRIGGER), move the lines for the BNL server up to the top of the list, so that the BNL Oracle server is picked when the LumiCalc? job is run.

 
<service name="oracle://BNL/ATLAS_COOLONL_TRIGGER" accessMode="read" authentication="password" />
 

And similar line for ATLAS_COOLOFL_TRIGGER .

3. Set an env variable to override the default location of dblookup.xml.

export CORAL_DBLOOKUP_PATH=$TestArea/XMLConfig

Setting up the good run list package(s)

1. Check out the recommended version of the good run list and related packages.

cd $TestArea
bnl_cmt co -r GoodRunsListsUser-00-00-04 DataQuality/GoodRunsListsUser
bnl_cmt co -r GoodRunsLists-00-00-57 DataQuality/GoodRunsLists
bnl_cmt co -r LumiBlockComps-00-00-58 LumiBlock/LumiBlockComps
bnl_cmt co -r CoolRunQuery-00-00-94 Database/CoolRunQuery

where command bnl_cmt will use the BNL local mirror of ATLAS SVN repository.

2. Now we can compile:

cd $TestArea/Database/CoolRunQuery/cmt/; cmt make
cd $TestArea/DataQuality/GoodRunsLists/cmt/; cmt make
cd $TestArea/LumiBlock/LumiBlockComps/cmt/; cmt make
cd $TestArea/DataQuality/GoodRunsListsUser/cmt/; cmt make

Setting up Athena and required package(s)

It is useful to make the script init.sh .

rel=15.5.1
SITEROOT=/opt/usatlas/kit_rel/$rel
proj=AtlasOffline

source $SITEROOT/cmtsite/setup.sh -tag=$rel,$proj
export CMTPATH=$PWD:${CMTPATH}
source $SITEROOT/$proj/$rel/${proj}RunTime/cmt/setup.sh

export TestArea=$PWD
export CORAL_DBLOOKUP_PATH=$TestArea/XMLConfig

Then do source init.sh This you will have to do every time you want to use the good run list packages in a new shell.

You are now ready to use the GoodRunsList and LumiBlock tools!

Creating good run list configurations and new good run lists

Please open the following two links as new tabs in your browser:

• The AtlRunQuery on webpage.
• List of available Data Quality flags.

Take a look at the list of available DQ flags. Try to find the list of secondary DQ flags, defining good physics objects. What number to they start with? Note the granularity of the Data Quality flags; usually they are defined in terms of barrel and end-caps only. (For the muons, they are also to be defined in terms of muon reconstruction algorithm.)

The secondary flags, which are to be used for a physics analysis, are (currently) in the process being finalized by CP groups and the DQ group. For this reason, in the tutorial below we will mostly work with the primary detector flags.

The AtlRunQuery tool - web-based version and in a shell

(If you are unfamiliar with AtlRunQuery tool, it is recommended to visit the AtlRunQuery webpage first before continuing further, and to try out a few example search queries. The run-query tool is a really powerful tool to search for good runs! A presentation by Joerg Stelzer can be found here.)

Good run lists are compiled from Data Quality information stored in the conditions database COOL, using the AtlRunQuery tool. The AtlRunQuery tool can be run in a shell and called on the AtlRunQuery webpage directly.

Query via wegpage

Let's run the following query in the AtlRunQuery webpage:

find run 132350+ / show run and events and dq

You will find more than a 100 recent cosmics runs that are listed in the search results. Next to each run you find shown: the number of lumi-blocks per run, the number events, and a list of all DQ flags available for that run. (Remember that DQ flags get assigned for each luminosity block in a run.)

Now scroll down to take a look at some of the DQ flags that have been set. If you don't know the meaning of all the flags (and note that there are quite a few!), remember that you can always find an overview of all available DQ flags here. Please take a look at this list to see the meaning of some of the flags you find, eg: TRMBI, RCOPS, EIDE, PIDCR.

You will find that many flags are actually not yet assigned ('n.a.'). Important note: this means the flag can (still) be any value, also green! Eg. there are many Pixel and SCT flags that are set in black, green, red, and sometimes yellow. Yellow detector flags are 'under evaluation', and still need to be converted to red or green. For some runs and DQ flags, part of the run is good and the other part is bad.

Scrolling all the way to the right you will find the DQ flags to be set by the combined performance groups, eg. for muons, missing et, jets, bjets. Note that the DQ flags for jets are already (sometimes) assigned.

Scrolling back to the left, and all the way down to the bottom of the screen, you will see a link: "Result table as XML file for input to good-run lists (right-click link to download)". You can take a quick look at the link. We will discuss the output file in more detail later, below. For example, note - at the top of the file - that the xml file contains the search query used to make the good run list.

For more examples of AtlRunQuery searches, scroll all the way to the top of the page, and click on the tab "Data Quality". Eg. do the query:

find run 132350+ and dq sct green SHIFTOFL / show run and events and dq

which again shows the same set of runs with the additional selection that all DQ flags with the string 'sct' in the name. Note that flags set to 'n.a.' will also be selected. (This will be fixed in the near future.)

The end-user information is currently stored in the DQ flags called SHIFTOFL, meaning "SHIFT OFFLINE" or offline data quality shifter. The the SHIFT OFFLINE DQ flags are currently filled on a daily basis for all runs with more than 100k events.

Query from shell command

Let's go back to the shell.

cd $TestArea/DataQuality/GoodRunsLists/run

You can do the same run query (as before) directly in a shell by doing:

AtlRunQuery.py "find run 132350+ and dq sct green SHIFTOFL / show run and events"

The good-run-list, with all selected good runs and luminosity blocks, is stored as the file data/MyLBCollection.xml. (More about this file later.)

A good run list configuration for the TopMix sample

The topmix MC sample, by Richard Hawkings, is a mixed event sample (initially of about 200 pb^-1 at 10 TeV? ), representing the events selected (including background) for typical ttbar and single top selections (semileptonic and dileptonic selections). The topmix MC sample has (dummy) DQ flags and (more realistic) luminosity information defined in COOL. As such, this is an excellent MC sample for testing DQ selections and the corresponding luminosity calculation. The sample consists of only one run, with number 108175, containing over 20000 luminosity blocks. For more information on the topmix sample, see here.

The GoodRunsLists package contains a predefined good-run-list configuration file for the TopMix MC sample.
Take a look at the file: $TestArea/DataQuality/GoodRunsLists/python/TopMixConfig.py

from GoodRunsLists.GoodRunsListConfig import GoodRunsListConfig

class TopMixConfig(GoodRunsListConfig):
  def __init__(self):
    GoodRunsListConfig.__init__(self)

    ## name in GRLB object, and below the xml filename
    self.name            = "TopMixConfig"
    self.listname        = "TopMixConfig.xml"

    ## Specify each selection of dq flags here.
    ## The query is interpreted as the logical AND of all elements in the array.

    ## All selection criteria at: Database/CoolRunQuery/python/AtlRunQueryParser.py
    ## possible dq flag colors : 'n.a.', 'unknown', 'green', 'yellow', 'red'
    ## possible detectors (incomplete): pix,sct,em,til,lar,any 

    ## See http://atlas-runquery.cern.ch for more user examples.
    ## Example: "f r 108175 and db MC and dq pix LBSUMM g and dq EMBA lbsumm g and ctag DetStatusLBSumm-TopMix1 / sh r and dq LBSUMM / nodef"

    ## Reset default settings (= cosmics data related)
    self.querydict = {}
    self.querydict['run']  = ['108175']
    self.querydict['db']   = ['MC']
    self.querydict['ctag'] = ['DetStatusLBSumm-TopMix1']

    ## Set the dqflags query
    self.querydict['dq']   = [ 'sct   green LBSUMM',
                             ]

It looks much like the configuration file for cosmics, except that this one uses run 108175, the MC instance of the COOL db, with the conditions tag DetStatusLBSumm-TopMix1.
Most importantly, the DQ selection is based on the COOL folder 'LBSUMM'. The LBSUMM folder contains the combined summary of all types of DQ flags set, containing on- and offline shifter information, automated on- and offline histograms checks, and DCS information (being: detector slow control, describing eg. per sub-detector temperature and voltages for noise levels). When all DQ flags are filled properly, which is currently not yet done for cosmics), LBSUMM will be the proper COOL folder to select on DQ flags.

Now do:

cd $TestArea/DataQuality/GoodRunsLists/run/
create_goodrunslist TopMixConfig

The resulting good run list is stored as data/MyLBCollection.xml.

For illustration, let's copy-paste the search query

find db MC and dq sct   green LBSUMM and run 108175 and ctag DetStatusLBSumm-TopMix1 / show run and events and dq LBSUMM / nodef

into the AtlRunQuery web page.
This shows the interesting property that, also within a run, DQ flags can be flagged as good or bad. Note that, not all DQ flags are set, most notably the secondary flags corresponding the quality of physics objects such as jets, muons, electrons, etc.

Interlude: Interaction of good-run-list (and trigger) selection with luminosity determination

Before we continue with the section about good-run-list selection tools in Athena, it is important to know how good-run-list selection interact with the luminosity determination.

In a physics analysis, the selection DQ flags needs to happen in twice:
1. In the event loop, to skip events from bad LBs.
2. In the book-keeping of all LBs processed, done by the LumiBlockMetaDataTool (discussed in more detail later).
In an athena ntuple production job, these two selections happen in two different places. The tool provided to select on DQ flags, called GoodRunsListSelectorTool (discussed below), has been interfaced with the LumiBlockMetaDataTool such that the identical LB selection is automatically applied in both places.

If you do not wish to use the provided Athena tools to do selection on DQ approved lumi blocks (for example because you wish to apply the selection yourself at the ntuple level), be sure to apply your DQ selection twice, both in event-loop and the LB book keeping.

On a similar note, the luminosity calculation requires two basic ingredients:
1. The DQ approved lumi-block collection
2. The list of trigger names used in your trigger selection (more about this in the follow-up tutorial)
Hence, the luminosity (calculation and) book-keeping tool not only interfaces with the GoodRunsListSelectorTool, but also needs the list of trigger names used in your selection.

In the Athena examples that follow, the selection of events from DQ approved lumi blocks is performed by an algorithm called GRLTriggerSelectorAlg. As the name implies, this algorithm can also do per-event trigger selection. (This feature is not turned on below, but will be used in the Luminosity Calculation part of the tutorial.) The algorithm GRLTriggerSelectorAlg automatically registers the trigger names and good-run-lists used, so they can be picked up in a follow-up luminosity calculation.

If you do not wish to apply trigger selection at the Athena level (for example because you wish to apply the selection yourself on the ntuple level), again keep in mind that a consistent set of trigger names needs to applied at the event level selection and at the luminosity calculation.

Good run list selection tools in Athena

We will exercise the selection of good runs and luminosity blocks running in Athena, using good run lists as input, with the topmix MC sample. In the following subsection, we use the jobOptions file
$TestArea/DataQuality/GoodRunsListsUser/share/GoodRunsListsUser_jobOptions.py,
which is called simply by running:
athena GoodRunsListsUser/GoodRunsListsUser_jobOptions.py .
(We will do this below.)

Looking at the jobOptions file, there are two things to mention.

First, there are four sub jobOptions files we will switch between in the following subsections.

UserAlgs=[ "GoodRunsListsUser/GoodRunsListsUser_oneSelection.py"
           #"GoodRunsListsUser/GoodRunsListsUser_twoSelections.py",
           #"GoodRunsListsUser/GoodRunsListsUser_preEventSelector.py",
           #"GoodRunsListsUser/GoodRunsListsUser_coolQuery.py"
           #"GoodRunsListsUser/LumiBlock_LumiCalc_GRL_dumper.py"
         ]

These jobOptions files are picked up and run in RecExCommon/RecExCommon_topOptions.py, included at the bottom of the file.
(The script LumiBlock_LumiCalc_GRL_dumper.py will be studied in the Luminosity Calculation tutorial.)

Second, we use two AOD files, namely

dcache:/pnfs/usatlas.bnl.gov/LOCALGROUPDISK/user/RichardHawkings/user.RichardHawkings.0108175.topmix_Muon.AOD.v4/user.RichardHawkings.0108175.topmix_Muon.AOD.v4._00014.pool.root
dcache:/pnfs/usatlas.bnl.gov/LOCALGROUPDISK/user/RichardHawkings/user.RichardHawkings.0108175.topmix_Muon.AOD.v4/user.RichardHawkings.0108175.topmix_Muon.AOD.v4._00015.pool.root

If you are not running at cern, you can find this sample as dq2-sample user.RichardHawkings.0108175.topmix_Muon.AOD.v5. See this wiki for more details on the topmix sample.

The following exercises use the good run lists grl_topmix_mu.xml, grl_topmix_e.xml, and blacklist.xml, found in the GoodRunsListsUser/run/ directory, which are based on the two topmic AOD collections listed above.

All following jobs will be in the GoodRunsListsUser/run/ directory:

cd $TestArea/DataQuality/GoodRunsListsUser/run/

Default good run list selection in an athena job

This section described the recommended procedure for applying a DQ selection based on good run lists in an athena job. The example below demonstrates the use can of one (set of) selections applied in an athena job that produces a flat ntuple with our (dummy) physics events of interest.

Before running our first job that performs good run and lumi-block selection in athena, let's take a look at the jobOptions file:
$TestArea/DataQuality/GoodRunsListsUser/share/GoodRunsListsUser_oneSelection.py .

First, the GoodRunsListSelectorTool. The GoodRunsListSelectorTool contains functions to perform the actual selection of good runs and luminosity blocks, based on the good- and black run lists provided as input.

ToolSvc += GoodRunsListSelectorTool() 
GoodRunsListSelectorTool.GoodRunsListVec  = [ 'grl_topmix_mu.xml' ]
GoodRunsListSelectorTool.BlackRunsListVec = [ 'blacklist.xml' ]

In this example, these are the xml files grl_topmix_mu.xml and blacklist.xml, and can be found in the run/ directory. (Note that black run lists will usually not be required. Here, the functionality is shown merely as an example.)

Behind the scene, the GoodRunsListSelectorTool interacts with the LumiBlockMetaDataTool to ensure that, in the bookkeeping of all luminosity blocks processed during our athena job, only DQ approved luminosity blocks are accounted for.

Next, the sequence of user algorithms run in our job.

job = AlgSequence()
job += AthSequencer("ModSequence1")
job.ModSequence1 += GRLTriggerSelectorAlg('GRLTriggerAlg1')
job.ModSequence1 += DummyDumperAlg('DummyDumperAlg1')

A dedicated sequence ModSequence1 is added to the top job-sequence.
The algorithms GRLTriggerAlg1 and DummyDumperAlg1 are added to ModSequence1.

GRLTriggerSelectorAlg uses GoodRunsListSelectorTool to determine for each event whether it passes the DQ selection, or whether the event is rejected.

job.ModSequence1.GRLTriggerAlg1.GoodRunsListArray  = ['grl_topmix_mu']
job.ModSequence1.GRLTriggerAlg1.BlackRunsListArray = ['blacklist']

As good- and black run lists GRLTriggerAlg1 uses: 'grl_topmix_mu' and 'blacklist', which are the names of the good run list objects defined in grl_topmix_mu.xml and blacklist.xml respectively.

(As the name suggest, the algorithm GRLTriggerSelectorAlg can also do per-event trigger selection. This feature is not turned on here, and will be used and discussed in the second part of the tutorial, 'Luminosity Calculation'.)

DummyDumperAlg1 is a dummy ntupledumper, writing out the (simple) root file selection1.root. Since DummyDumperAlg1 runs after GRLTriggerAlg1 in sequence ModSequence1, it is not executed in case GRLTriggerAlg1 rejects an event.

Let's set the outputlevel of the GoodRunsListSelectorTool to DEBUG in the GoodRunsListsUser_oneSelection.py script, and run the GoodRunsListsUser_oneSelection.py script by modifying GoodRunsListsUser_jobOptions.py, and then run athena

cd $TestArea/DataQuality/GoodRunsListsUser/run/
athena GoodRunsListsUser/GoodRunsListsUser_jobOptions.py 2>&1 | tee out.log

(During the job, please ignore the many warnings of 'RootClassLoader Info Failed to load dictionary for native class: "int"'.) The output file out.log shows several interesting things. Search for "Event accepted" and you'll see that, for example, all events of lumi-block 806 are accepted. Likewise, seach for "Event rejected" and you'll find that eg. all events of lumi-block 809, or 840 have been rejected. Towards the end of the logfile is an overview of the number of events selected by GRLTriggerAlg1

GRLTriggerAlg1                                       INFO GRL     : Processed 4266 events, accepted 3582 events.
GRLTriggerAlg1                                       INFO Trigger : Processed 3582 events, accepted 3582 events.

The good run list selection has accepted 3582 out of 4266 events. Since the trigger selection has not been turned on, it did not reject any events.

The LumiBlockMetaDataTool has produced the output file luminosity_grl_108175.806-108175.929.xml .
The name of the file corresponds to the first and last run-LB combination processed in the job. Let open this file in your favorite editor to take a look.

Note that the xml file has the same structure as a good run list produced by AtlRunQuery. It contains an overview of all lumi-blocks accepted by the DQ selection, as performed by the GoodRunsListSelectorTool. The GRL object inside the xml file has name LumiBlocks_GoodDQ0, and its metadata shows that it has been created with the query of good run lists _OR_grl_topmix_mu_AND_NOT_blacklist. The trigger selection is set to 1, meaning that no trigger selection has been applied.

Now open the produced dummy ntuple in root:
root -l selection1.root
Cross-check by doing tree->Draw("lumiBlock") or tree->Scan() that the accepted events indeed correspond to the (DQ accepted) lumi-blocks specified in luminosity_grl_108175.806-108175.929.xml .

We will come back to the produced xml files in a later exercise.

Two GRL selections in one job

This section described the recommended procedure for applying several GRL selections in one athena job. For example, one may be interested in applying several trigger selections, both with different good run list selections. The example below demonstrates the use can of two (sets of) selections applied in one an athena job, producing two corresponding flat ntuples with our (dummy) physics events of interest.

Let's take a look at the jobOptions file $TestArea/DataQuality/GoodRunsListsUser/share/GoodRunsListsUser_twoSelections.py .
This jobOptions file looks very much like the jobOptions file for one selection we looked at earlier.

The GoodRunsListSelectorTool reads in the additional xml file grl_topmix_e.xml, containing the good run list object grl_topmix_e. Also, an additional sequence ModSequence2 is added to the top job-sequence, with algorithms GRLTriggerAlg2 and DummyDumperAlg2. GRLTriggerAlg2 selects LBs based on good run list grl_topmix_e, and DummyDumperAlg2 stores the accepted events in the file selection2.root.

Modify GoodRunsListsUser/share/GoodRunsListsUser_jobOptions.py to include (only) the jOs file GoodRunsListsUser/GoodRunsListsUser_twoSelections.py. Now run athena twice. First over only over only the first of the two AOD collection, then over the second of the two AOD collections. (Let's pretend these are two separate grid jobs, producing separate root files and xml output files.)

The two jobs produce two xml files: luminosity_grl_108175.806-108175.867.xml and luminosity_grl_108175.870-108175.929.xml. (In a realistic analysis there will be many more xml files, of course -- one for each job.) Again, the name of each file corresponds to the first and last run-LB combination processed in the sequence.

Let's take a look at the first of these files. Note that there are two NamedLumiRange elements in the file, one labelled LumiBlocks_GoodDQ0, and the other LumiBlocks_GoodDQ1. They correspond to the two selections applied in the two sequences defined in the athena job. Like for the one-selection case, each lists the DQ approved runs and lumi-blocks processed in the sequence. Also, the GRL query and trigger selection for each sequence are described in the metadata.

Let's copy these two xml files to a dedicated directory:

mkdir -p lumi_files
cp -f luminosity_grl_108175.*.xml lumi_files/

We will use them again in a merging example below.

Merging of good run lists

Imagine the scenario where you have run all your (flat) ntuple production jobs. Each job used the 'official' good run list, as provided by the DQ group, to select good runs and lumi-blocks. Each job can have had one or several event selections applied. To make the scenario realistic: unfortunately, some of your jobs terminated with unresolvable crashes, and for others it turned out to be impossible to access the dq2 datasets, so you have not managed to run over the complete dataset. For the luminosity calculation, you need a list of all the runs and lumi-blocks to have actually processed. How to get this list?

One answer is to use the output xml files written out by LumiBlockMetaDataTool (though there are several approaches possible). Fortunately, for each ntuple production job that did succeed LumiBlockMetaDataTool has written out the xml file describing the list of lumi-blocks that has been processed, and you have taken care to retrieve all these files, and to store them properly. These xml files can now be merged. In the example below, we will work with the (previously created) xml files stored in the directory
$TestArea/DataQuality/GoodRunsListsUser/run/lumi_files/
Remember that each of these files contains the lumi-block collections of two selections.

To merge the xml files, simply do:

cd $TestArea/DataQuality/GoodRunsListsUser/run/
merge_goodrunslists lumi_files/

This produces the output:

o) Found 2 input xml files.
o) Now merging ... (this takes some time)
o) Found 2 independent good-runs lists. Now writing merged xml files.
--- TGoodRunsListWriter : GoodRunsList stored as        : merged_0_grl_108175.806-108175.927.xml
--- TGoodRunsListWriter : GoodRunsList stored as        : merged_1_grl_108175.806-108175.929.xml

The script looks for all xml files in the directory lumi_files, and tries to interpret the contents as good run lists. Next, the script tries to merge the lumi-block lists of all matching selections it has found. In this example there are two independent selections (corresponding to sequence1 and sequence2, as defined earlier) found in the xml files. Each merged selection of lumi-blocks is then written out to an independent xml file. These lists can then be used as input to the luminosity calculation (see second half of this tutorial).

As a cross-check, take a look at the contents of merged_0_grl_108175.806-108175.927.xml and confirm that it is identical to the merger of LumiBlocks_GoodDQ0 found in the xml-files in the directory lumi_files/ .

The script also takes care to remove any duplicate lumi-blocks found when trying to merge the lumi-block lists of matching selections. (In practice this should never happen though, and it would be a very bad sign if it did!)

[The script also allows you to merge xml-strings attached to TTrees in flat ntuples. Type merge_goodrunslists for the full command line. See the follow-up CoolLumiCalcTutorial for more details.]

Luminosity Calculation Tutorial

The second part of this tutorial deals with luminosity calculation, where good run lists obviously also play an important role. (The second ingredient is trigger selection.) The luminosity tutorial uses a few of the output xml files that you have already produced here.

IMPORTANT NOTES FOR ANY USER

• Even if you plan to apply the DQ and trigger selections at the ntuple level only, and not in your athena jobs, you should always have the LumiBlockMetaDataTool turned on so that the processed lumiblock collections or xml-strings get propagated to your ntuples!
• NEVER throw away output ntuples with no events! They still contain LumiBlock collections for the luminosity calculation!!
• Merging of the xml-string with the flat ntuple should always be done.
• You should always retrieve the produced luminosity-block xml-files from a grid ntuple job!

Setup for all exercises of luminosity calculation

$ cd $TestArea/LumiBlock/LumiBlockComps/run/

Exercise 1: Using iLumiCalc.exe from command line

Once you have the packages built you can already try out the command line tool iLumiCalc.exe the main purpose of which is to have an "easy" way of running integrated luminosity calculation on a list of lumiblocks. The input can be xml file, ROOT files, or TAG files created by ELSSI (for TAG files please see the TAG tutorial part of the PAT Tutorials).

1) Get the help menu

$ iLumiCalc.exe -h

2) Measure Luminosity from a GoodRunsLists processed TopMix xml files

In the previous GoodRunsLists Tutorial you should have created some xml files out of TopMix AODs. You can ask iLumiCalc.exe to accumulate the luminosity for the list of lumiblocks in the xml file, for a given trigger:

$iLumiCalc.exe --mc -t "EF_mu20" -x merged_0_grl_108175.806-108175.927.xml

$iLumiCalc.exe --mc -t "EF_e20_loose" -x merged_1_grl_108175.806-108175.929.xml

NOTE: the --mc flag needs to be specified to use the Monte Carlo database for TopMix runs!

You can get the full list of options using the -h flag. Here we used the following flags:

• --mc : use Monte Carlo database
• -t : specify the trigger item
• -x : specify the input xml file (which contains the list of run and lumiblock numbers)

But if say you wanted to measure the integrated luminosity of some recent cosmics run (for which only L1 triggers should be used!), then e.g. do:

$ iLumiCalc.exe -t "L1_MU1" -r 134793 --lbstart=0 --lbend=1000

NOTE: in cosmics we don't use the --mc option as we connect to a different database!

If you run this you should get a warning saying that the trigger L1_MU1 trigger item cannot be found, and a hint is given what trigger items exist. Say you pick L1_MU6_EMPTY :

$ iLumiCalc.exe -t "L1_MU6_EMPTY" -r 134793 --lbstart=0 --lbend=1000
...
--- LumiCalculator      : Beginning calculation for Trigger L1_MU6_EMPTY, Run 134793 LB [0-1000]
--- LumiCalculator      :      LumiB    L1-Acc    L2-Acc    L3-Acc    L1-pre    L2-pre    L3-pre  LiveTime IntL/nb-1
--- LumiCalculator      :              1173523         1         1         3         1         1   12429.6   41.4321
--- LumiCalculator      : >== Trigger  : L1_MU6_EMPTY
--- LumiCalculator      : IntL (nb^-1) : 41.4321
--- LumiCalculator      : L1/2/3 accept: 1173523   1         1         
--- LumiCalculator      : Livetime     : 12429.6
--- LumiCalculator      : Good LBs     : 227
--- LumiCalculator      : Bad LBs     : 0
--- iLumiCalc           : --------------------------------------------
...

So now we used the following flags:

• -t : specify the trigger item
• -r : specify the run number
• --lbstart : specify the start luminosity block number of the IOV range of interest
• --lbend : specify the end luminosity block number of the IOV range of interest

Try using the verbosity:

$ iLumiCalc.exe -t "L1_MU6_EMPTY" -r 134793 --lbstart=0 --lbend=1000 -V

You should get a more detailed output per each luminosity block.

Exercise 2: Using LumiBlockMetaDataTool, LumiCalc and GoodRunsListSelectorTool in ATHENA

1) Copy LumiBlock_jobOptions.py and others to your folder

cp ../share/LumiBlock_* .

and also copy the files

cp $TestArea/DataQuality/GoodRunsListsUser/share/LumiBlock_* .
cp $TestArea/DataQuality/GoodRunsListsUser/run/grl_topmix*.xml .
cp $TestArea/DataQuality/GoodRunsListsUser/run/blacklist.xml .

LumiBlock_jobOptions.py is going to be your main jobOptions file that you need to run with athena.py LumiBlock_jobOptions.py command. If you open it you'll see that it contains some general setup. What you should focus on are these lines:

...
# include your algorithm job options here
UserAlgs=[ "LumiBlock_LumiCalc_GRL.py" ]

# the Input AOD File
athenaCommonFlags.PoolAODInput = [
"dcache:/pnfs/usatlas.bnl.gov/LOCALGROUPDISK/user/RichardHawkings/user.RichardHawkings.0108175.topmix_Muon.AOD.v4/user.RichardHawkings.0108175.topmix_Muon.AOD.v4._00014.pool.root",
"dcache:/pnfs/usatlas.bnl.gov/LOCALGROUPDISK/user/RichardHawkings/user.RichardHawkings.0108175.topmix_Muon.AOD.v4/user.RichardHawkings.0108175.topmix_Muon.AOD.v4._00015.pool.root"
]
...

The UserAlgs... line just includes the LumiBlock_LumiCalc_GRL.py python job fraction that you have just copied to your folder. That contains the core functionality.

Open LumiBlock_LumiCalc_GRL.py and investigate its contents:

# add LumiBlockMetaDataTool to ToolSvc and configure
from LumiBlockComps.LumiBlockCompsConf import LumiBlockMetaDataTool
ToolSvc += LumiBlockMetaDataTool( "LumiBlockMetaDataTool" )
LumiBlockMetaDataTool.calcLumi = True # False by default
LumiBlockMetaDataTool.storeXMLFiles = True
LumiBlockMetaDataTool.applyDQCuts = True
LumiBlockMetaDataTool.OutputLevel = INFO

# add ToolSvc.LumiBlockMetaDataTool to MetaDataSvc
from EventSelectorAthenaPool.EventSelectorAthenaPoolConf import MetaDataSvc
svcMgr += MetaDataSvc( "MetaDataSvc" )
svcMgr.MetaDataSvc.MetaDataTools += [ ToolSvc.LumiBlockMetaDataTool ]

# Configure the goodrunslist selector tool
from GoodRunsLists.GoodRunsListsConf import *
ToolSvc += GoodRunsListSelectorTool()
GoodRunsListSelectorTool.OutputLevel = INFO
GoodRunsListSelectorTool.GoodRunsListVec = [ 'grl_topmix_mu.xml '] # or grl_topmix_e.xml
GoodRunsListSelectorTool.PassThrough = False

# add LumiCalcSvc to ServiceMgr and configure
from LumiBlockComps.LumiBlockCompsConf import LumiCalcSvc
LumiCalcSvc = LumiCalcSvc()
LumiCalcSvc.Triggers = ["EF_mu20"]
LumiCalcSvc.UseMC = True
LumiCalcSvc.LBCollNames = ["LumiBlocks", "IncompleteLumiBlocks"]
LumiCalcSvc.Verbose = False
svcMgr += LumiCalcSvc

Discussion:

• Here LumiBlockMetaDataTool is [1] instantiated, [2] added to ToolSvc and [3] added to MetaDataSvc. So now during the job this services lives on and does the bookkeeping of the luminosity block in your input AOD/ESD/etc file(s). This is done in the following way:
  • Each time a new input file is opened: the list of luminosity blocks from the file is stored locally
  • Each time a file is ended properly: the list of locally stored luminosity blocks is added to the locally stored luminosity blocks from the previous file(s) and stored in an object called LumiBlocks
  • Each time a file is ended improperly: the list of locally stored luminosity blocks is stored in a special object called IncompleteLumiBlocks

• Then GoodRunsListSelectorTool is [1] instantiated, [2] added to ToolSvc and configured. This Tool is called from within LumiBlockMetaDataTool and this selects only those lumiblocks from the input file that are included in the input grl2.xml good runs lists.

• Then LumiCalcSvc is [1] instantiated and [2] added to ServiceMgr. LumiCalcSvc is called from within LumiBlockMetaDataTool after all the luminosity block bookkeeping and good runs lists filtering has finished. Then LumiCalcSvc calculates the integrated luminosity in a way it is described in this talk. The input parameters that need to be set for an integrated luminosity calculation in general are:
  • Trigger name (in this case EF_mu20): optional, used only for this example, later will be overwritten by
  • Which luminosity block list to use (in this case both LumiBlocks and IncompleteLumiBlocks, but calculation is done separately)
  • What kind of Luminosity are you using (Monte Carlo, Offline or Online - in this case we use TopMix Monte Carlo therefore UseMC=True is set)

2) Now it's time to run the job

$ athena.py LumiBlock_jobOptions.py

Once it is finished you should see towards the end of the job something similar to this:

ToolSvc.LumiBlockMetaDataTool                        INFO  stop method: Event Loop Completed 
--- TGoodRunsListWriter : GoodRunsList stored as        : luminosity_grl_108175.806-108175.929.xml
LumiCalcSvc                                       WARNING Could not find LumiBlockCollection >>IncompleteLumiBlocks<< in MetaDataStore
LumiCalcSvc                                          INFO  Calculating Integrated Luminosity based on >>LumiBlocks<< in MetaDataStore
...
--- LumiCalculator      : IntL (nb^-1) : 649.002
...
LumiCalcSvc                                          INFO  Calculating Integrated Luminosity based on >>LumiBlocks_GoodDQ0<< in MetaDataStore
LumiCalcSvc                                          INFO  Description of >>LumiBlocks_GoodDQ0<< = luminosity_uniquegrlselector __ GRLQuery = _OR_grl2
...
--- LumiCalculator      : IntL (nb^-1) : 240.333
...

What you can see is the following:

• First LumiBlockMetaDataTool reports that the Event Loop has Completed and puts all the bookkept (complete and incomplete) luminosity blocks into MetaData store
• Then GoodRunsLists
  • picks up these luminosity blocks from MetaData store and filters them based on the input grl2.xml file,
  • puts the filtered luminosity block into MetaData store with a _GoodDQx postfix in the names,
  • converts them and copy them to the output file luminosity_grl_108175.806-108175.929.xml
• LumiCalculator class connects to COOL database for Trigger and Luminosity information
• Integrated luminosity calculation is done for each continuous IOVRange, and the total integrated luminosity is accumulated and reported

Note that 2 integrated luminosity values are reported:

• one for the full completely processed list of luminosity blocks (LumiBlocks)
• and the other for the !DataQuality filtered luminosity blocks (LumiBlocks_GoodDQ0)

3) Use iLumiCalc.exe to re-measure integrated luminosity of the filtered Luminosity Blocks

Just do:

$ iLumiCalc.exe --mc -t "EF_mu20" -x luminosity_grl_108175.806-108175.929.xml
--- iLumiCalc           : Trigger(s): EF_mu20, 
--- iLumiCalc           : Being in XML file mode...
--- iLumiCalc           : Processing file: 
--- iLumiCalc           : --------------------------------------------
--- LumiCalculator      : Luminosity database: COOLOFL_TRIGGER/OFLP200
--- LumiCalculator      : Trigger database: COOLONL_TRIGGER/OFLP200
--- CoolQuery           : Trying to connect to database COOLOFL_TRIGGER/OFLP200...
Data source lookup using /afs/cern.ch/atlas/software/builds/AtlasCore/15.5.0/InstallArea/XML/AtlasAuthentication/dblookup.xml file
--- CoolQuery           : Trying to connect to database COOLONL_TRIGGER/OFLP200...
--- LumiCalculator      : Beginning calculation for Trigger EF_mu20, Run 108175 LB [806-806]
...
--- LumiCalculator      : Beginning calculation for Trigger EF_mu20, Run 108175 LB [809-821]
...
--- LumiCalculator      : Beginning calculation for Trigger EF_mu20, Run 108175 LB [824-836]
...
--- LumiCalculator      : Beginning calculation for Trigger EF_mu20, Run 108175 LB [839-850]
--- LumiCalculator      :      LumiB    L1-Acc    L2-Acc    L3-Acc    L1-pre    L2-pre    L3-pre  LiveTime IntL/nb-1
--- LumiCalculator      :                 6427      6427      6427         1         1         1      2340   240.333
--- LumiCalculator      : >== Trigger  : EF_mu20
--- LumiCalculator      : IntL (nb^-1) : 240.333
--- LumiCalculator      : L1/2/3 accept: 6427      6427      6427      
--- LumiCalculator      : Livetime     : 2340
--- LumiCalculator      : Good LBs     : 39
--- LumiCalculator      : Bad LBs     : 0
...

We get the same integrated luminosity as the one reported by LumiCalcSvc in the Athena job for the Data Quality filtered luminosity blocks.