Minutes: U.S. M4 Data Analysis Meetings

<

Minutes, September 5, 2007

A very slim gathering over the phone today:
KevinB, RobertH, JohnH, HongMa, WalterL, SriniR

1) Event Displays from Kevin are posted on the Wiki page. He has been 
able to see muon tracks after opening up the errors on hits by x10.

2) muon t0's are now in the production DB and should have been replicated 
to BNL. Kevin to check on that.

3) Robert has been able to run the MuonCosmicValidator on the raw data files. 
These produces too many histograms and he will try to look at summarizing 
the many histograms. He will also look into setting up cron jobs that detects 
new raw data files appearing at T1 and running the validator automatically.

4) Walter reports that a stable patch for M4 cosmic run will probably come 
out Friday morning. We should have this installed at BNL asap and plan on
re-reconstruction with the full M4 data sample using PANDA. One hopes that 
the checksum problem will be fixed by then
Kaushik - can you keep an eye on that please and let us know so that we can 
plan the re-reconstruction at BNL accordingly.

5) Walter reports that the last data file for M4 is:
"as far as I can see from the ATLOG (https://pcatdwww.cern.ch/elog/ATLAS/ATLAS/) 
and castor (/castor/cern.ch/grid/atlas/t0/perm/DAQ) the last files for M4 was:
mrw-r--r--   1 atlascdr zp                 89433212 Sep 03 10:04
daq.m4_combined.0020919.debug.L1TT-b00000100.LB0000.SFO-6._0104.data
"
Alexei - Can you please check if all the RAW data files are replicated to BNL.

6) So where do we go from here?
Since the M4 data run is over, we propose NOT having daily meetings. The proposal 
is to have weekly meetings to follow up on the following (and a longer term goal 
of preparing for M5):

A) Re-reconstruct the M4 data at BNL.
B) Analyze the re-reconstructed data at the T1/T2 centers rapidly.
C) Prepare ourselves for the M5 run. For the M5 run -- we would like to have the
goal of providing real time monitoring from T1/T2 -- to be able to give feedback 
to ATLAS on data coming in asap (< 24 hours). This needs some amount of 
preparation (which we did not have for M4):
-- Make sure that all the infrastructure software works allowing us to replicate 
and re-reconstruct the data. Replication includes the ability to replicate 
conditions data quickly.
-- Form a coherent team with participation from all sub-systems.
-- Establish the sub-system software that allows us to monitor the data coming 
to T1/T2 and provide rapid feedback.

Getting a lot of this to work needs input from ATLAS itself -- and some of us 
will follow up on that. But I would like to push ourselves toward the goal of 
being able to perform remote monitoring -- this exercise will allow us to 
understand whether this is feasible during the real data taking phase. It 
will also require input from the sub-systems to be able to setup the software 
to monitor and have it ready -- We should start doing this now and test it on 
the M4 data, so US sub-system participation is also essential.

We will closely follow the above points at our weekly meetings -- which is 
suggested for Wednesday's at 12:30 PM. Next meeting on September 12. If you 
are interested in participating, please let me know. I will try to form a 
mailing list for this, so if you are not interested, you can un-subscribe.

7) We have to write a report on our experience. I will draft a report and 
send it out to this community for feedback. Once this is completed, Jim Shank 
will forward our feedback to Kors Bos and Fabiola.

Thank you all for participating in this -- Though not very successful, we 
have been able to find what the bottlenecks are, which itself is a useful 
exercise. I hope you will participate in M5 preparation.

Minutes, August 31, 2007

1) Software releases:
 -- 13.0.25.7 at BNL.
 --  Muon cosmic validator bug reported by Robert resulting in no endcap 
     muons. Requires jobOptions change, which has Robert has sent out.
 -- If patches become available over the weekend, Shuwei will promptly install 
    them at BNL.

2) Reconstruction
Still problems running pathena. Hong reports that the checksum in the file 
catalog may not be correct and pathena/panda requires this.

Alexei reports that T0 is not registering the file with a correct md5sum in 
the DQ2 catalog. There is an ongoing thread to resolve this problem. Apparently 
it looks like the md5sum is not available (because it is not provided by the 
TDAQ system) at the time of registering the file in the DQ2 catalog. Hence the 
T0 creates a dummy md5sum with the hope of updating it at a subsequent processing 
stage - but not clear if this is done. Anyway, pathena or panda is unable to run 
until the DQ2 catalog has a proper md5 sum.

3) Data Replication:
   --Alexei reports that raw data replication to T1 and ESD replication to T1 and 
T2 is ongoing smoothly.

Latest report from Point 1:
Triggering on Tiles were fixed yesterday morning and hopefully the datasets 
arriving today are good. These are being checked by the various sub-systems.

4) Sub-System analysis:
 TRT: Most of the current analysis based on TRT monitoring on lxplus. Evelyn 
trying to analyze datasets at BNL.
 LAr analysis:
   Fabien tried to work with pathena, now running reconstruction locally.   
 Tile: 
   Carlos Medinas has started to look at the data
 Muons
    Problems getting access to t-zero constants. The t0's are in the development 
database and needs to be propagated to the production DB. If this is done, then 
theoretically this should be replicated at the T1 immediately. It is not clear 
if the t-zero's are in the prod-db. Part of the problem is that the dev-db is 
down. As a backup, Kevin has now obtained the tzero's in ascii format and is 
able to run reconstruction on raw data. He will continue this approach until 
the t-0's become available in prod-db.

General analysis comments:
  -- Please try to look at Event Displays.
   -- Try to develop an analysis software (CBNT) that can run on ESD and produce 
ntuples with typical quantities. We can then use the same software when 
re-reconstructing from RAW. This is the fastest way to provide quick feedback on 
data coming in.

AOB
  -- Next meeting on Monday at 12:30
  -- Chat room (m4atlas) will remain open during the weekend.
  -- Please e-mail questions or results.

Urgent Action Items:
   -- Need to followup with CERN to address the md5sum in DQ2 catalog
   -- Develop an analysis software that can run on ESD and plot basic observables 
to validate the data.

Minutes, August 30, 2007

-- 12.0.25.6 released at BNL.
-- Walter reports bugs in 13.0.26:
 -) some weired histograms to show up in the ntuple-root file. Not understood why
but have workaround.
 -) There is also a bug in the reconstruction of LAr-data with only 5 samples.
Thanks to Denis for spotting this!
(Both bugs are fixed in the coming AtlasPoint1 nightly.)

-- Tadashi has pAthena successfully running on M4 data. Fabien is trying to 
exercise it.

-- Waiting for ATLAS to put out a stable AtlasProduction cache for submitting 
grid jobs. Once this is available, we will consider re-reconstructing the M4 
data sampleat T1.

-- Datasets are now being replicated at T0 and T1. Hiccups at T0 end often 
slowing this down. As of yesterday, no datasets for Run 20505 and up were 
defined.

-- Timing issue in Calorimeter fixed. However, Tile triggers turned off 
accidentally during yesterday's cosmic run. Hence only RPC triggers. Hopefully 
will have better
runs today.

-- Analysis activities ongoing:
  I have started to upload analysis plots from M4 on the Wiki page.
 * Fabien starting to look at LAr data.
 * Carlos Medinas from UTA identified to look at Tile.
 * Evelyn Thomson sent around TRT monitoring plots. Penn group to try and 
look at the data at T1. Beate Heinemann will also be looking at the TRT data, 
hopefully addressing the integration between TRT and other sub-systems.
 * Kevin Black and Robert Harrington actively looking at the muons. The t0 
constants for the muons are not available in the production DB at T0 (only 
in a development DB) and hence not replicated to T1. They are in contact 
with Ed Diehl and others to resolve this issue.

Minutes, August 29, 2007

Meeting time:
It was decided that we meet daily at 12:30 PM EDT (18:30 CERN time), 
this timing does not conflict with major atlas meetings during the 
week -- Next meeting, Thursday, August 30 at 12:30 PM EDT.

- More information is now available on the Wiki page.
  http://www.usatlas.bnl.gov/twiki/bin/view/AtlasSoftware/M4DataAnalysis
- 13.0.25.5 software release has been installed at BNL by Shuwei.
- Hong/Tadashi working on using pAthena on M4 data.
- Kaushik in touch with David Quarrie about getting a stable 
production patch in AtlasProduction. DQ writes:

> I now prefer a model where you run a single patch cache (which I was
> going to call AtlasProduction), and insert patches at a high rate for
> Point1, accepting them only infrequently for deployment. Thus
> 13.0.25.1, 13.0.25.2....13.0.25.8 might be just for Point1, then
> 13.0.25.9 might be deployed on the GRID as well as at Point1, then
> 13.0.25.10 through 13.0.25.15 might be just for Point1 etc.


This is good strategy. We should anticipate re-running reconstruction 
over the entire data sample at T1 when a stable patch comes through.

- Alexei reports that data replication is ongoing, ESDs now becoming 
available at T2.
- Walter reports that all Calo runs taken till today are useless 
because of a timing issue. This has been fixed, hopefully the run 
tomorrow will be good.
- Kevin in the process of identifying good muon runs.

Analysis Focus:

Brig Williams sent out a plot showing TRT tracks in M4 data yesterday. 
This is good and is the type of feedback we need. It would be useful 
to put added focus to the M4 analysis activities in U.S. The 
suggestion is two-fold:

1) To be able to analyze the data (either RAW or ESD) via specialized 
analysis or monitoring software to ensure that individual sub-systems 
are able to reconstruct tracks, clusters and other essential quantities 
and to further ensure that the data looks reasonable.  Part of the 
objective here is to do this within 24 hours at the T1 since the data 
is made available at T0.

2) Once each sub-system can sign off individually, we integrate the 
information connecting muons, calo and track and provide some 
quantitative feedback on this. Here it may help if we can agree 
on a common list of variables that each system could write out to 
the ntuples and we can try to do the integration at the ntuple level.

As a first step, the muons (Kevin, Robert, et. al.) and LAr (Hong, 
Fabien, et. al.) will come up with a list of observables and add 
them to the CBNT ntuple. They will also list the issues that is 
preventing them from doing this exercise - As an example, Kevin 
pointed out that the t0's needed for muon reconstruction are not 
yet in the database. Actively work on resolving the issues that will 
allow us to extract the observables and validate the data. Then work 
on step (2). Lar and Muons will give more details tomorrow. TRT and 
Inner Detector not at the meeting and someone from Tile has to be 
identified. 

Minutes, August 28, 2007

Brief minutes of today's meeting, next meeting tomorrow at 11:30 AM EDT.

13.0.25.4 installed at BNL -- see Wiki page for details.

>> Hong/Tadashi to try and get pAthena working at BNL. needs fix for 
AtlasPoint1.

>> To contact Pavel and see if he call help with the transformations 
needed to run panda production jobs.

Alexei reported that --
the 300 queued RAW datasets were because of a wrong subscriptions from 
Tier0 team. Datasets are now correctly re-subscribed and replication 
should begin soon.
For the ESD -- some of the datasets are now defined at T0, some 
replication already at BNL.

>> Alexei working on one establishing dataset at the DAQ run level, 
should have something in a couple of days.. The current dataset is 
established at sub-run level corresponding to a 30 minute running 
period. This results in very few events/file -- sometimes none! 
This is partly being resolved by changing the DAQ setting - so 
that at there are at least 20-30 events/file.

Distribution of ESD to T2 can start as early as tomorrow.

Analysis:

-- Need to identify which data files are good.
Walter reported that many reconstruction jobs still fail in muon 
reconstruction because it cannot handle corrupt events. Muon group 
reportedly working on this problem.
SCT ByteStream converter problem may still exist.

>> Walter Lampl and Kevin Black to identify useful data files and post 
on Wiki,

>> Hong has a software script running to look at M4 data. Fabien will 
exercise this. (however needs to turn off muon reconstruction currently). 
Hong to this on Wiki.

>> Brig and Penn group investigating a L2 TRT track trigger.

Muons: Kevin is identified as a point person for M4 analysis.
TGC are not operating yet, only RPC/barrel (and some endcap) events available.

>> Kevin to identify the fraction of good RPC triggers. Brig/Penn will 
investigate the double-Tile trigger - which have a higher quality, but poor 
statistics.

Srini has setup a Wiki page. Please follow Wiki page for detailed information.
http://www.usatlas.bnl.gov/twiki/bin/view/AtlasSoftware/M4DataAnalysis

>> Jim has sent out instructions for using AIM chat room. However, a 
couple of issues identified with this chat room -- a) Cannot join the 
chat room unless you are invited, this is painful as people will likely 
disconnect several times, b) Cannot look at the history of conversation 
after joining, can only see the history since being connected. Jim/Rob 
looking at alternatives.

Minutes, August 27, 2007

1) Shuwei to get the patches installed at BNL as they come out. 13.0.25.4 
should be out tomorrow. Walter to inform Shuwei of any new patches.

2) Hong/Tadashi to exercise and test pAthena on M4 raw data files at BNL.

3) Walter to send around pointers to location of CBNT and monitoring 
histograms at Tier0

4) Alexei to ensure all raw datasets are available at BNL. Around 300 
are in the queue and 100+ have already been replicated.

5) Alexei/Pavel to check with Luc/Armin on the availability of ESD 
datasets. Walter to point us to the location of ESD datafiles in castor.

6) Alexei to create datasets at a [coarser] DAQ run level - Present 
datasets at T0 are in 30 minute running interval.

7) Alexei/Pavel to replicate comcond.000002.lar_conditions.recon.pool.v0000 
dataset for LAr conditions needed for M4 reconstruction.

8) Other subsystems to check if relevant conditions data is available, 
if not we have to ensure they are replicated.

9) All sub-systems to report on their plans for analyzing M4 data sample 
at BNL. We have so far:
LAr (Hong, Fabien, Walter)
TRT (Brig and U Penn colleagues)
Muons (Stephane, Kevin Black, Bob Harrington). Huth to check with 
Stephane on whether we can get participation from Michigan.
Inner Detector Beate Heinemann.

10) Srini to setup a US ATLAS wiki page where all relevant information 
will be maintained.

11) Jim to send around instructions for using AIM chat room.