Subject:
Re: event data history
From:
Torre Wenaus <torre@wenaus.com>
Date:
Mon, 8 Oct 2001 15:30:17 -0400
To:
David Adams <dladams@bnl.gov>
CC:
Srini Rajagopalan <srinir@bnl.gov>, Hong Ma <hma@pony.phy.bnl.gov>, Pavel Nevski <nevski@bnl.gov>, Yuri Fisyak <fisyak@bnl.gov>, Alex Undrus <undrus@bnl.gov>, Valeri Fine <fine@bnl.gov>, Wensheng Deng <wdeng@bnl.gov>

Hi David,
This looks good. Some comments...

I think it could be clearer what job configuration information needs
to be part of the history. Not only the run time parameters for
the algorithm responsible for generating the data of interest are
needed, but also the specification and configuration of the full
chain of algorithms in the job producing the data (ie. the job
options file or equivalent), in case there are intermediary algorithms
in between the parent EDO and the algorithm generating the data of
interest which have an effect.

In general I think the history will need to include a prescriptive
(re)generation procedure for the data. Most of the time this will
simply be 'run Athena with the specified input and configuration', but
more generally the prescription can be more involved. Amounts to the
inclusion of the equivalent of a job script.

Another thing we may want to put in the history (or in the catalogs
referenced by the history) is cost estimation for the processing
required for data generation, to keep track of processing times, and
in 'virtual data' land to be able to estimate in advance the cost of
a virtual data generation request.

As you say,


> With the exception of the parent EDO's, this history information
> may be common to (and shared by) EDO's from many events. 


I think the natural granularity for much of the history information
(including the items above) will be the job level.

  - Torre

David Adams (dladams@bnl.gov) wrote on Thursday 2001/10/4 16:20:
 > All:
 > > I have been thinking a lot about event data history since we started 
 > discussing it last week and I have written down a set of requirements.
 > Please take a look at
 > > http://www.usatlas.bnl.gov/~dladams/data_history
> > Please let me know what you think. I have some ideas about design but I 
 > would first like to establish the requirements.
 > > Should we meet to discuss this, circulate it more widely or delete it?
 > > da
 > > -- 
 > David Adams                      desk: 631-344-6049
 > Brookhaven National Lab          fax: 631-344-5078
 > PAS group, Building 510A         email: dladams@bnl.gov
> Upton, NY 11973-5000             http://www.usatlas.bnl.gov/~dladams
> > 
-- -- Torre Wenaus, BNL Preferred: torre@wenaus.com Or: wenaus@bnl.gov -- ATLAS/LHC 631-344-4755 Fax 631-344-4206 -- B. 510A Room 1-220 http://www.wenaus.com/torre.html