Comments on PPDG RDL 0.40
David Adams
November 26, 2003

Responses are ordered by section number in the document.

This document is a work in progress. Comments on the comments
should probably refer to the above date.

3.2 Basic concepts

3.2.1

I am one of "some people". For analysis activities it is essential
to merge results from sub-jobs and carry the merged result back to
the user. This needs to be included in the specification.

Also missing is the means to check on the status of a result: is
it running, what fraction is done, etc? We should not stop at a
request definition language but also include means to monitor
request status and results. DIAL does this with the Job and Result
classes (with XML representations).

3.5 Applications

3.5.1
The current DIAL application interface has two entry points: task
installation and task execution. The installation may extract files
and use them to create other files, typically compiling and linking.
Verification naturally occurs in this step. The execution may make
use of the files extracted or produced in the installation and may
also directly access the task.

I agree that the application must specify the packages on which it
depends. This and the problem of application installation can be
handled by making the application a package and using a package
management system to handle installation and dependencies. The
data in the application XML can then be the corresponding package
name and version.

I don't think the application should know any details about
retrieving either task description or data files. Instead it should
expect each site will provide a service that can handle replication
of whatever type of logical files appear in the task or dataset.

Similarly the application should not be responsible for splitting,
matchmaking or any other job management tasks. In DIAL, these are
handled by the scheduler.

The application is responsible for handling the (sub-)dataset it
receives for processing. This and the task are the input for the
execution step.

Although DIAL does not do this at present, one might also require
the application provide an entry point for merging results.

3.5.4
Does missing "ver" mean any version is acceptable?

3.5.5
I am happy with the name application but I think we need to clearly
specify the entry points (e.g. installation and execution in the
current DIAL model).

3.5.6
If the application is a package and has well-defined entry points,
then the job management system can query the package management
system for the information needed to construct low-level JDL.

3.6 Tasks

3.6.2
Rather than assigning task names, I assign a unique ID so the task
can be registered and referenced in a provenance catalog.

I have found it sufficient to make the task a collection of named
files where the files may be embedded in the XML or be references to
physical or logical files. The application using the task expects
to find a service that can instantiate physical replicas of these
files.

One can make a case for carrying named parameters but these can always
put these in a file.

3.6.4
I would replace <script>, <macro>, etc with <file>. The tag <classpath>
might be replaced with <file name=classpath.dat>...</file> with the
ellipses replaced by the classpath definition in suitable format.

3.6.6
I agree that a task might be shared by multiple applications. This is
the reason that the task does not hold the application. Note that
the provenance of any result requires both application and task.

3.7 Datasets

This is by far the most complex element. Please see my note
"Datasets for the Grid" at http://www.usatlas.bnl.gov/~dladams/dataset.
I think we should include much more information in a common way
that can be shared by different realms.

DIAL envisions a hierarchy of schedulers where a job request made
up of application, task and dataset may be passed from scheduler
to scheduler. In this request flow, the application and task do
not change but the dataset may. The dataset splits when the job
splits and typically it becomes more tangible as it moves down
the chain, e.g. virtual is converted to logical is converted to
physical with splitting at on one or more points in between.

The scheduler does not treat the dataset as a black box. The scheduler
is responsible for splitting the dataset, matchmaking and for staging
files.  I believe we can do better than simply assuming that each
realm plugs in its own splitter for each type of dataset. Instead the
dataset should have enough common information that splitting can be
done with a generic splitter.

3.8 Extras

Rather than extras, I would like to see something like job configuration
and I do expect to see elements shared across realms. These include
fields such as identity, role, response time, maximum cost, etc.

3.9 Extension mechanisms

I believe the measure of success of a common RDL will be the ability
of users from different realms (e.g. different experiments) to process
requests using a common request scheduler. It should also be possible 
for different realms to share common applications such as ROOT.