DDM/DQ2 functionality enhancement (proposal). Dan Schrager. Aug 18, 2006


 The proposed  data transfer model follows ATLAS Computing TDR and DDM/DQ2 design 
considerations and policy.

 DQ2 high level policies remain unchanged. DQ2 topology decides which
sites are "close"  and which "highways" should be used for dataset
subscriptions.

 The transferred data will be registered in DQ2 and local files
catalogues and available via DQ2 client tools. The real networking usage
will be reduced, because the  proposed scheme will eliminate duplication
of data transmission to the sites (currently it is quite often, because of wide 
usage of globus-url-copy instead of DQ2 subscription mechanism).


Testing the new model functionality can be done with the current FTS/DQ2 
software and minor modification in TiersOfAtlas (ToA) for BNLTAPE and 
University Chicago (UC, UC_TP, UC_VOB) sites. 

The requested changes:

1) In both the FTS configuration files and in ToA two catch-all clauses 
can be added for each participating site.

2) The LFC python bindings and the lcg-cp command can be added to a 
site's DQ2 service installation.

As a result the participating US ATLAS T2 sites will be able to subscribe 
datasets among themselves and also from any European site. The European 
sites have natively the capacity to subscribe US ATLAS datasets.

Technicalities :

  The proposed changes regard only the low transport level, the FTS level. DQ2 will 
remain the main user of FTS servers.

 To better enforce the FTS load management which right now happens at 
 predefined channels level, I suggest the following:

 An FTS server will manage N storage endpoints grouped on its list of 
 serviceable endpoints.
 There will be only rule: any transfer request (glite-transfer-submit) 
 addressed to the FTS server will be served as long as at least one of 
 the two endpoints involved in the transfer request is listed by the 
 FTS server.

  The transfer requests will come from DQ2 - according to DQ2's high 
 level policies. But not only from DQ2. If everybody will use 
 glite-transfer-submit instead of the basic globus-url-copy then the 
 FTS server will enforce load management policies for all storage 
 endpoints serviced.

  The FTS server will manage the transfer loads for each serviced 
 endpoint in terms of both reading and writing at configurable levels 
 per endpoint (ex. number of simultaneous transfers per site).

   The FTS server will be also able to implement default transfer 
 priorities (beside explicit ones): for example, a transfer between two 
 endpoints both listed will have a higher priority then a mixed 
 transfer (where one endpoint is not listed).

   The FTS transport level could also serve as default choice criteria 
 for DQ2 when more than one endpoint candidate results from DQ2's high 
 level decisions (insufficient refined DQ2 topology or not used - in 
 dq2.py). For example, the endpoint that is listed with the same FTS 
 server will be preferred. Or, same class of FTS capable endpoint will 
 be preferred (both FTS capable or both non FTS servers).


P.S. The necessary add-ons are indeed described in my dq2.sh DQ2
push-button installation script, available at:
http://indico.cern.ch/materialDisplay.py?contribId=10&materialId=0&confId=4897
Just add the catch-all clauses for the participating sites in central
ToA so that automatic DQ2 mechanisms will update the local ToA caches.