r1 - 16 Apr 2008 - 04:57:49 - RobertGardnerYou are here: TWiki >  Admins Web > DQ2SiteServicesP4

DQ2SiteServicesP4

Introduction

Previous phases:

Goals for Phase 4 (Jan 1 - Mar 31, 2008)

  • Upgrade site services to DQ2 0.6 Delayed to Phase V.
  • LFC integration, as needed.
  • Interim upgrades to LRC including schema changes, etc, as needed.

Adler32 Upgrade (2/15/08)

Before doing anything else, stop dq2 (using either an init script or dashb commands), and make a backup copy of your LRC. Note that dq2 site services, the MySQL? database backing the LRC, and the LRC WWW interface might be on separate hosts, here referred to as dq2host, lrchost, and wwwhost.

dq2host# /etc/init.d/dq2 stop
wwwhost# /etc/init.d/apache stop
lrchost# mysqldump -u dq2user -p[password] localreplicas  > lrc_backup-`date --iso`.sql

Now create 2 new columns in the database:

lrchost# mysql -u dq2rw -p[password] localreplicas
    -> insert into t_metaspec values ('adler32', 'string');
    -> alter table t_meta add adler32 blob default NULL;
    -> quit;

Three files that are part of the LRC WWW interface need to be modified to add adler32 support. These are lrc.py, lrcIPC.py and LrcApi.py. You can simply replace the three files with the versions here. Note that if you replace the files, you must correctly set siteid in lrcIPC.py

This is also a good time to upgrade dq2 site services.

dq2host# apt-get update
dq2host# apt-get install dq2-siteservices=0.5.4-1
Also, make sure that there are no references to dashb-atlas-data-producer.cern.ch in dq2-agents-subscription.cfg. The hostname should be dashb-atlas-data.cern.ch.

Now, you can restart dq2 and the WWW interface

dq2host# /etc/init.d/dq2 start
wwwhost# /etc/init.d/apache start

To test, we will create a fake LRC entry for a non-existent file, with an adler32 checksum. You should be able to do the test from any host which can access the LRC. You need the uuidgen program to generate the guid, but this should be installed on most Linux hosts.

host# uuid=$(uuidgen)
host# curl -d "pfns=testLRC_Adler32.1&lfns=testLRC_Adler32.1&guids=$uuid&fsizes=111&md5sums=abcde&archivals=V&adler32s=12345" http://[WWWHOST]:8000/dq2/lrc/files

Now try fetching the LRC record for this file:

host# curl "http://WWWHOST:8000/dq2/lrc/PoolFileCatalog?guids=$uuid"

If the output contains the line

 <metadata att_name="adler32" att_value="12345"/>

then you have succeeded.

Note that for historical reasons, some sites may have modified versions of the WWW LRC interface, and some modifications to the above procedure may be required. These notes may be of some use.

You might also want to clean up the test LRC entry:

lrchost# for db in t_pfn t_lfn t_meta; do 
    echo delete from $db where guid=\"$uuid\" \; | mysql -u dq2rw -p[PASSWORD] localreplicas
done

Testing a site

  1. Registration
    1. There are 7 parameters you can specify
      1. pfns=abc://xyz1+abc://xyz2+.... where xyz1 is physical file name
      2. lfns=abc1+abc2+... where abc is logical file name
      3. guids=gids1+gids2+... where gids is guid
      4. fsizes=fsize1+fsize2+... where fsize is the file size in byte
      5. md5sums=md5a+md5b+... where md5 is the md5sum of the file (NOTE: This is not optional. However, you can pass empty string for entire list to ignore entries.
      6. archivals=av1+av2+... where av is the archival bit of file (V, P, ...)
      7. adler32s=ad1+ad2+... where ad is the adler32 checksum of the file (NOTE: This is optional. If you don't pass, it will set to '')
    2. Registration is done via http post to http://LRC:8000/dq2/lrc/files
      1. You can use curl to place values. However, you must escape charaters, eg. + --> %2B, : --> %3A, /--> %2F, etc.
      2. General command (NOTE: "+" is not escaped for viewing purpose only. You must escape to use it.)
        1. curl -d "pfns=pfn1+pfn2+..." -d "lfns=lfn1+lfn2+..." -d "guids=gid1+gid2+.." -d "fsizes=fsize1+fsize2+..." -d "md5sums=md51+md52..." -d "archivals=av1+av12..." -d "adler32s=ad1+ad2+..." http://LRCHOST:8000/dq2/lrc/files
        2. You can skip adler32s since they are optional. The following is fine.
          1. curl -d "pfns=pfn1+pfn2+..." -d "lfns=lfn1+lfn2+..." -d "guids=gid1+gid2+.." -d "fsizes=fsize1+fsize2+..." -d "md5sums=md51+md52..." -d "archivals=av1+av12..." http://LRCHOST:8000/dq2/lrc/files
        3. You can pass '' to md5sums if you don't want to set md5sums
          1. curl -d "pfns=pfn1+pfn2+..." -d "lfns=lfn1+lfn2+..." -d "guids=gid1+gid2+.." -d "fsizes=fsize1+fsize2+..." -d "md5sums=" -d "archivals=av1+av12..." http://LRCHOST:8000/dq2/lrc/files
    3. To test your service, you can do the following.
      1. single file registration
        1. all parameters are specified:
          1. first generate a unique id (guid) via uuidgen . In this example, it is called gid. (NOTE: Again, make sure to escape characters. However, you probably don't need to escape / chracter. )
          2. curl -d "pfns=test%3A%2F%2Ftest%2FMyLrcTestAtYourSite.1" -d "lfns=MyLrcTestAtYourSite.1" -d "guids=gid" -d "fsizes=1" -d "md5sums=2" -d "archivals=V" -d "adler32s=3" http://LRCHOST:8000/dq2/lrc/files
        2. skip adler32s
          1. do the same as above to get guid via uuidgen (make sure to change pfns and lfns.
          2. curl -d "pfns=test%3A%2F%2Ftest%2FMyLrcTestAtYourSite.2" -d "lfns=MyLrcTestAtYourSite.2" -d "guids=gid" -d "fsizes=1" -d "md5sums=2" -d "archivals=V" http://LRCHOST:8000/dq2/lrc/files
        3. pass empty string to md5sums
          1. do the same as above to get guid via uuidgen (make sure to change pfns and lfns)
          2. curl -d "pfns=test%3A%2F%2Ftest%2FMyLrcTestAtYourSite.3" -d "lfns=MyLrcTestAtYourSite.3" -d "guids=gid" -d "fsizes=1" -d "md5sums=" -d "archivals=V" -d "adler32s=3" http://LRCHOST:8000/dq2/lrc/files
      2. bulk file registration
        1. repeat the same test as single file registration with muliple files. Here, it will use 2 files.
          1. first generate two unique ids (guids) via uuidgen twice. gid1 and gid2 (note. Make sure to use different pfns and lfns)
          2. curl -d "pfns=test%3A%2F%2Ftest%2FMyLrcTestAtYourSite.1a%2Btest%3A%2F%2Ftest%2FMyLrcTestAtYourSite.1b" -d "lfns=MyLrcTestAtYourSite.1a%2BMyLrcTestAtYourSite.1b" -d "guids=gid1%2Bgid2" -d "fsizes=1%2B2" -d "md5sums=3%2B4" -d "archivals=V%2BV" -d "adler32s=5%2B6" http://LRCHOST:8000/dq2/lrc/files
        2. skip adler32s
          1. Do the same as the above to generate two ids.
          2. curl -d "pfns=test%3A%2F%2Ftest%2FMyLrcTestAtYourSite.2a%2Btest%3A%2F%2Ftest%2FMyLrcTestAtYourSite.2b" -d "lfns=MyLrcTestAtYourSite.2a%2BMyLrcTestAtYourSite.2b" -d "guids=gid1%2Bgid2" -d "fsizes=1%2B2" -d "md5sums=3%2B4" -d "archivals=V%2BV" http://LRCHOST:8000/dq2/lrc/files
        3. pass empty string to md5sums
          1. Do the same as the above to generate two ids.
          2. curl -d "pfns=test%3A%2F%2Ftest%2FMyLrcTestAtYourSite.3a%2Btest%3A%2F%2Ftest%2FMyLrcTestAtYourSite.3b" -d "lfns=MyLrcTestAtYourSite.3a%2BMyLrcTestAtYourSite.3b" -d "guids=gid1%2Bgid2" -d "fsizes=1%2B2" -d "md5sums=" -d "archivals=V%2BV" -d "adler32s=5%2B6" http://LRCHOST:8000/dq2/lrc/files
  2. Query LRC (don't escape + in http get)
    1. Query is done with http get (curl --get) to http://LRCHOST:8000/dq2/lrc/PoolFileCatalog
      1. You can query LRC either by lfns or guids
        1. lfns=lfn1+lfn2+.... where lfn is logical file name
        2. guids=guid1+guid2+... where guid is the guid of the file
      2. (OPTIONAL) at BNL storage=dCache, xrootd/rootd, or All are supplied to support root:// file
        1. storage=dCache : This is default option. You don't need to specify this option It will only returns dCache file
        2. storage=xRootd : This will returns only root:// physical file
        3. storage=All : This will returns all files.
      3. If successful, it will return the information in xml format.
    2. Testing your previous test registration.
      1. single file query by guid
        1. curl --get -d "guids=gid" http://LRCHOST:8000/dq2/lrc/PoolFileCatalog
        2. Make sure it returns in xml format and all information is what you specified previously
      2. single file query by lfn
        1. curl --get -d "lfns=MyLrcTestAtYourSite.1" http://LRCHOST:8000/dq2/lrc/PoolFileCatalog
        2. Make sure it returns in xml format and all information is what you specified previously
      3. bulk file query by guids
        1. curl --get -d "guids=gid1+gid2" http://LRCHOST:8000/dq2/lrc/PoolFileCatalog
        2. Make sure it returns in xml format and all information is what you specified previously
      4. bulk file query by guids
        1. curl --get -d "lfns=MyLrcTestAtYourSite.1a+MyLrcTestAtYourSite.1b" http://LRCHOST:8000/dq2/lrc/PoolFileCatalog
        2. Make sure it returns in xml format and all information is what you specified previously

Documentation


-- CharlesWaldman - 15 Feb 2008

-- RobertGardner - 30 Jan 2008

-- RobertGardner - 16 Apr 2008

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


Attachments

 
Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback