Panda applied to ITB testing (OSG Integration Testbed)

---

Project description

Panda provides a centralized and self-documenting method of submitting jobs to multiple of OSG sites, which makes it ideally suited for ITB testing which so far has been a cumbersome process.

Weekly ITB conference call

Documented here

Sites

This table is assumed to be current:

Notes	ITB Site Name	Panda Queue Name	Job Manager
Operational	BNL_ITB_Test1	BNL_ITB_Test1-condor	gridtest01.racf.bnl.gov/jobmanager-condor
Operational	UC_ITB	UCITB_EDGE7-pbs	uct3-edge7.uchicago.edu/jobmanager-pbs
Operational	LBNL_DSD_ITB	LBNL_DSD_ITB-condor	grolsch.lbl.gov/jobmanager-condor
Operational	OUHEP_ITB	OUHEP_ITB-condor	osgitb1.nhn.ou.edu/jobmanager-condor
Operational	TTU_TESTWULF_ITB	TTU_TESTWULF_ITB	testwulf.hpcc.ttu.edu/jobmanager-sge
MyProxy client needs installation	FNAL_FERMIGRID_ITB	FNAL_FERMIGRID_ITB-condor	fgitbgkc2.fnal.gov/jobmanager-condor
WN setup issues resolved on site	LIGO_CIT	LIGO-CIT-ITB-condor	osg-itb.ligo.caltech.edu/jobmanager-condor
Setting up Condor on site	UW_VDT_ITB	UW_VDT_ITB	vdt-itb.cs.wisc.edu/jobmanager-condor

Current issues/solutions

04/08/2010

• To host and parse log files produced by ITB jobs, we need to decide on location and system. Currently, a setup at UC is being proposed.
• ITB output will be tagged according to previously considered conventions

02/25/2010

Five sites are operational, initial testing of glexec-enabled pilot done Fermilab, LIGO in final stages of setup

12/16/2009

We are able to run ITB jobs via Panda, at UoC, BNL, LBNL. Other functional sites are awaiting testing by ITB group. Including recent two we are setting up, total count is now 9.

Technical Notes

Using aggregated views in Panda Monitor

If ITB jobs are appropriately tagged, it will be possible to have an aggregated view in Panda, regardless of origin. To achieve this, add the "processingType" option to the job submission command line:

$ sendJob.py [... rest of options ...]  --processingType ITB_Integration

Then the information becomes available at

http://panda.cern.ch:25980/server/pandamon/query?job=*&processingType=ITB_Integration&detail=no

A quick example of using MyProxy for job submission

In a nutshell, usage scenario looks like this

• Generate and preserve your first proxy, e.g.

  $grid-proxy-info ... -out /tmp/proxy_for_itb

• Delegate a proxy to a MyProxy server
• Default proxy lifetime is one week (configurable)
• Default lifetime of retrieved proxies is 12 h (configurable)

An example of MyProxy interaction will look like:

$myproxy-init -s pandaprx.usatlas.bnl.gov -Z  /DC=org/DC=doegrids/OU=People/CN=Jose\ Caballero\ 511275 -d -k ITB -A

Of course, you'll want to insert a proper DN. This is what you have to do every week, if using default lifetimes.

Proxy renewal:

$myproxy-logon -s pandaprx.usatlas.bnl.gov --no_passphrase  -l  /DC=org/DC=doegrids/OU=People/CN=Jose\ Caballero\ 511275 -a /tmp/proxy_for_itb -o /tmp/proxy_for_itb -k ITB

More info

More MyProxy info can be found here

Archive: Meetings and Conference Calls

Weekly ITB Meetings are documented on the main ITB Meeting page.

Here are bits of information of historical interest.

• ITB conference call on 02/04/2010: LBNL issues resolved, unltimately traced to an extraneous line in the pilot code, which lead to PATH including /usr/bin, not matched by LD_LIBRARY_PATH. TTU wants to join, OUHEP needs to be revived on Panda
• ITB conference call on 12/10/2009: a few issues in job submission scripts resolved. Will configure pilot submission for more sites in a week.
• ITB conference call on 11/19/2009: Panda client scripts have been improved. Initial testing done with site_verify.pl script running on Panda at BNL and Chicago. Decision made to host and manage transformations (payload jobs) on a server at BNL. We'll aim to create a user friendly web service based on Django to allow site admins and other testers to manage the test job payload and configuration. glexec still needs to be integrated into the pilot
• ITB conference call on 11/12/2009: We'll use an identity belonging to OSG VO to manage pilot submission for ITB. This identity will have a VOMS annotation role=pilot. Since we (Panda team) need to do configuration work for new sites (or modify existing ones) ourselves, it makes sense that we also manage actual Pilot submission. This also guarantees that a proper (authorized) host is used. FNAL is one of very few sites using glexec and it's a requirement there. There is extra work to be done in order to update the non-Atlas version of the Pilot to successfully use glexec on that site, for ITB testing. Test jobs may take arguments which may involve double dash, and we need to escape these as they have special meaning in a few places in Panda. We'll create a simple distribution package to give to site admins, which will contain tools to submit jobs to Panda.

---

Major updates:
-- TWikiAdminGroup - 23 May 2012