Workload Management Extensions of Panda

---

Introduction

The Applications Area of the Open Science Grid organizes and manages a small number of science-driven projects in extending the existing grid middleware to support higher level services and offer them to the OSG community. One of these projects is in workload management, specifically in developing a 'just-in-time' workload management system and tool set based on the 'pilot job' approach to job submission and management.

For some background see this excerpt of an OSG proposal in this area. For more information on the OSG extensions program at BNL see OSGAtBNL.

A major part of the project is generalizing the US ATLAS Panda system to become a VO-neutral workload management system supported on OSG for general use. This involves generalized mechanisms of data movement which should be easy to deploy and maintain for VOs with limited resources.

Program outline

These are principal activities:

• Generalization of existing Panda to an project-neutral just-in-time workload manager
  • Remove ATLAS specificity and make it a generic, modular system usable by any VO via standard interfaces and VO-specific customization via plugins (such as for data movement)
  • Usable by, and supported for, any OSG VO
  • Supporting VO-defined back end job submission tools and data management tools
• Selective middleware technology studies and functionality/performance evaluations
  • Select technologies for integration with generic Panda
• Inter-experiment collaboration
  • Identifying and integrating high level components from other experiments which are/could be common tools

Project participants

• Torre Wenaus, BNL (ATLAS)
• Maxim Potekhin, BNL (OSG)
• Miron Livny et al, U Wisconsin Madison (Condor)

Current activities

Data handling

• Integration with Globus Online. This is related to the needs of non-Atlas users of Panda who would like to have a platform-neutral tool for reliable data movements

noSQL Solution for Panda Monitoring Upgrade

• Mnemosyne -- a Cassandra-based Web Service for archived Panda Monitoring Data

Pilot and Job Submission for Virtual Organizations outside of Atlas

• • Job submission for non-Atlas VOs
  • Pilot submission for non-Atlas VOs
  • Design of Reusable Components for Panda Information Systems (general discussion with technical notes on database access and legacy components)

Former Projects

• • Light-weight data movement for non-Atlas VOs
  • Pandjango: Panda Monitoring Evolution. Code at https://svnweb.cern.ch/trac/panda/browser/panda-monitor/current/pandamonitor/pandjango.
  • ATLAS Production Monitoring Tool prototype - technology demonstration project
  • Support of LBNE and Daya Bay projects
  • SBGRID Panda migration feasibility study
  • Using Panda for ITB testing

Older information (archives)

• Work plans and milestones
• Integration of Condor component into Panda: the Pilot Factory

---

Major updates:
-- MaximPotekhin - 23 Sep 2011
-- MaximPotekhin - 05 May 2010
-- MaximPotekhin - 18 Mar 2010
-- MaximPotekhin - 23 Nov 2009
-- MaximPotekhin - 06 Nov 2009
-- MaximPotekhin - 26 May 2009
-- MaximPotekhin - 14 Oct 2008
-- TorreWenaus - 19 Sep 2006