Workload Management Extensions of Panda

Introduction

The Applications Area of the Open Science Grid organizes and manages a small number of science-driven projects in extending the existing grid middleware to support higher level services and offer them to the OSG community. One of these projects is in workload management, specifically in developing a 'just-in-time' workload management system and tool set based on the 'pilot job' approach to job submission and management.

For some background see this excerpt of an OSG proposal in this area. For more information on the OSG extensions program at BNL see OSGAtBNL.

A major part of the project is generalizing the US ATLAS Panda system to become a VO-neutral workload management system supported on OSG for general use. This involves generalized mechanisms of data movement which should be easy to deploy and maintain for VOs with limited resources.

Program outline

Four principal, concurrent activities:

• Generalization of existing Panda to an project-neutral just-in-time workload manager
  • Remove ATLAS specificity and make it a generic, modular system usable by any VO via standard interfaces and VO-specific customization via plugins
  • Usable by, and supported for, any OSG VO
  • Supporting VO-defined back end job submission tools and data management tools
• Selective middleware technology studies and functionality/performance evaluations
  • Select technologies for integration with generic Panda
• Integration of select middleware components -- particularly Condor components -- into generic Panda
  • Collaborating with Condor et al on needed middleware extensions
  • Program divided into two phases, integration phase 1 (IP1) and IP2
• Inter-experiment collaboration
  • Identifying and integrating high level components from other experiments which are/could be common tools

Project participants

• Torre Wenaus, BNL (ATLAS)
• Maxim Potekhin, BNL (OSG)
• Jose Caballero, BNL (OSG)
• Miron Livny et al, U Wisconsin Madison (Condor)

Project components, plans and milestones

• Work plans and milestones
• Integration of Condor component into Panda: the Pilot Factory
• Creation of a lightweight mechanism for generic VO data transport in Panda: Panda Data Host
• For those VOs with more demands to scalability and efficiency of data movement, leveraging automated data movement features in Panda which use file catalog (LFC)

-- MaximPotekhin - 06 Nov 2009
-- MaximPotekhin - 26 May 2009
-- MaximPotekhin - 14 Oct 2008
-- TorreWenaus - 19 Sep 2006