OSG Workload Management -- Extending Panda for the OSG

Introduction

The Applications Area of the Open Science Grid (OSG? ) organizes and manages a small number of science-driven projects in extending the existing grid middleware to support higher level services and offer them to the OSG community. One of these projects is in workload management, specifically in developing a 'just-in-time' workload management system and tool set based on the 'pilot job' approach to job submission and management.

For some background see this excerpt of an OSG proposal in this area. For more information on the OSG extensions program at BNL see OSGAtBNL.

A major part of the project is generalizing the US ATLAS Panda system to become a VO-neutral workload management system supported on OSG for general use. A major part of this generalization is an integration program with Condor, to be defined and carried out through this project, with the objective of moving functionality out of Panda proper and into a Condor layer. This will slim down the Panda layer while leveraging the functionality, generic nature, and grid-universality of Condor to reach the objective of a powerful and broadly usable just-in-time workload management. The community can either use the full system or subcomponents such as Condor extensions for highly scalable just-in-time workload management that we expect to emerge from this project.

The project involves a mix of ATLAS in-house work funded by US ATLAS, primarily on generalizing the existing Panda into a generic system, plus middleware extension and integration work funded by OSG, carried out primarily at BNL in collaboration with the Condor project (Miron Livny/UWisconsin et al) and CMS (Frank Wuertheim/UCSD et al). Collaboration with STAR (Jerome Lauret/BNL et al) on some aspects is also anticipated.

Project participants

• Torre Wenaus, BNL (ATLAS)
• Barnett Chiu, UT Arlington/BNL (ATLAS)
• Sudhamsh Reddy, UT Arlington/BNL (ATLAS)
• Miron Livny et al, U Wisconsin Madison (Condor)

Collaborating partners:

• Frank Wuertheim, UC San Diego (CMS)

Current activities

The project is just starting up (Sep 2006), with the initial highest priority activities as discussed in the program description below.

Initial activities:

• PilotFactory
• TestPilot
• StorageExtensions

Program outline

Four principal, concurrent activities:

• Generalization of existing Panda to an experiment-neutral just-in-time workload manager
  • Remove ATLAS specificity and make it a generic, modular system usable by any VO via standard interfaces and VO-specific customization via plugins
  • Usable by, and supported for, any OSG VO
  • Supporting VO-defined back end job submission tools and data management tools
• Selective middleware technology studies and functionality/performance evaluations
  • Select technologies for integration with generic Panda
• Integration of select middleware components -- particularly Condor components -- into generic Panda
  • Collaborating with Condor et al on needed middleware extensions
  • Objective to 'slim down' Panda itself while increasing its functionality and generic support for any VO and a wide range of grid resources
  • Program divided into two phases, integration phase 1 (IP1) and IP2
• Inter-experiment collaboration
  • Identifying and integrating high level components from other experiments which are/could be common tools
  • Particular focus on CMS and STAR

Descriptions of these activities follow.

Generalizing Panda

Issues in making Panda generic

• Standardized capabilities/requirements descriptions -- ClassAds?

Selective middleware technology studies and functionality/performance evaluations

A-List tools and technologies to take up, validate, integrate

• CMS/ARDA Dashboard together with client tools (MonaLisa, http (?)) as basis for monitoring

'Yes' list -- things worth looking at

• Standard testing infrastructure (test harness, testing strategy) for workload management components
• Condor-C Native
• Condor-C bulk scheduling
• "schedd-on-the-side"? New. CMS reports good results. 10k jobs/day with excellent reliability in CMS T2s, keeping queues full. Not much scalability testing done.
• glide-in
  • glide-in with GCB with multiple schedds. Running in CDF. Production ready.

'Maybe' list -- too little known to say at this point. Worth learning more

'No' list -- not worth investigating at this stage

• Condor-C gLite
• pre-WS GRAM
• WS GRAM
• Resource broker with classic SE
• Resource broker with gLite SE

Open issues in tools and technologies

• Accounting
• Timescale for glexec production release, validation (by someone else -- Fermilab?)
  • https://savannah.cern.ch/task/?func=detailitem&item_id=3285

Condor (et al) middleware integration

Integration Phase 1

• Site-level pilot factory based on Condor
  • schedd, glide-in based
  • evaluate Condor-C
• Security: pilot authentication with user identity
  • glexec integration
• Application of glide-ins
  • in the pilot factory
  • as basis for pilots, if sufficiently scalable?
• ClassAds for describing/matching capability/requirements
• Pilot generalization as gateway interface to grid computing resource (eg Condor pool)
• Condor submission into Panda?

Integration Phase 2

• Phase 1 carry-over, iteration after feedback, program adjustments
• Generalized data-CPU co-location mechanism
• Scalability enhancements
• Matchmaking?
• Workflow logic

Inter-experiment collaboration

Milestones and effort

Major milestones are in bold.

Program for 2009+:

• Iterative development with lessons from production deployment, OSG community feedback
• Ongoing middleware evaluation and feedback
• Integration of newly matured and validated middleware
• Scaling in support of scale-up by science communities
• Shift emphasis to hardening and support from integration/development