Overview

AutoPyFactory (APF) is the pilot factory/provisioning component used by the ATLAS PanDA? system. It is a single-process, multi-threaded daemon, written in Python, that handles pilot job submission for ATLAS. It handles each distinct workflow (typically one Panda queue + site gatekeeper) with a separate internal APF queue. Every cycle (typically six minutes) it queries for idle work and running/pending jobs, calculates new pilots needed, and submits them via Condor-G.

Although originally created to handle Panda pilot submission specifically for ATLAS, APF is now WMS/target agnostic and is useful in any context where waiting work can be quantified and an action needs to be taken. E.g. APF can query for arbitrary waiting jobs in a Condor queue, and invoke virtual machines on a Cloud platform to run them.

Design

See also the CHEP Poster about APF:

• apf_chep_2013_poster.pdf: apf_chep_2013_poster.pdf

Documentation and Deployment

All documentation is now distributed within the code. It can be found in SVN, or after checking out, underneath directory docs/. To check out read-only copy of the project:

svn co http://svnweb.cern.ch/guest/panda/panda-autopyfactory/current

APF is typically deployed via RPM/YUM from a dedicated repository. Instructions for installing from there are in the documentation.

-- JohnHover - 03 Dec 2013