Overview
AutoPyFactory (APF) is the pilot factory/provisioning component used by the ATLAS PanDA? system. It is a single-process, multi-threaded daemon, written in Python, that handles pilot job submission for ATLAS. It handles each distinct workflow (typically one Panda queue + site gatekeeper) with a separate internal APF queue. Every cycle (typically six minutes) it queries for idle work and running/pending jobs, calculates new pilots needed, and submits them via Condor-G.
Although originally created to handle Panda pilot submission specifically for ATLAS, APF is now WMS/target agnostic and is useful in any context where waiting work can be quantified and an action needs to be taken. E.g. APF can query for arbitrary waiting jobs in a Condor queue, and invoke virtual machines on a Cloud platform to run them.
Design
See also the CHEP Poster about APF:
Documentation and Deployment
All documentation is now distributed within the code. It can be found in SVN, or after checking out, underneath directory docs/. To check out read-only copy of the project:
svn co
http://svnweb.cern.ch/guest/panda/panda-autopyfactory/current
APF is typically deployed via RPM/YUM from a dedicated repository. Instructions for installing from there are in the documentation.
--
JohnHover - 03 Dec 2013
About This Site
Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.
Attachments
AutoPyFactory_Design.png (99.7K) |
JohnHover, 03 Dec 2013 - 11:56 |
apf_chep_2013_poster.pdf (649.7K) |
JohnHover, 03 Dec 2013 - 12:02 |