r1 - 03 Dec 2013 - 12:03:19 - JohnHoverYou are here: TWiki >  Projects Web > AutoPyFactory

Overview

AutoPyFactory (APF) is the pilot factory/provisioning component used by the ATLAS PanDA? system. It is a single-process, multi-threaded daemon, written in Python, that handles pilot job submission for ATLAS. It handles each distinct workflow (typically one Panda queue + site gatekeeper) with a separate internal APF queue. Every cycle (typically six minutes) it queries for idle work and running/pending jobs, calculates new pilots needed, and submits them via Condor-G.

Although originally created to handle Panda pilot submission specifically for ATLAS, APF is now WMS/target agnostic and is useful in any context where waiting work can be quantified and an action needs to be taken. E.g. APF can query for arbitrary waiting jobs in a Condor queue, and invoke virtual machines on a Cloud platform to run them.

Design

AutoPyFactory_Design.png

See also the CHEP Poster about APF:

Documentation and Deployment

All documentation is now distributed within the code. It can be found in SVN, or after checking out, underneath directory docs/. To check out read-only copy of the project:

svn co http://svnweb.cern.ch/guest/panda/panda-autopyfactory/current

APF is typically deployed via RPM/YUM from a dedicated repository. Instructions for installing from there are in the documentation.

-- JohnHover - 03 Dec 2013

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


Attachments


png AutoPyFactory_Design.png (99.7K) | JohnHover, 03 Dec 2013 - 11:56 |
pdf apf_chep_2013_poster.pdf (649.7K) | JohnHover, 03 Dec 2013 - 12:02 |
 
Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback