r3 - 27 Aug 2008 - 12:01:38 - RobertGardnerYou are here: TWiki >  Admins Web > LFCMeetAug27



  • Meeting of the sub-committee to address LFC migration in the US cloud.
  • Coordinates:
    • Wednesday, August 27, 10am Central/5pm CERN
    • Phone: (605) 475-6000, Access code: 735188; Dial *6 to mute/un-mute.
  • Background: SubCommitteeLFC
  • Attending: Massimo, Torre, Charles, Rob, Hiro, Xin, Dantong, Jean-Philip, Alexei, Marco, Paul

http interface for LFC

Dantong Yu wrote:


I summarized what I learned today:

1) Pilots at work nodes need to access LFC catalog, which requires all work nodes running Panda must have functioning LFC interface software being available.

2) The current data Catalog (LRC) for USATLAS Panda production, Pilots effectively use curl to access data catalog. Therefore, no additional LRC client nodes are required to be available at worker nodes.

3) OSG based USATLAS Tier 1/Tier 2 sites will deploy VDT version LFC client and servers. Paul is testing LFC work nodes clients with the new Pilot code. A more thorough tests of LFC worker node client with Pilot should be done on a variety of platforms which run Panda jobs.

4) What is the upgrade path for OSG version LFC worker node clients? How often do we anticipate LFC worker node updates and patches? How often will OSG site administrators have to install worker nodes package to allow Panda jobs?

Item 3) and item 4) will determine the urgency of LFC additional web interface. If Item 3) and Item 4) have some problems, then the web interface will be needed to ensure Panda to run on OSG platforms. If Item3 and item 4) are well addressed, the request is probably in the category of 'nice to have' rather than 'need to have'.

Paul will provide more details about his test results. If Alain can comment on Item 4), that will be highly appreciated.

Sorry to answer you late. I do not know that all related person will be able to make tomorrow's phone call. There will be a LFC subcommittee phone meeting sometime next week organized by Rob.

Thank you very much, Dantong

  • Jean-Philip has been developing an interface with python-http bindings. Has methods to list files in directories, inspect guids, etc.
  • Charles: existing http interface can be accessed w/ curl - would like to use the same interface for compatibility.
  • Torre: long wanted to get pool out of the system. URLs might have pool embedded. Is there a direct mapping? Use this as a moment to clean up, authentication, etc?
  • Alexei - also should consult existing client utilities - will replace some of the existing LRC utilities. Point for creating a coherent set of tools.
  • Main motivation for the http interface remains flexibility it adds for the pilot.
  • Will need to deploy the http server in front of the LFC. Will this run in apache, or will the lfc daemon expose the http interface?
  • Massimo agreed this is a worthwhile investment.

Follow-up on the program of work

  • LFC server deployed at BNL DONE
  • LFC client, client-interfaces, and server packaged in VDT, initial tests: Aug 20 - Charles
  • LFC-enabled OSG worker-node client prepared and tested: Aug 27 - Marco
    • Marco working with Alain on compilation problems with LFC client in VDT.
    • Compilation issue with _lfc.so in VDT to see what causes the problem. Now working with Alain and LFC developers, Akos Frohner
  • LFC deployment process defined: Aug 27 - Rob
    • See FileCatalog
    • Install of LFC server
    • MySQL backend for Tier 2
    • LRC migration script and process for migration defined - Hiro
  • Pilot candidate based on existing (ad-hoc) assembled packages and testbed at BNL: Aug 27 Paul, Facility-contact (Xin)
    • Using a special pilot wrapper
    • Allows the standard pilot module callout to lfc python module
    • there is some junk returned in get-replica commands - being sorted out w/ VDT.
    • Problem with lcg-cp on stage-in.
    • Site mover used for AGLT2 seems to be working okay.
  • Pilot candidate based on new OSG worker-node client at BNL: Sep 3 Paul, Facility-contact (Xin)
  • LFC-based utilities package: Aug 27 - Charles
    • Assess existing LFC utilities
    • Map legacy US tools:
      • Storage element maintenance utilities (cleanse.py, etc)
      • User dataset deletion package (Charles or LCG sites already) Sep 30
      • checkse, cleanse - Patrick, Charles: Sep 30
      • pool-cleaner for dcache sites - Charles Sep 30
  • Create python-based www interface for reading content (?)
  • First US LFC utilities package created: Sep 30 - Charles
  • BNL LFC fully deployed with LRC retirement (or leave as read-only): Sep 30 - depends on Panda readiness, essentially deployed. -Hiro
  • LFC deployed at two Tier 2 sites for Panda testing:
    • UTA - Sep 24 - Patrick
    • UC - Sep 10 - Charles
  • LFC deployment fully defined and readied for Facility: Sep 30 - Charles (first cut), Patrick
  • LFC fully deployed at US Tier2's: Oct 15 - all Tier 2

Carryover issues


  • Next meeting: Wed, September 3

-- RobertGardner - 26 Aug 2008

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback