r15 - 24 Oct 2007 - 14:20:50 - JohnHoverYou are here: TWiki >  Admins Web > FileCatalogP2

FileCatalogP2

Introduction

This page was originally intended to host a performance evaluation of candidate file catalogs for replacement of the current LRC infrastructure in the facility. After some initial research, it is looking like there is little reason to do extensive performance testing, per se, as it would be comparing apples to oranges. Below is a layout of the situation as I see it, which can serve as a starting point for discussion rather than an endpoint.

LFC

LFC is a client-server application written in C. It is integrated with GSI and VOMS security and implements a full virtual namespace with POSIX ACLs, secondary groups, and symlinks. It has a complex multi-table back end, server-side bulk methods, and provides extensible mechanisms for user-defined metadata. It is essentially a full virtual UNIX filesystem. Testing presented at a CHEP poster shows 250Hz GUID lookup using bulk methods. Because it has a server component, arbitrarily sophisticated data models and optimizations can be used to do complex tasks (i.e. it is not limited by what raw SQL can define).

https://twiki.cern.ch/twiki/bin/viewfile/LCG/DataManagementUsefulPresentations?rev=1;filename=chep07_poster_LFC.ppt

LRC

LRC is a essentially a database schema and a thin client-side API, with no integrated security, just that of whatever RDBMS is used. (Although I have heard that there is a way to use GSI/grid-cert security to control access at the DB port level. This would provide course-grained, all-or-nothing access.) Its raw performance will be limited only by the intrinsic speed of the underlying RDBMS (along with proper indexing), which should always be very fast.

But since there is no server component, all the intelligence lies in the clients. Only operations that can be efficiently defined as SQL queries will be feasible. And only course-grained read-write access control could ever be added.

Update: The LRC server built into the DQ2 codebase (by Pedro Salgado) provides a REST-based web services interface to the LRC schema. This implementation provides a REST-based callable interface which makes the MySQL? conneciton locally. Currently this implementation provides course-grained security to LRC instance based on the user's SSL DN.

Other Miscellaneous Info

  • Setting up a stand-alone LFC instance here at Brookhaven was fairly straightforward, and does not require a full gLite/Yaim setup or integration with the EGEE info system. The RPMS can be installed and subsequent setup can be done by hand.
  • LFC will only currently run on SL/Redhat 3. SL/RH4 support will come when the LFC node type is rolled out for Glite 3.1.
  • Client APIs in C, Python, and Perl are provided for LFC.
  • LRC setup is very simple--just create the schema in the DB and provide for external access. GSI authentication for MySQL? would be more complex.
  • The EGEE community's experience with LFC is an open question. More info about this would be useful.

Summary

So the thing that should drive any decisions should be needed features, now and in the future, rather than raw performance provided 250Hz meets current and near-future USATLAS needs. LRC as currently implemented will always be faster than LFC. But if LRC must eventually be made to do more sophisticated tasks, then it will unavoidably suffer performance hits.

If USATLAS is sure that they

  1. will never need fine-grained security,
  2. will never need backward-compatible extensibility,
  3. will never need to be EGEE/gLite compatible,
  4. will never need more than simple, SQL-capable queries,
  5. can do whatever minimal code maintenance is required,

then there is really no reason to consider a change.

But, if any of these premises are not true, then it would make sense to switch as soon as possible so that

  1. Whatever effort that would have been wasted on the home-grown solution can be applied elsewhere,
  2. Site admins can begin to get experience with LFC ASAP (if we would be using a model with LFC at Tier 2s).
  3. Any modifications/features/bugfixes that LFC needs can be requested.
  4. Needed USATLAS LFC infrastructure can be put in place.

-- JohnHover - 26 Sep 2007

-- RobertGardner - 25 Sep 2007

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


Attachments

 
Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback