r14 - 17 Aug 2010 - 19:07:36 - MaximPotekhinYou are here: TWiki >  AtlasSoftware Web > PandaInfo

Design of reusable components for Panda Information Systems


Project Description

Currently, there is a need to provide Panda monitoring data to aggregation services at CERN, such as Atlas Production Dashboard. In addition, there are various motivations to upgrade Panda Monitoring system for both Atlas users and operators, and for the needs of other VOs, as well as Open Science Grid projects such as Integration Testbed (ITB). We aim to create a common set of tools which will allow us to implement these upgrades, while reusing majority of code. In this approach, a web service is created with Panda databases as its back end, serving requests from either clients like Web browsers, or from other Web services (e.g. Dashboard). We are considering use of AJAX technology with
  • JSON as the format of the data container
  • Django as the web framework

Django framework has also used in the DataHost (a simple data hosting service with X.509 security)

Technical notes

Web Design

  • If the state of the system is kept in Javascript objects, it will be impossible to provide links to particular views (state is stored in Javascript objects). This can happen quite naturally if one extensively applies AJAX/JSON and jQuery callbacks. There are a number of solution for that, like the bbq library for jQuery.

Objects in GUI

  • Pilot table
  • Pilot
  • Queue table
  • Queue

Oracle

Intro to Oracle

There is a nice quick reference to Oracle. A few useful commands are documented below:

  • to show a schema, use
    desc ATLAS_PANDAMETA.SCHEDCONFIG;
  • or,
    SELECT * FROM ALL_IND_COLUMNS WHERE TABLE_NAME='XXXXX';
  • to limit number of rows output, do
    select col from ATLAS_PANDAMETA.SCHEDCONFIG where rownum<=20;
  • to show a list of tables and their owners,
    select owner,table_name from all_tables;

When connecting to an Oracle database from clients of most types, it is helpful to set the environment variable ORACLE_HOME to a certain path, then create the path network/admin under it and put the file tnsnames.ora (which can be copied from CERN) into that directory. If that doesn't work (which it often doesn't), try setting the variable TNS_ADMIN to point to that directory.

Command history in Oracle

There is no built-in command history functionality in Orace. One can use xemacs buffer to archive history and output:
M-x sql-oracle
. It would be helpful to truncate the number of rows in the output (see above) for emacs buffer to not overflow. The last command can actually be edited in sqlplus: use "edit" command to achieve that.

Django and Oracle

There are semantic differences in interfacing Sybase clones such as MySQL and Oracle, they are conceptually different. While it is theoretically possible to start an Oracle client without it referring to the file tnsnames.ora (which acts similar to DNS), somehow it rarely works in practice. One solution is indeed to obtain a copy of tnsnames.ora containing service definitions that you need, and place it in the directory $ORACLE_HOME/network/admin. The location of $ORACLE_HOME is not critical as long as the path is readable, and of course $ORACLE_HOME should be set before Django is started so the client library is aware of that location.

This is an exerpt from the Django file settings.py:

DATABASE_ENGINE = 'oracle'
DATABASE_NAME = 'atlas_panda'
DATABASE_USER = 'SECRET_PANDA'
DATABASE_PASSWORD = 'secret_foo'
DATABASE_HOST = ''
DATABASE_PORT = ''

Note that the DATABASE_NAME variable is pointing to a tag in tnsnames.ora rather than a database in a usual sense.

Integration with Legacy Databases is described in a separate chapter of Django Book.

Oracle/Django tricks

Django versions prior to 1.1 do not support object aggregation very well. Absent that, one can use Python collections classes that have a comparable or better performance. For discussion see this link. Here are some excerpts: Example:
counts = defaultdict(int)
for i in Item.objects.all():
    counts[i.category] += 1
This provides an equivalent of count(*) semantics. A slightly more obscure example of Django 1.0 code:
query_set = Item.objects.extra(select={'count': 'count(1)'}, 
                               order_by=['-count']).values('count', 'category')
query_set.query.group_by = ['category_id']

Experience

Multiple Databases

Versions of Django up to and including 1.0 do not have native support of multiple databases. We upgraded to 1.2.1 in 2010, and tested multi-DB asccess.

Another solution (now deprecated) is to create a service for each database involved, which in itself is a trivial matter and may actually have performance advantages (at the cost of having to maintain more than one app). This has been successfully tried in a recent prototype of Production Monitor. Note that when running development instances on LAN and using curl to communicate, there may be a problem if the proxy server does not allow you to connect to a particular port number etc. In that case, when running multiple Django instances on same machine, you need to do something like: setenv NO_PROXY localhost

Serialization

Somewhat unexpectedly, object serialization in Python is a CPU hog. This is an important consideration in designing our sites. There are third-party serializers witn more flexible functionality than exists in Django: link. Arbitrary classes can be serialized as explained here. Another article provides a different set of examples.

Notes on AJAX/JSON and jQuery

Parsing of XML in jQuery:
$.ajax({
type: "GET",
url: "example.xml",
dataType: "xml",
success: parseXml
 });


function parseXml(xml)
{
  $(xml).find("Author").each(function() { $("#output").append($(this).find("name").text());});
}

Notes on the "Legacy" Panda monitoring software

TestPilot.py

Blocks of info on Autopilot screen

showServices
  • Keys the database name as 'testpilot-r' (twice de-referenced in PandaMonitorUtils and mond_conf)
  • utils.initDB('testpilot-r')
  • query = "SELECT * FROM servicelist ORDER BY id"
  • toshow = ('id', 'host', 'userid', 'config', 'pid', 'status', 'lastmod', 'cyclesec')

showQueues
query = "SELECT * FROM schedconfig where 1=1 %s ORDER BY %s" % ( where, order )

showPilots
Note that the key into the table is the "nickname" as it is defined in SubmitHosts? table.

Blocks of info on Production screen

Tasks by Cloud
    'taskdb-waiting-user':'panda-reader1',
    'table-active'            : 'jobsActive4',
    'table-archive'           : 'jobsArchived4',
    'table-oldarchive'           : 'jobsArchived4_current',
    'table-defined'           : 'jobsDefined4',
    'table-waiting'           : 'jobsWaiting4',
    'table-files'             : 'filesTable4',
    'table-oldfiles'             : 'filesTable4_current',
    'table-proddatasets'      : 'ProductionDatasets',
    'table-swreleases'        : 't_sw_release',
    'table-swtransformations' : 't_sw_transformation',
    'table-sites'             : 'sitesInfo',
    'table-datasets'          : 'Datasets',
    'table-sitespecinfo'      : 'siteSpecifics',
    'table-prodperiods'       : 'ProductionPeriod'

dbname in 'active', 'waiting', 'defined', 'archive', 'activeCERN', 'archiveCERN', 'definedCERN', 'waitingCERN', 'monitor', 'monitorCERN', 'monitor-r':
            db = ...['oracle_taskdbhost'] ('ATLAS_PANDAMON')
            dbid = ...['oracle_taskdb'] ('ATLAS_PANDA')
            dbuser = ...['oracle_taskdbuser-r']'(ATLAS_PANDAMON_READER')


---
*Major updates*: 

-- Main.TWikiAdminGroup - 20 Nov 2017

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To [[https://www.usatlas.bnl.gov/twiki/bin/edit/AtlasSoftware/PandaInfo][edit the content of this page]], click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


Attachments

* *[[AtlasSoftware.WebHome][Software Home]]* * *[[WebSearch][Search]]* * *[[TWiki.TWikiRegistration][Register]]* --- * *[[Support.WebHome][User support overview]]* --- * *Hot topics* * [[EventDataAccessAtBNL][Accessing data]] * [[https://www.racf.bnl.gov/docs/howto/grid][Getting a grid certificate]] * [[AnalysisJamborees][Analysis Jamborees]] --- * *Software support* * [[AtlasSoftware.AnalysisSupport][Analysis Support]] * [[WorkingAtTier1][Working at BNL]] * [[ProofAtBNL][PROOF Farms at BNL]] * [[DatasetsOnXRD][Datasets at BNL xrootd]] * [[https://twiki.cern.ch/twiki/bin/view/Atlas/DAonPanda][Analysis on Panda]] * [[InformationAndHelp][General info & help]] --- * *Software projects* * [[PanDA][Panda]] * [[OsgDdm][DDM]] * [[OSGAtBNL][OSG extensions]] --- * *Facility support* * [[https://www.racf.bnl.gov/docs][RACF facility guide]] * [[https://www.racf.bnl.gov/docs/howto/grid][Using the grid]] * [[https://www.racf.bnl.gov/docs/rt][Submit a help ticket]] * [[https://www.racf.bnl.gov/about/contactinfo][Facility contacts]] * [[http://www.opensciencegrid.org/?pid=1000100][OSG support]] --- * *Facility information* * [[Support.SystemMonitoring][Facility & grid monitors]] * [[Support.UserPolicy][ResourceUsagePolicy]] --- * *Other Links* * [[https://twiki.cern.ch/twiki/bin/view/AtlasComputing/AtlasComputing][ATLAS computing]] * [[https://twiki.cern.ch/twiki/bin/view/Atlas/WorkBook?skin=print.pattern][ATLAS computing workbook]] * [[https://twiki.cern.ch/twiki/bin/view/AtlasProtected/PhysicsAnalysisWorkBook][ATLAS physics analysis workbook]] * [[https://twiki.cern.ch/twiki/bin/view/Atlas/][ATLAS wiki]] * [[https://espace.cern.ch/atlas-forums/default.aspx][ATLAS eGroup]] * [[http://del.icio.us/atlascomp/.main][ATLAS del.icio.us]] * [[http://www.usatlas.bnl.gov/][US ATLAS home]] * [[OSG:Main][OSG wiki]] --- * *AtlasSoftware web* * [[WebIndex][Page index]] * [[WebChanges][Recent changes]] --- * *All webs* * [[Admins.WebHome][Admins]] * [[AtlasSoftware.WebHome][AtlasSoftware]] * [[IB.WebHome][IB]] * [[Main.WebHome][Main]] * [[PAS.WebHome][PAS]] * [[PanDA.WebHome][PanDA]] * [[Projects.WebHome][Projects]] * [[ProofXrootd.WebHome][ProofXrootd]] * [[Sandbox.WebHome][Sandbox]] * [[Support.WebHome][Support]] * [[TWiki.WebHome][TWiki]] * [[TemplateWeb.WebHome][TemplateWeb]] * [[Trash.WebHome][Trash]] * [[Trash/BNLTWiki.WebHome][Trash/BNLTWiki]] * [[Trash/DatasetsOnXRD.WebHome][Trash/DatasetsOnXRD]] * [[Trash/LocalAdmin.WebHome][Trash/LocalAdmin]] * [[USATLASMain.WebHome][USATLASMain]] * [[WebLeftBar][Left Bar]]
 
Powered by TWiki *
*
*
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback