r18 - 06 May 2013 - 12:45:49 - ShawnMckeeYou are here: TWiki >  Projects Web > PerfSONAR_PS_Mesh

USATLAS (or WLCG) perfSONAR-PS Mesh Setup

There is a new (as of August 2012) "mesh" configuration for perfSONAR-PS installations created by Aaron Brown/Internet2. The idea is to provide a URL which defines the member perfSONAR-PS instances and the tests the should be running. Each perfSONAR-PS instance can be configured to reference that URL. A set of local tools can generate the appropriate configuration for the toolkit from the JSON configuration stored in the URL.

These instructions should work for either bandwidth or latency nodes and the intent is to use the mesh-configuration for both.

Setup for Central Configuration

NOTE: As of November 8, 2012 we have a simple shell script courtesy of Dave Lesny to do all of the setup steps documented here. If you want to use this script, download it from perfSonarMesh.sh (Updated Nov 29, 2012), edit the file to change the email destination in the mesh configuration (or other customization) and execute it. Also you can safely ignore the expected first-run error similar to:
2012/11/27 08:04:44 (7576) ERROR> Agent.pm:706 perfSONAR_PS::MeshConfig::Agent::__load_mesh_cache - Couldn't open /var/lib/perfsonar/mesh_config/agent_saved_meshes.db
To be clear, this script runs ALL the manual steps on this page for you If you use the script you shouldn't need to do the various 'wget' and 'cp' operations below.

First a "mesh" configuration file needs to be created. The Internet2 repo has all the needed RPMS:

name = Internet2 RPM Repository - software.internet2.edu - main
baseurl = http://software.internet2.edu/rpms/i386/main/
enabled = 1
protect = 0
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Internet2
gpgcheck = 1

Set this up in your YUM repo (/etc/yum.repos.d/) as 'Internet2.repo' if you don't have it already.

Details about the setup are at http://code.google.com/p/perfsonar-ps/wiki/MeshConfigurationInstallation

First get the software in place:

  • Update your perfSONAR-PS node: yum update
  • Install the required RPMS: yum install perl-perfSONAR_PS-MeshConfig-Agent

The mesh configuration is stored in a .json file that can be retrieved from a well-known URL. To ease the building of the .json, there is a script included with the RPM called "build_json". This script converts a config file in Config::General format into the appropriate .json. There is an commented example file in the RPM available in "/opt/perfsonar_ps/mesh_config/doc/example.conf". You can view it online here.

To use the build_json tool, you can run `build_json -input example.conf -output example.json`. This will convert the configuration file named example.conf into a json file named example.json. This file can then be placed into a well-known location to be downloaded by agents running on the mesh's hosts.

For USATLAS we created a usatlas.conf file which captures the current configuration shown on the modular dashboard at: http://perfsonar.racf.bnl.gov:8080/exda/?page=25&cloudName=USATLAS

This was converted into a JSON file via the build_json tool. Here is the USATLAS mesh-config reference file.

For other meshes (Italian, UK, etc.), Simone Campana has setup a location in AFS at /afs/cern.ch/project/gd/wlcg-ops/perfsonar/conf/ where we can store the various required mesh configurations. As of May 6, 2013 we have moved the USATLAS config noted above here. To reference any mesh file in the CERN AFS location you can use a URL like https://grid-deployment.web.cern.ch/grid-deployment/wlcg-ops/perfsonar/conf/...

Example Mesh-Config Display Using MadDASH for Testing the USATLAS Config

Once a usatlas.conf draft configuration was ready, Aaron used it to setup a maddash dashboard:


This is an example of the power of having a single configuration which defines a set of organizations, sites, hosts and tests.

Setting up perfSONAR-PS Instances to Use the USATLAS Mesh Configuration

Any site wishing to use the "central" USATLAS configuration should do the following (but see the gotcha's below!!!). For WLCG sites you will need to make minor adjustments for the URL locations and mesh config file names (NOTE see script mentioned near the top which does all of the following after editing to change the email address!):

    configuration_url           http://www.usatlas.bnl.gov/twiki/bin/view/Projects/rsrc/Projects/PerfSONAR_PS_Mesh/usatlas.json
This line replaced the default configuration_url near the beginning.
admin_email     smckee@umich.edu
admin_email     laurens@pa.msu.edu
These lines were added to the end.
  • Setup a cron-task to regular check/update the local configuration when the central usatlas.json file gets updated. Once configured, the agent can be run manually, or by setting up a cron job to run it regularly. There is an example cron script in the 'scripts' directory of the software package. This script can be copied to /etc/cron.d directly, or modified as desired. The RPM installs this crontab entry by default.

NOTE: There are some "gotcha's" currently (NOTE: These are handled by the install script and only need to be addressed if you are manually installing):

  • *If you set this up on a host with existing tests the current version will lose those tests*. You will end up with only the tests defined by the usatlas.conf file. This is under investigation and will be fixed in a future version of the tools.
    • This was addressed by an update from Aaron on October 3, 2012. If you run the mesh configuration now it should preserve any existing tests you already have. However there is still a limitation. You cannot "save" new configurations from the GUI (web) interface or it breaks your configuration file. A future update will address this.
  • The 'OWAMP' and 'BWCTL' tests created by the mesh configuration are not run both ways. This means that for the "modular dashboard" monitoring there will be "brown" results in the columns of sites running the mesh configuration. A future update will have an option to configure bi-directional testing the same as we previously had when manually configuring tests.
    • On October 5, 2012 Aaron released an update with allows the use of a new keyword in the mesh configuration file force_bidirectional. If this is set to '1' in a test definition, the resulting tests are setup bi-directionally now: force_bidirectional 1
  • There is a minor fix needed with the config_daemon configuration file that requires an update:
  • There is a configuration difference between the existing perfSONAR-PS toolkit and what the mesh sets up. To allow the GUI to be able to see the new tests you need to replace one of the perl packages as follows:


Some questions (and answers) that have arisen about the Mesh Configuration

  • Is it possible to download multiple configuration files for different meshes?
Yes  For sites participating in more than one mesh, they can point to more than one URL in their mesh configuration file.   At the LHCOPN/LHCONE meeting we also discussed have a unique URL per site that would allow central control of site-specific tests.  This would allow us to quickly setup temporary meshes between sites that don't normally test to one another.

  • How to modify the existing configuration of running servers? Do I have to delete all the existing test hosts that will, from now on, be inserted in the json file, leave the remaining test hosts untouched, and then start downloading the file?
You can update the running server using the instructions (or shell script suitably edited) above.   It should create a new set of tests that replicate the existing tests for the mesh you configure.  Once you verify the new tests are in place you can remove the duplicate (older) tests from the host.  From then on the mesh file will control the tests and sites for that mesh.

For existing hosts and tests not part of a mesh config, you can just leave them as part of the normal host configuration and they should continue to work as before.

  • Does the same file fit both latency and bandwidth test servers (i.e. only one configuration file, regardless if the site has one or two servers)?
Yes, that is the idea.  If there is only one host doing both tests, the mesh configuration file should have both tests associated with that single host.  You only need to update that one host and point to the mesh configuration which should configure both types of tests for that host on that mesh.

  • Do all the sites in the mesh have to have reconfigured their hosts for all the thing to work? I'm worried for the "interim" period of the deployment, for I'm not sure that all sites will reconfigure their perfsonar servers at the same time, so I fear the interaction between old/new configurations.
No, you can mix legacy sites and new mesh sites.   The sites not using the mesh configuration should continue to work as before.  They refer to other sites by their FQDN in their configuration and that shouldn't change (they don't refer to any details of how the remote site configures its tests).   Same for new mesh configured sites...they don't know (or care) if the other sites are configured manually or by the mesh configuration.

Send along any new questions or FAQ entries to Shawn for inclusion

-- ShawnMckee - 02 Oct 2012

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


conf usatlas.conf (26.7K) | ShawnMckee, 30 Oct 2012 - 16:16 | V1.4 of the USATLAS mesh-configuration
json usatlas.json (25.3K) | ShawnMckee, 30 Oct 2012 - 16:16 | V1.4 of the USATLAS mesh-configuration
sh perfSONARMesh_setup.sh (7.3K) | ShawnMckee, 26 Nov 2012 - 16:49 | Added note about editing admin email
sh perfSonarMesh.sh (8.5K) | ShawnMckee, 29 Nov 2012 - 15:12 | Updated/cleaned-up version from Dave Nov 29 2012
Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback