MinutesReleaseDeployAug8

Introduction

• Meeting of the sub-committee to address release installations in the US cloud.
• Previous meeting: MinutesReleaseDeployJul8
• Coordinates:
  • Tuesday, Aug 8, 9am Central/5pm CERN
  • Phone: (605) 475-6000, Access code: 735188; Dial *6 to mute/un-mute.
• Attending: Fred, Saul, Alessandro, Rob

From Alessandro:

• Release installation pages
• pacball

Notes

• Synch between repository and cache for pacballs is now working (has been).
• Alessandro: How to remove an installation? Saul: cannot - only designed to install.
• Pacball + DQ2 delivery: Xin is not here, but Alessandro thinks this should now be possible.
• Webserver overload - may be a concern.
• Pacballs - will Stan register these in DQ2 for posterity. However, begs question of which space tokens to use. (Stan)
• Xin will need to create special panda jobs to install, using the pacball as input datasets.
• Alessandro will give Xin a script to execute that installs from the pacball.
• The scripts could modified to publish to the portal. (When Alessandro returns from vacation.)
  • Also runs kitval
• Pacballs datasets need to be subscribed to BNL.
• Will need to publish presence of release to Panda monitor or database - Xin will take to Torre's group.
• Will need to send panda jobs will role=software (usatlas2)

AOB

• Fred will summarize actions in email.
• Freds summary:

Begin forwarded message from Fred:

From: Frederick Luehring <luehring@indiana.edu> Date: August 8, 2008 4:56:21 PM CDT To: Xin Zhao <xzhao@bnl.gov> Cc: Robert Gardner <rwg@hep.uchicago.edu>, Saul Youssef <youssef@bu.edu>, Alessandro De Salvo <Alessandro.De.Salvo@cern.ch>, Stan <a.s.thompson@physics.gla.ac.uk>, Alexei Klimentov <alexei@mail.cern.ch>, Michael Ernst <mernst@bnl.gov> Subject: Re: Following-up on ATLAS release installation methods

Hi Everyone,

Here are my minutes from todays phone call on the release installation issues. I would like to thank Rob making available to me the notes he took during the meeting. Please bear with me - the minutes are a little long. Do read to the end where I listed suggested action items for most people. As always please let me know of any corrections. Thanks greatly.

Fred

Follow up meeting on ATLAS release installation - 8/8/8

Present: Rob Gardner, Fred Luehring, Alessandro de Salvo, Saul Youssef, Xin Zhao Apologies: Alexei Klimentov, Stan Thompson

Note on vacations:

Fred is on vacation through August 10 with email (starting August 9) Alessandro is on vacation through August 17th without email and through 24 with limited email (starting August 11). Stan is on vacation through Tuesday August 12 (started about a week ago).

Pacball Readiness

Alessandro and Saul reminded us that the pacball distribution method had been confirmed to be working and that test pacballs were available on the Roma web site. The synchronization of new pacballs onto the web site works correctly (and has for some time already).

Removing Pacballs

Alessandro asked Saul if it was possible to remove pacballs similar to the "pacman -remove" command used to remove pacman installations. Saul reported that this is not possible. Install can be erased with "rm -rf" but this erases all pacball installation in the directory tree and not just the one that is supposed to be removed.

Delivering Pacballs with DQ2

This should be possible but testing is needed. Solveig has provided naming rules for AMI meta data specifying information about pacballs store in DQ2.

Web Server Overload

Rob asked Alessandro about the possibility of overloading the webserver providing the pacballs if lots of sites install at the same time. This needs further investigation.

DQ2 + Pacballs

When Stan is back he will begin registering pacballs in DQ2. We need to decide which space token will be used to store the pacballs in DQ2.

Using Alessandro's Installation System

Xin will create a Panda job that uses Alessandro's script to install the pacballs. The pacballs will be the input file for the panda job. Alessandro will provide Xin with the scripts before Monday. Alessandro needs to modify the scrips slightly before the Xin uses them. Fred asked Alessandro if the install script ran Kit Validation and Alessandro confirmed that this is indeed the case. When Alessandro returns from vacation, the Kit Validation script will be modified so that the results will be reported back to the GKV portal. The benchmark results from the worker node used to run the Kit Validation will also be reported.

First Testing

Test pacball datasets need to be subscribed to the US Tier 1. Alexei will need to set this up. Xin will use these datasets to test installing the pacballs on non-production systems. Xin's test will use panda to run Alessandro's installation scripts.

Publishing Release Availability to Panda

Xin will talk to Torre about how to publish the availability of a release in either Panda monitor or a database. We need to make sure that the correct set of releases are installed and verified on all sites and that sites do not have the releases installed more than once.

Which Account to Use

We agreed to use the usatlas2 account to install the ATLAS software on OSG sites.

Action List

Alexei: Register subscriptions of test pacballs to BNL

Alessandro: Supply revised installation script to Xin

Stan: Put test pacballs into DQ2/AMI.

Xin: Get revised installation script to work in Panda, Talk to Torre about publishing releases, and test panda installation

Saul: We didn't discuss this today but it would be useful for Saul to look at the removing a pacball installation issue.

From Alessandro:

Hi Fred, some small corrections:

"Removing Pacballs" It's possible to do a pacman -remove on a pacball installation, using plain pacman, so all in all my question was not so useful. My installation script already handles this, and I've just tried it works correctly on pacballs installation too (as I would have expected!). I don't think that Saul should investigate further on this for the moment.

"Using Alessandro's Installation System" The installation script is already able to send the KV results to the GKV portal, so this will be available immediately. What will not be immediately available will be the CGIs to send the data from the script, so during the panda job, to the central Installation DB. I have almost everything ready, but I need to organize it. The Installation DB will also need to be extended to be multi-Grid enabled, although it's already prepared for this. My idea is to have a very high integration with the system, including the notification system to the users when a release is installed. If possible, I will also implement the partial dump of the logfiles, so that in the Installation System you'll be also able to see the logfiles of the jobs while the jobs are still running (this has to be generalized, but it's already in use for LCG).

"Which Account to Use" It's important that we correctly define the unix account name. In LCG we distinguish betweek pool accounts and static accounts using the numbers at the end of the username. An account atlassgm will be static, while atlasgm001 is a pool account. This defines the permission to be set to the software area: a static account will need the files to be writable only by the owner, while a pool account will need to have them writable but the sgm group. Which kind of software manager accounts do you have in OSG? The insattaltion script is already able to handle both cases, but we need to define the rules. At the moment is the user has a number at the end is considered to be a pool account, but could not be the case for OSG, so I need to understand better this issue.

Another issue:

- The script will need the following information to publish correctly the information about the releases installing in a site. These information will have to be passed as CLI parameters when calling it * CE name * Site name * OS version/name/revision * Resource name (usually it's /jobmanager-....)

Thanks for the minutes Fred! I'll be traveling to Rome almost all day tomorrow, I'll send you the script and the instructions as soon as possible, before going to vacation. Cheers,

Alessandro

-- RobertGardner - 08 Aug 2008