BACK TO TOP

NICOS (NIghtly COntrol MANagement) facilitates nightly builds of software projects. Nightly builds are common in software project involving multiple developers. NICOS is the system originally developed for ATLAS nightly builds. It was evolved in the independent software package that can be applied to more than one experiment or application.

NICOS performs nightly build processes in distinctive steps. Each step allows plug-ins and can be configured for particular use case. The compilation itself is one of initial steps followed by tests, analysis of errors, and creating web pages with build summaries. Developers are notified about problems with their packages via e-mail.

NICOS supports the organization when the releases are built on a local disk (usually fast and reliable) and then copied to a distributed file system (e.g. AFS). The builds can be incremental, i.e. when only new versions of packages are compiled, or from scratch. While NICOS is originally developed as the nightly build control system it can also build stable and local "personal" releases (in latter case the release contents is determined by a developer).

NICOS allows to build the chain of nightly projects. In this case the projects are built in the certain order determined by declared projects dependencies. ATLAS nightlies releases include (as of June 2007) 11 projects: AtlasCore, DetCommon, AtlasConditions, AtlasEvent, AtlasReconstruction, AtlasSimulation, AtlasTrigger, AtlasAnalysis, AtlasOffline, AtlasProduction, AtlasPoint1. Some projects can be built in parallel (e.g. AtlasReconstruction and AtlasSimulation).

NICOS core does not contain ATLAS specific features. For convenience of ATLAS software infrastructure management the scripts with self-evident (and long) names for typical tasks are added to NICOS distribution. These scripts require very few parameters and allow to run NICOS without reading manuals.

The most recent version of NICOS is installed in
/afs/cern.ch/atlas/software/dist/nightlies/nicos_2_0.
ATLAS nightlies use this installation. ATLAS software releases (and ATLAS CVS repository) contain NICOS in
AtlasTest/NightlyTestTools/Nicos.
ATLAS plug-ins and extensions are collected in atlas and the scripts for typical atlas tasks in atlasjobs subdirectories.

BACK TO TOP

NICOS executable is nicos_job (or similar name), a Bourne shell script. Before running NICOS the following variables can be defined (only two variables NICOS_HOME and NICOS_CONFIG_AREA are mandatory):

• NICOS_HOME is the location of NICOS installation (at CERN /afs/cern.ch/atlas/software/dist/nightlies/nicos)
• NICOS_CONFIG_AREA is an area where configuration files are located. For ATLAS nightly builds the configuration areas are located in: /afs/cern.ch/atlas/software/dist/nightlies/conf/nicos_config_area<suffix>
• NICOS_SUFFIX_PREPEND. NICOS calculates an abbreviated name for each nightly job called "nicos suffix". This suffix is appended to names of files and directories created in the course of nightly builds. NICOS_SUFFIX_PREPEND is the first part of "nicos suffix" specific to a nightly branch (e.g. Dev for development nightlies). The other parts of the "suffix" indicate 32 or 64-bit platform (32B or 64B), major version of SL operating system (e.g. S3 for SLC4), abbreviated project name, compiler option (Dbg or Opt)
• NICOS_PROJECT_RELNUMB_FIXED sets the fixed number of the nightly release. Otherwise the number of nightly release is derived from today's date. For instance if NICOS_PROJECT_RELNUMB_FIXED=4 then the nightly name is rel_4
• NICOS_DOMINANT_PROJECTS is the list of "nicos suffixes" of projects that should be completed before the compilation of the current project starts. By default the hierarchy of ATLAS projects is defined in NICOS configuration files. For instance the "downstream" AtlasCore and AtlasConditions are "dominant" for AtlasEvent by default
• NICOS_DOMINANT_PARALLEL is the list of "nicos suffixes" for the builds on other platforms performed on the same machine and sharing the code with the current platform. In this case one platform is a master platform responsible for checkouts and other platforms should have NICOS_DOMINANT_PARALLEL defined accordingly. For instance if debug and optimized builds are performed on the same machine then the debug build is usually responsible for a checkout. Then the optimized build should have the debug "nicos suffix" defined in NICOS_DOMINANT_PARALLEL. It insures that the optimized "make" loop will not start until the checkout (performed by the debug build) is completed. Usually, in ATLAS nightlies the builds for different platforms are performed on separate machines
• NICOS_CHECKOUT_OPTION specifies additional options in checkout command line. In ATLAS nightly builds cmt co command is used for which an additional "-t" option (to "touch" the directories of checked out packages)
• NICOS_LOG_SCRATCH - if is equal "no" then NICOS logfiles are not cleaned. This may be desirable at rebuilds
• NICOS_BUILD_FROM_SCRATCH - if "yes" than the build area is cleaned before the build. The value set before the start of NICOS job overrides the definition in NICOS cache file
• NICOS_KEEP_STAMPS - if "yes" then NICOS will not clean timestamps associated with the current project. NICOS maintains the database of timestamps for synchronization of projects. Sometimes it is necessary to rebuild a downstream project without interfering with ongoing builds of other projects. In this case the value NICOS_KEEP_STAMPS should be set to "yes" before the start of a rebuild
• NICOS_RECALCULATE_TAGS - if "yes" then NICOS creates the tags lists for each project in the beginning of that project build. By default NICOS creates the tag lists for all projects in the beginning of the first project build (AtlasCore)
• NICOS_CREATE_LOGFILES - if "no" then NICOS does not split the build log files into individual fragments. Instead it expects that the compilation management utility, such as tbroadcast, produces the build logfiles of individual packages for NICOS

NICOS uses three configuration files (see descriptions and examples below).

• nicos_cache (fixed name) must be located in ${NICOS_CONFIG_AREA}. It contains all definitions needed for nightly builds
• NICOS tags database is a text file with names of packages, tags, and addresses of software developers. The name and location of this file is defined in nicos_cache. In case of ATLAS nightly builds NICOS runs a plug-in that retrieves the tags from ATLAS Tag Collector. However the tags in NICOS text file override the tags from Tag Collector
• NICOS test database (optional) is a text file with names of tests, test suites, locations in the software release, and addresses of software developers. The name and location of this file is defined in nicos_cache. In case of ATLAS nightly builds NICOS uses ATN tool that automatically finds in the release tree and runs integration tests. It reads the NICOS test database for information on additional or canceled tests

Once the above-mentioned variables are defined and nicos_cache is configured the nicos job can be started with
${NICOS_HOME}/nicos_job [OPTIONS]

The options are

• -b | --begin : starting point of nicos_job; allowed values: all (full), build, tests, inttests, docs (documentation); default full
• -s | --scratch : release area is cleaned before build. This option works only for full release builds
• --copy : repeat copy to the permanent release area
• --nocopy : cancel copy to the permanent release area. This option works when the release is build in a local area and then copied to a distributed file system
• --nowait : when building the chain of projects: do not wait for completion of projects that current project uses
• -r | --relnum : release number (not name, just a number in the end of name, e.g. "5"). It overrides automatically generated number
• -h | --help : show help

BACK TO TOP

This file in XML-like format contains all necessary configuration parameters. Each build step is associated with a Markup tag that is followed by environment definitions and commands for particular step. A markup tag can contain some name or base directory definitions. Below are the examples from ATLAS nightly builds configuration files with comments.

• INITIALIZATION STEP: PROJECT SETTINGS. At this step general environment variables are defined (valid for all other steps). ATLAS has several nightly releases branches (~ 10 as of June 2007) with similar configuration. The "common" parameters can be specified in a shared configuration file (its name should be declared in NICOS_COMMON_CONFIG_AREA variable). If a shared configuration is specified then NICOS
  1. set environment specified in the project configuration file
  2. set environment specified in the shared configuration file. Hence the shared configuration file can use variables from the project configuration file
  3. reset environment according the project configuration. Therefore the definitions of project configuration file are dominant.

  <PROJECT SETTING name="some name" relname=rel nightly="13.X.Y" lifetime=7 inc=yes>
  NICOS_PROJECT_HOME=/build/atnight/localbuilds/nightlies/rel
  

  In a Markup tag several names are defined:
  1. name of project, e.g. AtlasCore. NICOS copies it to the internal variable NICOS_PROJECT_NAME
  2. name of directories with nightly releases (rel). The suffix with the release number is appended, for instance for seven day cycle the releases names are rel_0 - rel_6, where the number is a day of week
  3. name of nightly: usually an ATLAS release number. This name appears on the NICOS web page with all nighties summary
  4. lifetime indicates number of releases in a cycle (1 - 365). For instance, if it is equal to 7 then seven nightly releases exist ( "0" release is built on Sunday, "1" is built by Tuesday etc. )
  5. inc indicates that the builds are incremental (i.e. use the results of previous nightlies)

  The tag content example:

  NICOS_PROJECT_HOME=/build/atnight/localbuilds/nightlies/projects/AtlasCore
  # Area where nightlies are built
  NICOS_COPY_HOME=/afs/cern.ch/atlas/software/builds/nightlies/dev/AtlasCore
  # Area in distributed file system to which nightlies are copied 
  NICOS_STAMPS_HOME=/afs/cern.ch/atlas/software/builds/stamps
  # Area for NICOS time stamps used for synchronization of dependent projects
  NICOS_BUILD_FROM_SCRATCH="yes"
  # All files are wiped out from the release area before the build if "yes",
  # The sources are updated and release area is not cleaned if "no"
  NICOS_PROJECT_ARRAY="AtlasCore DetCommon AtlasConditions AtlasEvent
  AtlasReconstruction AtlasSimulation AtlasTrigger AtlasAnalysis AtlasOffline 
  AtlasProduction AtlasPoint1"
  NICOS_PROJECT_ARRAY_S="Core Det Cond Evt Rec Sim Trg Anl Offl Prod Pnt"
  NICOS_PROJECT_DEPS="00000000000 10000000000 11000000000 10100000000 10110000000
  10111000000 10111100000 10111110000 10111111000 10111111100 10111111110"
  # The arrays specify the projects names and dependencies. 
  # NICOS_PROJECT_ARRAY_S contains abbreviated project names for use in 
  # NICOS file and directory names.  
  # NICOS_PROJECT_DEPS is a list of dependency "bits" for each project.
  # The bit value "1" indicates dependency. In this example the bits for
  # AtlasConditions are 11000000000, therefore it depends on two first projects,
  # namely AtlasCore and DetCommon
  NICOS_ATLAS_RELEASE="13.1.0"
  NICOS_RELEASE_ARRAY="13.1.0 13.0.0 13.1.0 13.1.0 13.1.0 13.1.0 13.0.0 13.0.0 
  13.0.0 13.0.0 13.0.0"
  # Either NICOS_ATLAS_RELEASE or NICOS_RELEASE_ARRAY should be specified for
  # reading tags from the Tag Collector database. Also alternative release
  # numbers can be indicated: 
  # NICOS_ATLAS_ALT_RELEASE="13.0.20"
  # NICOS_ALT_RELEASE_ARRAY="13.0.10 13.0.0 13.0.10 13.0.10 13.0.10 13.0.10 
  # 13.0.0 13.0.0 13.0.0 13.0.0 13.0.0"
  # The tags from alternative releases are added to the list of tags of main
  # release (without overriding the tags of main release)
  NICOS_TITLE_COMMENT="tags for ${NICOS_ATLAS_RELEASE}" 
  # This line is added in the title of NICOS web pages
  NICOS_HOUR=15
  # This variable indicates that the builds started after 15:00 
  # get release name with the suffix for the next day. For example
  # if release starts on Monday at 21:00 then its name gets suffix _2
  NICOS_WEBPAGE="http://atlas.web.cern.ch/*****"
  NICOS_COMMON_WEBPAGE="http://atlas.web.cern.ch/*****"
  NICOS_WEBDIR=/afs/cern.ch/atlas/software/dist/nightlies/nicos_web_area${NICOS_SUFFIX}
  NICOS_COMMON_WEBDIR=/afs/cern.ch/atlas/software/dist/nightlies/global
  # These variables store URL's and locations of directories with
  # NICOS html files. Common web area holds the summary page for
  # all available nightlies  
  NICOS_WEB_HOME="http://cern.ch/atlas-computing/links/buildDirectory/nightlies/dev/${NICOS_ATLAS_PROJECT}"
  # This variable points to the web address of the nightly release location
  NICOS_DBFILE=${NICOS_CONFIG_AREA}/tags.db
  # NICOS database file with names of packages, tags, and addresses of
  # software developers (default shown)
  NICOS_TEST_DBFILE=${NICOS_CONFIG_AREA}/test.db
  # NICOS test database file with names of tests, suites, locations,
  # and addresses of software developers (default shown)
  #NICOS_PROJECT_TYPE="parallel"
  # This definition indicates that the nightly uses the code checked out 
  # by another nightly working for a different platform
  NICOS_COORDINATOR="undrus@bnl.gov"
  NICOS_RELEASE_COORDINATOR="undrus@bnl.gov"
  # NICOS send some notifications to these addresses.
  PATH=/afs/cern.ch/sw/java/i386_redhat73/jdk/sun-1.5.0_02/bin:${PATH}
  # The path to java executable for taginfo application (read Tag Collector)   
  CVSROOT=*****
  # CVSROOT is required for checkouts
  # basic CMT variables should be defined:
  SITEROOT=/afs/cern.ch
  CMTBASE=/afs/cern.ch/sw/contrib/CMT
  CMTVERS=v1r20p20070720
  CMTROOT=${CMTBASE}/${CMTVERS}
  CMTCONFIG=i686-slc4-gcc34-opt
  CMTSITE=CERN
  CMTCVSOFFSET=offline
  #Variables for CMT setup.
  

• STEP 1: RELEASE TOOL SETUP. At this step, in ATLAS nightly builds cmt is initialized and the Tag Collector database is accessed by the external script. The commands inside <BYPASS> tag are not run for partial builds procedures (e.g. when only tests are repeated). In a Markup tag the work directory (relative to the release directory $NICOS_RELHOME) is defined. If command=source is indicated then the variables defined at this step are valid for subsequent steps.

  <RELEASE TOOL SETUP dir=. command=source>
  source ${CMTROOT}/mgr/setup.sh
  CMTPATH=${NICOS_RELHOME}
  <BYPASS>
  ${NICOS_HOME}/atlas/atlas_read_tc_all_proj_new
  </BYPASS>
  

• STEP 2: CHECKOUT. At this step the sources are checked out. In a Markup tag the following is defined:
  1. method of tag storage; for ATLAS use case it is file that is text database file
  2. directory in which sources are stored in CVS repository (could be ".")
  3. command for checkout, if different from cvs
  4. shell for checkout commands. The ATLAS CVS repository access requires TGT kerberos token

  <CHECKOUT tagstorage=file cvsdir=offline command=cmt shell=/bin/sh>
  

• STEP 3: PROJECT SETUP. At this step additional commands for project setup can be performed. In this example the external scripts generate cmt/requirements file for [Atlas Project]Release package and setup CMT. In a Markup tag the work directory (relative to the release directory $NICOS_RELHOME) is defined. If command=source is indicated then the variables defined at this step are valid for subsequent steps.

  <PROJECT SETUP dir=. command=source>
  <BYPASS>
  if [ "$NICOS_PROJECT_TYPE" != "parallel" ]; then
  cd $NICOS_RELHOME/${NICOS_ATLAS_PROJECT}Release/cmt
  ${NICOS_HOME}/atlas/atlas_generate_req ${NICOS_ATLAS_PROJECT}Release
  cd ${NICOS_PROJECT_HOME}/${NICOS_PROJECTSETUP_DIR}
  </BYPASS>
  source ${NICOS_HOME}/atlas/setup_cmt.sh
  

• STEP 4: PROJECT BUILD: Commands for release build are listed (example from ATLAS nightlies). NICOS needs NICOS_BUILDLOG_SEPARATOR be defined. It is required that the release build tool prints a separating line between builds of different packages. This line should contain the name of package to be built (preferably with all containers, for instance Utilities/FileCatalog if package FileCatalog is in the container Utilities), in which case the variable NICOS_FULLNAME_INSEPARATOR is set to "yes". In a Markup tag the work directory (relative to the release directory $NICOS_RELHOME) is defined. Also the desirable shell can be indicated.

  <PROJECT BUILD dir=${NICOS_ATLAS_PROJECT}Release/cmt shell=/bin/zsh>
  source setup.sh
  cmt broadcast - make
  NICOS_FULLNAME_INSEPARATOR="yes"
  NICOS_BUILDLOG_SEPARATOR="Now trying"
  

• STEP 4': COPY AFTER BUILD: Optional tag for copy to a distributed file system. The copy is performed with tar. In a the Markup tag the work directory (relative to the release directory $NICOS_RELHOME), desirable shell and copy type is defined. The possible copy types are:
  1. scratch - the target area is cleaned prior the copy start
  2. update - the target area is not cleaned prior the copy start. Some areas are excluded from the copy, for example cmt and InstallArea
  3. scratch_update - update without exclusions
  4. merge - the binaries of another platform are added. This job must be performed by an external script or command in the tag content

  <COPY AFTER BUILD dir=${NICOS_ATLAS_PROJECT}Release/cmt shell=/bin/sh type=scratch>   
  #the additional commands can be added in the content, for example,
  #for fixing soft links in the copied release.
  

• STEP 5: QA TESTS: Command for unit tests are listed.

  <QA TESTS dir=${NICOS_ATLAS_PROJECT}Release/cmt>
  cp -a ${NICOS_RELHOME}/Tools/Scripts/share/checkreq.sh ~/.
  cmt broadcast - ~/checkreq.sh
  NICOS_FULLNAME_INSEPARATOR="yes"
  NICOS_BUILDLOG_SEPARATOR="Now trying"
  

• STEP 6: UNIT TESTS: Command for unit tests are listed. Currently This step is not used.

  <UNIT TESTS dir=.>
  

• STEP 7: INTEGRATED TESTS: Commands for the ATN testing tool are listed.

  <INTEGRATED TESTS dir=NICOS_area/NICOS_qmtest${NICOS_SUFFIX} tool=qmtest> >
  ATN_HOME=/afs/cern.ch/atlas/software/dist/nightlies/atn_12
  NICOS_INTTESTS_FILES=".sh$"
  NICOS_INTTESTS_LOCATION="AtlasTest/TestTools/scripts/nightlies"
  NIGHTLYAREA=${NICOS_RELHOME}
  cd $NICOS_RELHOME/${NICOS_ATLAS_PROJECT}Release/cmt
  source setup.sh
  

  In a Markup tag the directory is defined from which this step is performed. If tool is qmtest then the Python based QMTest tool is used for running tests.

• STEP 7: COPY AFTER TESTS: Optional tag for copy to a distributed file system. In a Markup tag the work directory (relative to the release directory $NICOS_RELHOME), desirable shell and copy type is defined. As this step is for adding files created by tests, the recommended type is "update".

  <COPY AFTER TESTS dir=${NICOS_ATLAS_PROJECT}Release/cmt shell=/bin/sh type=update>
  #the additional commands can be added in the content, for example,
  #for fixing soft links in the copied release (as rsync does not
  #modify soft links automatically).
  

• STEP 8: ERROR ANALYSIS: The following options should be specified.
  1. NICOS_TEST_SUCCESS_PATTERN: this pattern must be present in output of successful integration test.
  2. NICOS_TEST_FAILURE_PATTERN: this pattern must be absent in output of successful integration test.
  3. NICOS_TEST_WARNING_PATTERN: this pattern triggers warning level of alarm
  4. Similar patterns can be indicated for QA and build logfiles checks
  5. NICOS_MAIL: if "yes" then automatic e-mail notifications about problems are sent (to addresses from NICOS database file).

  <ERROR ANALYSIS>
  NICOS_TEST_SUCCESS_PATTERN=" "
  NICOS_TEST_FAILURE_PATTERN="FAILURE "
  NICOS_TEST_WARNING_PATTERN="test warning"
  NICOS_QA_SUCCESS_PATTERN=" "
  NICOS_QA_FAILURE_PATTERN="FAILURE "
  NICOS_QA_WARNING_PATTERN="checkreq warning"
  NICOS_BUILD_FAILURE_PATTERN="ERROR"
  NICOS_BUILD_WARNING_PATTERN="WARNING"
  NICOS_MAIL="no"
  NICOS_MAIL_UNIT_TESTS="no"
  NICOS_MAIL_QA_TESTS="no"
  
  

• STEP 9: WEB PAGE. This step at which web pages with build summaries are generated does not require additional commands or definitions.

  <WEB PAGE>
  

BACK TO TOP

NICOS database file is a text file. Each line contains name of package, tag of package, and e-mail addresses of developers (could be multiple) separated by unlimited number of spaces. For example:

FileCatalog    FileCatalog-0-0-1   developer1@cern.ch developer2@bnl.gov
Utilities/FileCatalog    Utilites-0-0-1   developer1@cern.ch 

There are several options for tags:

1. Exact tag or the first symbols of the tag. In latter case the latest tag with indicated symbols will be selected. For instance if FileCatalog is specified as a tag then the latest tag with name starting with FileCatalog will be selected (e.g. FileCatalog-89-99-99).
2. recent : the recent CVS submission is selected (not necessarily tagged).
3. development : the latest tagged CVS version is selected, the format of tag name is not restricted.
4. official : the latest tag in the format <package name>-[0-9][0-9]-[0-9][0-9]-[0-9][0-9] or <package name>-[0-9]-[0-9]-[0-9] is selected.