Installation¶

The installation of Comp Ops tools and scripts can be facilitated by the OpsSpace repository:

git clone https://github.com/CMSCompOps/OpsSpace.git
cd OpsSpace

Full installation is done by using the setup.py script. Since users of these tools are often also developing scripts inside, this setup script (optionally) does not compile and install packages like a typical setup.py. The other method of installation follows these basic steps, which can also be done by hand, or by running setup.py with various options described below:

The package is downloaded through a given user’s repository on GitHub or through the CMSCompOps repository if no user is given.
setup.py searches for requirements.txt immediately inside any package and tries to install the contents through pip.
Any script that matches the pattern install.?? inside the package is then run.
The script appends the OpsSpace directory to the user’s $PYTHONPATH if the appropriate flag is passed.

The current list of valid packages and options are shown by running with no arguments:

./setup.py

This gives:

Usage: ./setup.py [options] package1 package2 ... 
         --or-- 
       ./setup.py install [--force]

Options:
  -h, --help            show this help message and exit
  -u UserName, --user-name=UserName
                        GitHub user name, where the packages will be searched
                        for first (default CMSCompOps)
  -p, --add-path        Add the location of this package to the user's
                        PYTHONPATH in their .bashrc or .bash_profile. The
                        default behavior is to let the user adjust PYTHONPATH
                        on their own. This option can be run without selecting
                        any packages.
  --force               Install all possible packages when running ./setup.py
                        install. --force used since readthedocs tries it.

Valid package names:

  WorkflowWebTools
  SiteAdminToolkit
  WmAgentScripts
  dynamo
  dynamo-consistency

To install a non-listed package as a submodule of OpsSpace, Make sure that the package is in either the CMSCompOps or your own GitHub account (preferably both). Add that repository name to the file PackageList.txt in the OpsSpace root directory. Now your repository will be considered a valid package. Then continue with the setup. Sending a pull request to the CMSOpsSpace/OpsSpace repository will place your repository on the Tools & Integration radar, and they will help you to standardize, document, and test your code.

Example Setup¶

If I want to freshly install the WorkflowWebTools from dabercro’s GitHub on a new machine, I simply do the following:

./setup.py -u dabercro WorkflowWebTools

The WorkflowWebTools exists in both the CMSCompOps group and dabercro’s fork, so the setup script will clone from dabercro. The origin repository will be set to https://github.com/dabercro/WorkflowWebTools.git and the remote repository CMSCompOps assumes a centralized version exists and will be added:

cd WorkflowWebTools && git remote -v

CMSCompOps  https://github.com/CMSCompOps/WorkflowWebTools.git (fetch)
CMSCompOps  https://github.com/CMSCompOps/WorkflowWebTools.git (push)
origin      https://github.com/dabercro/WorkflowWebTools.git (fetch)
origin      https://github.com/dabercro/WorkflowWebTools.git (push)

Note

Both remote repositories will fetch and push over https, so you will need to manually convert the remote origin to ssh, if desired.

If the dabercro repository did not exist, then the repository would be cloned from CMSCompOps. If the -u option is left blank, then only the CMSCompOps repository is checked.

After downloading desired repositories in this way, you can either install by adding <path/to/inside>/OpsSpace to your $PYTHONPATH, or through standard distutils behavior under the install subcommand. If you are okay with editing your ~/.bashrc or ~/.bash_profile, then you can automatically add to $PYTHONPATH by running with the -p option:

./setup.py -p

Or you can run the following, which uses the standard distutils package:

./setup.py install

This first is the recommended method of installation since most users will also be developing or adjusting their tools as they use them.

Todo

Handle updating (git pull, etc.) inside setup.py properly.

Troubleshooting¶

A list of problems and solutions that come up during OpsSpace installation and usage should be placed here.

ImportError when using installed pycurl¶

pycurl is needed for the DBS package, which is installed with OpsSpace. It is possible for pycurl to be compiled using the wrong ssl backend. Then when trying to import dbs.apis.dbsClient, you might get the following error:

ImportError: pycurl: libcurl link-time ssl backend (nss) is different from compile-time ssl backend (none/other)

To fix this, uninstall pycurl and reinstall, setting the environment variable PYCURL_SSL_LIBRARY to match the link-time ssl backend:

pip uninstall pycurl
PYCURL_SSL_LIBRARY=nss pip install pycurl --no-cache-dir

Other systems may use openssl as the backend. See here for more details.

Reference¶

The setup script can also be imported to gain access to the following members, but that is an unlikely use case. However, I will add this so that OpsSpace/docs/setup.rst can be used as an example for someone documenting their own scripts or modules.

File mostly intended to set up working environment for operator as a script.

author:	Daniel Abercrombie <dabercro@mit.edu>

class setup.Installer(github_user=None)[source]¶

Class the holds the information for installing workspace

CentralGitHub = 'CMSCompOps'¶: Default GitHub account containing packages that can be installed

InstallDirectory = '/home/docs/checkouts/readthedocs.org/user_builds/cms-comp-ops-tools/checkouts/v0.7'¶: Location of the OpsSpace

ValidPackages = ['WorkflowWebTools', 'SiteAdminToolkit', 'WmAgentScripts', 'dynamo', 'dynamo-consistency']¶: List of valid packages that can be installed with this script. Set in set_packages().

add_pythonpath()[source]¶: Appends modified $PYTHONPATH variable to bash profile

gitHubUrl = 'https://github.com/'¶: Remote repository location

install_package(package_name)[source]¶

Install a given package into the operator workspace.

If the file requirements.txt exists inside of the package, the requirements are installed through pip.

Parameters:	package_name (str) – must match the repository name in the valid packages list and GitHub

install_packages(package_list)[source]¶: Calls install_package for a list of packages

install_requirements(package_name)[source]¶

Look for requirements.txt and install requirements, if needed. Also runs any install.?? script in the package

Parameters:	package_name (str) – is the directory that will be searched for the requirements.txt and install.?? file

possibleProfiles = ['.bashrc', '.bash_profile']¶: List of profiles searched for in user’s home to append to PYTHONPATH

print_valid_packages()[source]¶: Displays valid packages for the user

set_packages()[source]¶: Read from list of valid package and append to valid list.

setup.main()[source]¶

Main functionality of the install script.

Uses system arguments to determine which packages to install.