CMSToolBox Modules

The CompOps tool box contains a number of modules that may be useful for multiple projects and tools. Most of the functions in here will be (hopefully) monitoring tools that are maintained by the experts and used by the masses.

Note

At the moment, we just have some basic tools that are used by the setup script and WorkflowWebTools. Edit OpsSpace/docs/toolbox.rst to add your module to this page!

Fetching JSON Files

A high-level module for getting JSON information from web services. The HTTP vs HTTPS and certificate handling is done here.

author:Daniel Abercrombie <dabercro@mit.edu>

Get the header infomation that meeds to be passed to the server. This function by default requires a .rsa key inside your ~/.globus directory. To generate this run the following commands from ~/.globus:

openssl rsa -in userkey.pem -out userkey.rsa
chmod 400 userkey.rsa

This function also requires cern-get-sso-cookie to be installed on your machine.

Parameters:
  • url (str) – The location to get the cookie from
  • cookie_file (str) – The location where to store the cookie
  • pem (str) – The location of the .pem file (default ~/.globus/usercert.pem)
  • key (str) – The location of the .rsa key (default ~/.globus/userkey.rsa)
  • refresh_time (float) – The number of seconds until the next time the cookie is generated (default to three hours)
Returns:

The cookie string for each host to be set as the 'Cookie' element of the request header
Return type:

dict
cmstoolbox.webtools.get_json(host, request, params='', body='', headers=None, port=None, retries=3, **kwargs)

Function for getting JSON from a URL that handles the connection and certificates with just a couple of bools.

Parameters:
  • host (str) – The name of the host to connect to
  • request (str) – The request to make to the host
  • params (dict) – The parameters to pass to the request
  • body (str) – The body to send in a POST request
  • headers (dict) – Headers to pass to request. If None, {'Accept': 'application/json'} will be passed.
  • port (int) – The port to access, if a not default value
  • retries (int) – The number of times to try to get the JSON response before giving up
  • kwargs –

    Additional aruments that can be used to change the connection behavior. These are listed below:

    • use_https (bool) - Uses HTTP connection by default
    • use_cert (bool) - Does not pass a certificate by default
    • cert_file (str) - Default is $X509_USER_PROXY.
    • use_post (bool) - Determines whether to use POST or GET.
    • cookie_file (str) - Location of a Shibboleth cookie to pass in header
    • cookie_pem (str) - Location of .pem file to generate cookie
    • cookie_key (str) - Location of .rsa key to generate cookie
    • cookie_time (float) - Time, in seconds, until generating a new cookie
Returns:

The JSON from the query
Return type:

dict

Site Info

A module that returns various information about sites

author:Daniel Abercrombie <dabercro@mit.edu>
cmstoolbox.siteinfo.get_domain(site)

Get the domain name of a given site for use in hostname matching.

Note

Is currently a simple map, and we’d love for someone to come along and improve it.

cmstoolbox.siteinfo.get_site(host)

Get the site of a given hostname

Note

Is currently a simple map, and we’d love for someone to come along and improve it.

cmstoolbox.siteinfo.pfn_from_phedex(site_name, lfn)

Get the PFN of a directory from phedex for a given LFN.

Parameters:
  • site_name (str) – is the name of the site to check
  • lfn (str) – is the LFN to find
Returns:

PFN of the folder
Return type:

str

Site Readiness Metric

Module the caches and holds the Site Readiness status

Todo

Try to get the average uptime.

author:Daniel Abercrombie <dabercro@mit.edu>
cmstoolbox.sitereadiness.i_site_readiness()

Iterates over site readiness for the user

Returns:iterator tuple with site, readiness, and drain status
Return type:generator
cmstoolbox.sitereadiness.site_drain_status(site_name)

Returns the drain status for a given site

Parameters:site_name (str) – Name of the site
Returns:Readiness status. Possibilities and their meanings are:
  • ’enabled’: The site is running
  • ’disabled’: The site is not running
  • ’drain’: The jobs at the site are being drained
  • ’test’: This site is being commissioned or something?
  • ’none’: Site not found
Return type:str
cmstoolbox.sitereadiness.site_list()

Returns the list of sites

Returns:The sorted list of site names
Return type:list
cmstoolbox.sitereadiness.site_readiness(site_name)

Returns the readiness status for a given site

Parameters:site_name (str) – Name of the site
Returns:Readiness status. Possibilities and their meanings are: - ‘green’: Ready - ‘yellow’: Waiting Room - ‘red’: Morgue - ‘none’: Site not found
Return type:str

SAM Status History

A module that checks the SAM status of a site

authors:

Max Goncharov <maxi@mit.edu>

Daniel Abercrombie <dabercro@mit.edu>
cmstoolbox.samstatus.is_sam_good(site, time_span=24, success=0.85)

Checks the SAM tests for success rate, and returns True if SAM tests were passing

Parameters:
  • site (str) – Name of the site to check
  • time_span (int) – Amount of time to check in hours
  • success (float) – Fraction of success desired in timespan for site
Returns:

True if SAM tests are good, otherwise, False
Return type:

bool

E-mail Tools

Contains methods for sending emails.

author:Daniel Abercrombie <dabercro@mit.edu>
cmstoolbox.emailtools.send_email(sender, recipients, subject, message_text, message_html='', method='sendmail', **kwargs)

Sends an email to someone

Parameters:
  • sender (str) – Who the email is from
  • recipients (str or list) – Who to send the email to
  • subject (str) – The subject of the email
  • message_text (str) – The text version of the message
  • message_html (str) – The html version of the message
  • method (str) –

    Method of sending email, based on utilities available. Valid entries are:

    • ’smtplib’: uses the default stplib.SMTP()
    • ’sendmail’: uses the system sendmail command
  • kwargs –

    Keyword arguments for setting additional values depending on the email method:

    • ’smtplib’: Keyword arguments can be ‘host’ and ‘port’
    • ’sendmail’: No keyword arguments are supported

Simple File Manipulation

Contains methods for manipulating txt files and sourcing bash scripts.

author:Daniel Abercrombie <dabercro@mit.edu>
cmstoolbox.simplefiletools.append_to_file(file_name, lines)

Appends lines to a file. This function may be useful for the exceedingly lazy.

Parameters:
  • file_name (str) – is the file to append to
  • lines (str or list) – is the list of lines or a single line to append to the file
cmstoolbox.simplefiletools.load_env(configs)

Sources bash files and loads the resulting environment into os.environ

Parameters:configs (str or list) – is a list of file names that should be sourced. A string for a single configuration file is also accepted.

Loading Test Paths

This module is used to add directories listed in various packages’ test/path.txt file to the system path. If imported from a directory with a file called path.txt, it will assume that it is inside of a packages test directory and modify the system path to add those directories too. This is done by calling add_path('.') at the end of this module. In general, this package should not be used for loading modules. Its purpose is to temporarily put python scripts into the system’s path so the scripts can be imported for documentation and unit tests. If your package consists of modules that are regularly imported, lay out your repository appropriately to live in the OpsSpace directory.

author:Daniel Abercrombie <dabercro@mit.edu>
cmstoolbox._loadtestpath.add_path(package)

Add the directories for a given package to Python’s import path. The default behavior is to look for ../<package>/test/path.txt. For each directory listed in that file, ../<package>/<dir> will be added to the system path. The reason for the leading .. is that the module is designed to be called from within the OpsSpace/docs directory or a package’s test directory. Paths are appended to the front of the path, so directories at the bottom of the test/path.txt will be loaded first.

Parameters:package (str) – Is the name of the package to load the test path for.