CMSToolBox Modules¶

The CompOps tool box contains a number of modules that may be useful for multiple projects and tools. Most of the functions in here will be (hopefully) monitoring tools that are maintained by the experts and used by the masses.

Note

At the moment, we just have some basic tools that are used by the setup script and WorkflowWebTools. Edit OpsSpace/docs/toolbox.rst to add your module to this page!

Fetching JSON Files
Cached Dashboard Information
Site Info
Site Readiness Metric
SAM Status History
E-mail Tools
Simple File Manipulation
Workflow Information
Loading Test Paths

Fetching JSON Files ¶

A high-level module for getting JSON information from web services. The HTTP vs HTTPS and certificate handling is done here.

author:	Daniel Abercrombie <dabercro@mit.edu>

cmstoolbox.webtools.get_cookie_header(url, cookie_file, pem=None, key=None, refresh_time=None)¶

Get the header infomation that meeds to be passed to the server. This function by default requires a .rsa key inside your ~/.globus directory. To generate this run the following commands from ~/.globus:

openssl rsa -in userkey.pem -out userkey.rsa
chmod 400 userkey.rsa

This function also requires cern-get-sso-cookie to be installed on your machine.

Parameters:	url (str) – The location to get the cookie from cookie_file (str) – The location where to store the cookie pem (str) – The location of the `.pem` file (default `~/.globus/usercert.pem`) key (str) – The location of the `.rsa` key (default `~/.globus/userkey.rsa`) refresh_time (float) – The number of seconds until the next time the cookie is generated (default to three hours)
Returns:	The cookie string for each host to be set as the `'Cookie'` element of the request header
Return type:	dict

cmstoolbox.webtools.get_json(host, request, params='', body='', headers=None, port=None, retries=3, **kwargs)¶

Function for getting JSON from a URL that handles the connection and certificates with just a couple of bools.

Parameters:	host (str) – The name of the host to connect to request (str) – The request to make to the host params (dict) – The parameters to pass to the request body (str) – The body to send in a POST request headers (dict) – Headers to pass to request. If `None`, `{'Accept': 'application/json'}` will be passed. port (int) – The port to access, if a not default value retries (int) – The number of times to try to get the JSON response before giving up kwargs – Additional aruments that can be used to change the connection behavior. These are listed below: use_https (bool) - Uses HTTP connection by default use_cert (bool) - Does not pass a certificate by default cert_file (str) - Default is `$X509_USER_PROXY`. use_post (bool) - Determines whether to use POST or GET. cookie_file (str) - Location of a Shibboleth cookie to pass in header cookie_pem (str) - Location of `.pem` file to generate cookie cookie_key (str) - Location of `.rsa` key to generate cookie cookie_time (float) - Time, in seconds, until generating a new cookie
Returns:	The JSON from the query
Return type:	dict

Cached Dashboard Information ¶

A module to gather and hold information from the dashboard.

author:	Jean-Roch Vlimant <jean-roch.vlimant@cern.ch>

class cmstoolbox.dashboard.DocCache¶

This class gathers various information, mostly from the dashboard when it’s requested. The information is cached with slightly random lifetimes to allow requests to be staggered. All of the information is obtained through system curl commands.

get(label, fresh=False)¶

This is the function that should be used to access data from various monitors of CMS. The DocCache class stores a cache, for the various information which is only filled if requested and updates every 20 to 30 minutes. The times are staggered to avoid making all of the calls for information at the same time.

Todo

Create descriptions for all of the members of the cache.

Parameters:

label (str) –
dictionary key of cache for the information desired. The list of valid labels and their description.
- T0_CH_CERN_MSS_usage - Node usage for T0_CH_CERN_MSS
- T1_DE_KIT_MSS_usage - Node usage for T1_DE_KIT_MSS
- T1_ES_PIC_MSS_usage - Node usage for T1_ES_PIC_MSS
- T1_FR_CCIN2P3_MSS_usage - Node usage for T1_FR_CCIN2P3_MSS
- T1_IT_CNAF_MSS_usage - Node usage for T1_IT_CNAF_MSS
- T1_RU_JINR_MSS_usage - Node usage for T1_RU_JINR_MSS
- T1_UK_RAL_MSS_usage - Node usage for T1_UK_RAL_MSS
- T1_US_FNAL_MSS_usage - Node usage for T1_US_FNAL_MSS
- detox_sites - no description
- gwmsmon_prod_site_summary - no description
- gwmsmon_site_summary - no description
- gwmsmon_totals - no description
- hlt_cloud - no description
- mcore_ready - no description
- mss_usage - no description
- ssb_106 - column 106 in the dashboard
- ssb_107 - column 107 in the dashboard
- ssb_108 - column 108 in the dashboard
- ssb_109 - column 109 in the dashboard
- ssb_136 - column 136 in the dashboard
- ssb_158 - column 158 in the dashboard
- ssb_159 - column 159 in the dashboard
- ssb_160 - column 160 in the dashboard
- ssb_237 - The Site Readiness
fresh (bool) – forces the cache to be refreshed if true. Otherwise, the cache timestamp is checked.

Returns:

the values of the given cache if a correct label is given, otherwise returns None. The cache is returned as a dictionary with the following keys.

data - is the data for a given cache. These have different forms of being either a list, a dict, or a str.
timestamp - is the time that the cache was last downloaded or when the object was created.
expiration - is the time in seconds that the entry stays until it is redownloaded. It is a random time between 20 to 30 minutes.
getter - is a function with zero parameters that will download and return the data corresponding to this column
cachefile - is the location of a local file where the cache is saved.
default - is the default value of the cache. This should be an empty variable of the appropriate type.
description - is the description of the cache member. This is used to generate the documentation for the label parameter.

Return type:

dict or None

cmstoolbox.dashboard.get_node_usage(url, node)¶

Get the disk function useage at a given site.

Note

This function only works on systems that have the environment variable X509_USER_PROXY defined. Without this function, the cached labels that are formed like <site>_usage cannot be refreshed.

Parameters:	url (str) – is the URL of the reporting service node (str) – is the site name to check
Returns:	The disk usage at the site in TB, or None if node info is not found
Return type:	int

Site Info ¶

A module that returns various information about sites

author:	Daniel Abercrombie <dabercro@mit.edu>

cmstoolbox.siteinfo.get_domain(site)¶: Get the domain name of a given site for use in hostname matching.

Note

Is currently a simple map, and we’d love for someone to come along and improve it.

cmstoolbox.siteinfo.get_site(host)¶: Get the site of a given hostname

Note

Is currently a simple map, and we’d love for someone to come along and improve it.

cmstoolbox.siteinfo.pfn_from_phedex(site_name, lfn)¶

Get the PFN of a directory from phedex for a given LFN.

Parameters:	site_name (str) – is the name of the site to check lfn (str) – is the LFN to find
Returns:	PFN of the folder
Return type:	str

Site Readiness Metric ¶

Module the caches and holds the Site Readiness status

Todo

Try to get the average uptime.

author:	Daniel Abercrombie <dabercro@mit.edu>

cmstoolbox.sitereadiness.i_site_readiness()¶

Iterates over site readiness for the user

Returns:	iterator tuple with site, readiness, and drain status
Return type:	generator

cmstoolbox.sitereadiness.site_drain_status(site_name)¶

Returns the drain status for a given site

Parameters:	site_name (str) – Name of the site
Returns:	Readiness status. Possibilities and their meanings are: ’enabled’: The site is running ’disabled’: The site is not running ’drain’: The jobs at the site are being drained ’test’: This site is being commissioned or something? ’none’: Site not found
Return type:	str

cmstoolbox.sitereadiness.site_list()¶

Returns the list of sites

Returns:	The sorted list of site names
Return type:	list

cmstoolbox.sitereadiness.site_readiness(site_name)¶

Returns the readiness status for a given site

Parameters:	site_name (str) – Name of the site
Returns:	Readiness status. Possibilities and their meanings are: ’green’: Ready ’yellow’: Waiting Room ’red’: Morgue ’none’: Site not found
Return type:	str

SAM Status History ¶

A module that checks the SAM status of a site

authors:

Max Goncharov <maxi@mit.edu>

Daniel Abercrombie <dabercro@mit.edu>

cmstoolbox.samstatus.is_sam_good(site, time_span=24, success=0.85)¶

Checks the SAM tests for success rate, and returns True if SAM tests were passing

Parameters:	site (str) – Name of the site to check time_span (int) – Amount of time to check in hours success (float) – Fraction of success desired in timespan for site
Returns:	True if SAM tests are good, otherwise, False
Return type:	bool

E-mail Tools ¶

Contains methods for sending emails.

author:	Daniel Abercrombie <dabercro@mit.edu>

cmstoolbox.emailtools.send_email(sender, recipients, subject, message_text, message_html='', method='sendmail', **kwargs)¶

Sends an email to someone

Parameters:

sender (str) – Who the email is from
recipients (str or list) – Who to send the email to
subject (str) – The subject of the email
message_text (str) – The text version of the message
message_html (str) – The html version of the message
method (str) –
Method of sending email, based on utilities available. Valid entries are:
- ’smtplib’: uses the default stplib.SMTP()
- ’sendmail’: uses the system sendmail command
kwargs –
Keyword arguments for setting additional values depending on the email method:
- ’smtplib’: Keyword arguments can be ‘host’ and ‘port’
- ’sendmail’: No keyword arguments are supported

Simple File Manipulation ¶

Contains methods for manipulating txt files and sourcing bash scripts.

author:	Daniel Abercrombie <dabercro@mit.edu>

cmstoolbox.simplefiletools.append_to_file(file_name, lines)¶

Appends lines to a file. This function may be useful for the exceedingly lazy.

Parameters:	file_name (str) – is the file to append to lines (str or list) – is the list of lines or a single line to append to the file

cmstoolbox.simplefiletools.load_env(configs)¶

Sources bash files and loads the resulting environment into os.environ

Parameters:	configs (str or list) – is a list of file names that should be sourced. A string for a single configuration file is also accepted.

Workflow Information ¶

Module containing and returning information about workflows.

authors:	Daniel Abercrombie <dabercro@mit.edu>

class cmstoolbox.workflowinfo.Info¶

Implements shared operations on the cache

cache_filename(attribute)¶

Return the name of the file for caching

Parameters:	attribute (str) – The information to store in the file
Returns:	The full file name to store the cache
Return type:	str

reset()¶: Reset the cache for this object and clear out the files.

class cmstoolbox.workflowinfo.PrepIDInfo(prep_id, url='cmsweb.cern.ch')¶

A class that just holds a small amount of information about a given PrepID.

get_requests(*args, **kwargs)¶

Returns:	The requests for the Prep ID from ReqMgr2 API
Return type:	dict

get_workflows()¶

Returns:	A list of tuples containing (workflow name, timestamp of request)
Return type:	list

get_workflows_requesttime()¶

Returns:	A list of tuples containing (workflow name, timestamp of request)
Return type:	list

class cmstoolbox.workflowinfo.WorkflowInfo(workflow, url='cmsweb.cern.ch')¶

Class that holds methods for accessing various information about a workflow.

get_errors(*args, **kwargs)¶

A wrapper for errors_for_workflow() if you happen to have a WorkflowInfo object already.

Parameters:	get_unreported (bool) – Get the unreported errors from ACDC server
Returns:	a dictionary containing error codes in the following format: {step: {errorcode: {site: number_errors}}}
Return type:	dict

get_explanation(errorcode, step='')¶

Gets a list of error logs for a given error code.

Parameters:	errorcode (str) – The error code to explain step (str) – The full name of the step to return explanations from
Returns:	list of error logs
Return type:	list

get_prep_id()¶

Returns:	the PrepID for this workflow
Return type:	str

get_recovery_info(*args, **kwargs)¶

Get the recovery info for this workflow.

Returns:	a dictionary containing the information used in recovery. The keys in this dictionary are arranged like the following: { task: { 'sites_to_run': list(sites), 'missing_to_run': int() } }
Return type:	dict

get_workflow_parameters(*args, **kwargs)¶

Get the workflow parameters from ReqMgr2, or returns a cached value. See the ReqMgr 2 wiki for more details.

Returns:	Parameters for the workflow from ReqMgr2.
Return type:	dict

site_to_run(task)¶

Gets a list of sites that a task in the workflow can run at

Parameters:	task (str) – The full name of the task to find sites for
Returns:	a list of site to run at
Return type:	list

cmstoolbox.workflowinfo.cached_json(attribute, timeout=None)¶

A decorator for caching dictionaries in local files.

Parameters:	attribute (str) – The key of the `WorkflowInfo` cache to set using the decorated function. timeout (int) – The amount of time before refreshing the JSON file, in seconds.
Returns:	Function decorator
Return type:	func

cmstoolbox.workflowinfo.errors_for_workflow(workflow, url='cmsweb.cern.ch')¶

Get the useful status information from a workflow

Parameters:

workflow (str) – the name of the workflow request
url (str) – the base url to find the information at

Returns:

a dictionary containing error codes in the following format:

{step: {errorcode: {site: number_errors}}}

Return type:

dict

cmstoolbox.workflowinfo.explain_errors(workflow, errorcode)¶

Get example errors for a given workflow and errorcode

Parameters:	workflow (str) – is the workflow name errorcode (str) – is the error code
Returns:	a dict of log snippets from different sites.
Return type:	list

cmstoolbox.workflowinfo.list_workflows(status)¶

Get the list of workflows currently in a given status. For a list of valid requests, visit the Request Manager Interface.

Parameters:	status (str) – The status of the workflow lists being looked for
Returns:	A list of workflows matching the status
Return type:	list

Loading Test Paths ¶

This module is used to add directories listed in various packages’ test/path.txt file to the system path. If imported from a directory with a file called path.txt, it will assume that it is inside of a packages test directory and modify the system path to add those directories too. This is done by calling add_path('.') at the end of this module. In general, this package should not be used for loading modules. Its purpose is to temporarily put python scripts into the system’s path so the scripts can be imported for documentation and unit tests. If your package consists of modules that are regularly imported, lay out your repository appropriately to live in the OpsSpace directory.

author:	Daniel Abercrombie <dabercro@mit.edu>

cmstoolbox._loadtestpath.add_path(package)¶

Add the directories for a given package to Python’s import path. The default behavior is to look for ../<package>/test/path.txt. For each directory listed in that file, ../<package>/<dir> will be added to the system path. The reason for the leading .. is that the module is designed to be called from within the OpsSpace/docs directory or a package’s test directory. Paths are appended to the front of the path, so directories at the bottom of the test/path.txt will be loaded first.

Parameters:	package (str) – Is the name of the package to load the test path for.