WMAgent Scripts¶
Setting Up Scripts¶
Download the Scripts¶
The easiest wy to download the WmAgentScripts is using git on lxplus or your own machine:
git clone https://github.com/CMSCompOps/WmAgentScripts.git
Note
Does this work? I know a lot of things depend on dbsClient. OpsSpace tries to install this, but it’s not tested yet.
Creating Proxy¶
Most of the scripts need to load a proxy, so first you need to setup a certificate: New Operator Setup
On SL6¶
Generate your proxy:
voms-proxy-init -voms cms
Type your key password and should display something like this:
Contacting voms.cern.ch:15002 [/DC=ch/DC=cern/OU=computers/CN=voms.cern.ch] "cms"... Remote VOMS server contacted succesfully. Created proxy in /tmp/x509up_uXXXX. Your proxy is valid until Thu Oct 09 21:53:28 CEST 2014
Export the X509_USER_PROXY variable to the environment (so it can be used by python), use proxy location in the previous step:
export X509_USER_PROXY=/tmp/x509up_uXXXX
This is a one line command for all this procedure:
export X509_USER_PROXY=$(voms-proxy-init -voms cms | grep Created | cut -c18- | tr -d '.')
Loading WMAgent Environment¶
Some of the scripts need WMAgent libraries, which at the moment are only installed on WMAgent machines. (i.e. vocms049 … vocms174) Log in to the machine and type:
source /data/srv/wmagent/current/apps/wmagent/etc/profile.d/init.sh
Scripts that interact with the Request Manager¶
Note
I’ll leave this up to you guys, but here’s an example of how to display the help message.
assign.py¶
Quick request assignment, useful if you want to avoid assigning by Web interface and reqmgr.py is too unflexible.
Usage: assign.py [options] [WORKFLOW]
Options:
-h, --help show this help message and exit
-t TEAM, --team=TEAM Type of Requests
-s SITES, --sites=SITES
"t1" for Tier-1's and "t2" for Tier-2's
--special=SPECIAL Use it for special workflows. You also have to change
the code according to the type of WF
-r, --replica Adds a _Disk Non-Custodial Replica parameter
-p PROCVERSION, --procversion=PROCVERSION
Processing Version, if empty it will leave the
processing version that comes by default in the
request
-a ACTIVITY, --activity=ACTIVITY
Dashboard Activity (reprocessing, production or test),
if empty will set reprocessing as default
--xrootd Assign with TrustSitelists=True (allows xrootd
capabilities)
--no_xrootd Assign with TrustSitelists=False
--secondary_xrootd Assign with TrustPUSitelists=True (allows xrootd
capabilities)
--no_secondary_xrootd
Assign with TrustPUSitelists=False
-l LFN, --lfn=LFN Merged LFN base
-v, --verbose Verbose
--testbed Assign in testbed
--test Nothing is injected, only print infomation about
workflow and Era
-f FILE, --file=FILE Text file with a list of wokflows. If this option is
used, the same settings will be applied to all
workflows
-w WORKFLOW, --workflow=WORKFLOW
Workflow Name, or coma sperated list
-m MEMORY, --memory=MEMORY
Set the Memory parameter to the workflow
--lumisperjob=LUMISPERJOB
Set the number of lumis per job
--maxmergeevents=MAXMERGEEVENTS
Set the number of event to merge at max
-c MULTICORE, --multicore=MULTICORE
Set the multicore parameter to the workfllow
-e ERA, --era=ERA Acquistion era
--procstr=PROCSTRING Overrides Processing String with a single string
--checksite
Note
- You can provided the -l LFN parameter, be careful to assign the proper one MergeLFN. By default the script will assign the lfn from que original request asociated, in case of ACDCs.
- You can use this script to assign any kind of workflow.
- You can use a text file to assign multiple workflows at the same time.
- You may use additional options to:
- enforce disk replica subscription
- change dashboard activity
- change processing version
- fix a processing string or acquisition era
- You can also provide a list of sites separated by commas (no spaces) T1_US_FNAL,T2_US_UCSD,…
- You can use -s acdc: It will assign to the sites taken from the ACDC server.
- You can use -s all: It will assign to all sites available (Works for any taskchain acdc).
- You can skip -s option: It will assign to the “good site” list (Works for any clone you need).
changePriorityWorkflow.py¶
Changes priority of a workflow or a list of workflows
Usage: Usage changePriorityWorkflow.py [WF1 WF2 ... | -f FILE] PRIO
Options:
-h, --help show this help message and exit
-f FILE, --file=FILE Text file
changeSplittingWorkflow.py¶
This script allows to change the splitting of a request, on a given task name
Usage: python changeSplittingWorkflow.py [-t TYPE| -e | -l | -a] WORKFLOW TASKPATH VALUE
Options:
-h, --help show this help message and exit
-t TYPE, --type=TYPE Type of splitting (event, lumi, eventaware or merge),
or use the other options
-e Use EventBased splitting
-l, --lumi Use
-a, --eventaware Use EventAwareLumiBased
-m, --merge Splitting for Merge tasks
Note
- The TASKPATH should be the full task path in which you want to change the splitting, i.e. StepOneProc, StepOne /StepOneProcMerge, Production, etc.
- The TYPE is the algorithm for splitting.
forceCompleteWorkflows.py¶
Moves a workflow or list of workflows from running-closed to force-completed. This causes every production job to be aborted leaving only log-collect jobs and cleanups
Force-complete a list of workflows.
Usage: python forceCompleteWorkflows.py [WF1 WF2 ... | -f FILE]
Options:
-h, --help show this help message and exit
-f FILE, --file=FILE Text file
makeACDC.py¶
Simple creating of acdc’s Use: The workflow name and the initial task name. It will copy all the original workflow parameters unless specified
no DB connection
Usage: makeACDC.py (-w workflow|-f filelist) (-t TASK|--all) [--tesbed]
Options:
-h, --help show this help message and exit
-f FILE, --file=FILE Text file with a list of workflows
-w WORKFLOW, --workflow=WORKFLOW
Coma separated list of wf to handle
-t TASK, --task=TASK Coma separated task to be recovered
-p PATH, --path=PATH Coma separated list of paths to recover
-a, --all Make acdc for all tasks to be recovered
-m MEMORY, --memory=MEMORY
Memory to override the original request memory
-c MCORE, --mcore=MCORE
Multicore to override the original request multicore
--testbed
Note
- Before creation, ACDC documents should be already in couch (usually, it happens when the workflow is completed).
- If you want to create all possible ACDCs given a workflow, add the option –all.
- If you need to create an ACDC for an specifc task, you need to have the full task path (not just the last part), i.e. for a workflow with StepOneProc and StepTwoProc:
- If you want to create an ACDC on StepTwo the taskname is StepOneProc /StepOneProcMerge/StepTwoProc* (or something similar).
recoverMissingLumis.py¶
For recovering a list of missing lumis on a workflow with input dataset. For detailed information please go here https://twiki.cern.ch/twiki/bin/view/CMS/CompOpsPRWorkflowTrafficController#Recovering_Workflows
Traceback (most recent call last):
File "../WmAgentScripts/recoverMissingLumis.py", line 3, in <module>
from WMCore.ACDC.CouchCollection import CouchCollection
ImportError: No module named WMCore.ACDC.CouchCollection
reject.py¶
The script allows us to reject or abort (regarding its state) a workflow, or a set of them
Usage:
python reject.py [options] [WORKFLOW_NAME]
WORKFLOW_NAME: if the list file is provided this should be empty
Options:
-h, --help show this help message and exit
-f FILE, --file=FILE Text file of workflows to Reject and Clone
-c, --clone Are the workflows going to be cloned? The default
value is False
-i, --invalidate Invalidate datasets? The default value is False
-u USER, --user=USER The user for creating the clone, if empty it will use
the OS user running the script
-g GROUP, --group=GROUP
The group for creating the clone, if empty it will,
use 'DATAOPS' by default
-m MEMORY, --memory=MEMORY
Set max memory for the clone. At assignment, this will
be used to calculate maxRSS = memory*1024
reqMgrClient.py¶
This client encapsulates several basic queries to request manager. This uses ReqMgr rest api through HTTP url parameter is normally ‘cmsweb.cern.ch’
Usage: reqMgrClient -j <file.json>
Options:
-h, --help show this help message and exit
-j JSON, --json=JSON A json file with "createRequest" or just a request
schema
--testbed Create in testbed
--test Just print the schema
--no_approve In case you do not want the workflow in assignment-
approved right away
resubmit.py¶
This script clones and resubmits a workflow lying either in production or testbed. Be careful with this one, it is being used by reject.py
Usage:
python resubmit.py [options] [WORKFLOW_NAME]WORKFLOW_NAME: if the list file is provided this should be empty
Options:
-h, --help show this help message and exit
-a ACTION, --action=ACTION
There are two options clone (clone) or extend a
worflow (extend) .
-u USER, --user=USER User we are going to use
-g GROUP, --group=GROUP
Group to send the workflows.
-b, --backfill Creates a clone for backfill test purposes.
-v, --verbose Prints all query information.
-f FILE, --file=FILE Text file with a list of workflows
--bwl=BWL The block white list to be used
-e EVENTS, --events=EVENTS
# of events to add
-l FIRSTLUMI, --firstlumi=FIRSTLUMI
# of the first lumi
-m MEMORY, --memory=MEMORY
Set max memory for the event. At assignment, this will
be used to calculate maxRSS = memory*1024
--TimePerEvent=TIMEPEREVENT
Set the TimePerEvent on the clone
--testbed Clone to testbed reqmgr insted of production
Note
- The workflow is created but NOT assigned, if you need to get it running, follow the instructions here: assign.py
- When you use the -b option at the end, the script will add the particle “Backfill” to the requestString, AcquisitionEra, Campaing and ProcessingString, so it can be correctly identified as backfill.