# ScriptingInHEP

Scripting In HEP - LHC

NOTE To get all the functionality of the TWIKI, please remove the http:// protection of your browser! (shown here below for Firefox and Chrome)

 Table of Content

Important Note: To use the Table links, please expand the days below before

 The objective of this page is to deliver a tutorial in scripting with focus in analysis of LHC data into the ATLAS Collaboration.

# Introduction: Automation

Picking into the large amount of different concepts with can find, an useful definition can be: The use of tools, techniques and ingenuous in other to create algorithms that allow you to request to the computer to perform a series of complex tasks, in a sequential mode, for an single execution or with a deterministic frequency, in an automatic way with a single execution o command line.

## What is a script?

It's the translation to a particular language of the set of instructions resulted of the automation algorithm's development for what you need and/or want this automatic execution(s),

## Hi!, this is a set of commands into a script:

setup some variables

## This is a comment

source a set of other scripts

## This is another comment

set your input and output files and locations

## Finally

execute your main program of set of applications

## Done


## Why to use scripts?

Scripts are used to put to work the computers for you.
They are a set of instructions that allows to perform repetitive operations in an automatic way. So, just your imagination can define the limits of a script. Starting for simple checks into your current working area, passing for the setup of simple of complex environments, or for backup creation to interactions with web-based (internet) resources, including the integration of databases, online monitoring of resources to... malware... the reason to have scripts is basically to get something done as far and human-independent as possible. And as we will see, we can wrap several different applications and tools into a single set of instructions, so, it is like to mix many different software into a single box:

## Scripting Languages and the OS as environment

Scripts are not restricted to a single computer language. In fact we will see that we can write scripts in several languages (and for several OS) as soon as we can reach the tools we need.
At this point, we will focus the attention in bash and python scripts since their use in Linux OS is very well known and in particular for HEP-ATLAS analysis, they are supported by a large community. The tools that we use in ATLAS for management of data samples, coding, sharing and final publications production pass through C++ and other several open source applications that have command lines interface support.

A list of useful (but far to be completed) of kernels for scripts:

Language/Kernel Covered here? Use in HEP?
... ... ...
bash yes yes
C++ a bit yes
JavaScript not yes
PowerShell not not?
Python yes yes

## Q&A

?? Do you have any question so far?

# Anatomy of a first script (bash)

#!/bin/bash

## The line above helps to let the system know that you are asking for the bash shell

## As you can see, using #'s allows you to introduce comments.
## Comments are EXTREMELY useful for self documentation of the code!
## Even if you never share the script (or you code in general) you will thank yourself after few weeks/months and you come back to check what you actually did :)
## And if you share your code, is a nice way to let your user or college know that you follow one of the Good Practices in Programming and Computer Science in general

## A first sugestion is to add to every script you plan to keep for more that few hours:
## - Date of last update: xx/yy/zz
## - Contact email

## Saying all that  (excuse me) lets create a first script following the traditional "Hello World"
## "echo" is a function that will print in a command-shell -terminal- what you put after the "echo" word:

echo "Hello World"

## End


Now, lets use some LINUX/UNIX terminal commands into a script. Here you can start to see what I mean with: Automation. We are "replacing" your fingers (the human) for a scripts (a robot) that can do exactly the same:

    1#!/bin/bash
2
4pwd
5
6## Listing the files in the current directory
7
8ls
9
10## Listing the files in your home directory
11
12ls $HOME 13 14## Listing the files in the temporary directory 15 16ls /tmp/ 17 18## End 19 Where as you can see, the lines 4,8,12,16 are the only lines that perform an action in this script. ## Linux (UNIX) command refreshing review LINUX has an almost infinite number of commands and shell apps that can do virtually any contemporary operation in a computer or network of computers. some of them are: cat > cat ## Concatenate and print (display) the content of files  cd > cd ## Change Directory  chmod > chmod ## Change access permissions  clear > clear ## Clear terminal screen  comm > comm ## Compare two sorted files line by line  cp > cp ## Copy one or more files to another location  cut > cut ## Divide a file into several parts  date > date ## Display or change the date & time  diff > diff ## Display the differences between two files  du > du ## Estimate file space usage  echo > echo ## Display message on screen •  egrep > egrep ## Search file(s) for lines that match an extended expression  exit > exit ## Exit the shell  export > export ## Set an environment variable  find > find ## Search for files that meet a desired criteria  grep > grep ## Search file(s) for lines that match a given pattern  gzip > gzip ## Compress or decompress named file(s)  history > history ## Command History  kill > kill ## Kill a process by specifying its PID  make > make ## Recompile a group of programs  mkdir > mkdir ## Create new folder(s)  more > more ## Display output one screen at a time  mv > mv ## Move or rename files or directories  nohup > nohup ## Run a command immune to hangups  passwd > passwd ## Modify a user password  paste > paste ## Merge lines of files  ping > ping ## Test a network connection  ps > ps ## Process status  pwd > pwd ## Print Working Directory  rcp > rcp ## Copy files between two machines  rm > rm ## Remove files  scp > scp ## Secure copy (remote file copy)  sed > sed ## Stream Editor  sleep > sleep ## Delay for a specified time  sort > sort ## Sort text files  source > source ## Run commands from a file '.'  su > su ## Substitute user identity  tail > tail ## Output the last part of file  time > time ## Measure Program running time  uniq > uniq ## Uniquify files  vi > vi ## Text Editor > vim ## Text Editor  wait > wait ## Wait for a process to complete •  wc > wc ## Print byte, word, and line counts  wget > wget ## Retrieve web pages or files via HTTP, HTTPS or FTP  Another interesting use is the inclusion of special characters as is the case of a blank space: ## to include a black space into a command line, use an "\" and add the needed space, for example: > echo Ciao Mondo Ciao Mondo > echo Ciao\ Mundo Ciao Mondo > echo Ciao Mondo Ciao Mondo > echo Ciao\ \ \ \ \ Mondo Ciao Mondo  ## Some scripts in action ### Generic automatic Email for Account "management" An example of the creation of a generic email template for users into an (computer) User Interface:  1#!/bin/bash 2################################################################################################### 3## Author: Arturo Sanchez (arturos@cern.ch) ## 4## Created: 07th May 2014 ## 5## Updated: 28th October 2015 ## 6## ## 7## How to use this script: ## 8## > ./new_user.sh Name\ LastName nickname 12345 > email_new_users/email_nickname.txt ## 9## Example: ## 10## > ./new_user.sh <Name\ LastName> <username> <password> > email_new_users/email_<username>.txt ## 11################################################################################################### 12 13 14echo "Bongiorno," 15echo "" 16echo "The account for the user:$1 into the ATLAS Napoli User Interface have been created."
17echo ""
18echo ""
19echo "Your username is:  $2" 20echo "Your password is:$3"
21echo ""
22echo ""
23echo "You can access using the ssh connection from a terminal:"
24echo ""
25echo "> ssh -YX -p5022 $2@atlasui.na.infn.it" 26echo "" 27echo "" 28echo "Or for a specific UI use the number 01, 02 or 03 as below:" 29echo "" 30echo "> ssh -YX -p5022$2@atlasui03.na.infn.it"
31echo ""
32echo ""
33echo ""
34echo "As soon you enter the password, you will get something as below:"
35echo ""
36echo "[$2@atlasui03 ~]$"
37echo ""
38echo ""
39echo "Where you can check where you are:"
40echo ""
41echo "[$2@atlasui03 ~]$ pwd"
42echo "/home/$2" 43echo "" 44echo "" 45echo "Please, change your password as soon as possible, using the command:" 46echo "> yppasswd" 47echo "" 48## echo "> passwd" 49## echo "IMPORTANT! (Please, use the atlasui03 to do this procedure)" 50## echo "" 51echo "You will get something like (to introduce your current pass and the new one):" 52echo "" 53## echo "[$2@atlasui03 ~]$passwd" 54echo "[$2@atlasui03 ~]$yppasswd" 55echo "Changing password for user$2."
56echo "Changing password for $2." 57echo "(current) UNIX password:" 58echo "New password: " 59echo "" 60echo "" 61echo "" 62echo "The UI has space dedicates to big files that the users could need to store there." 63echo "The path of such a space is: " 64echo "" 65echo "/data/<username>" 66echo "" 67echo "So, in your case is:" 68echo "" 69echo "/data/$2"
70echo ""
71echo ""
72echo "Your /home/$2 folder has a quota of 10GB." 73echo "" 74echo "If you have an estimation of the possible space that you will need in /data/ and/or /home/, we will appreciate that information in order to setup your quota according to your work necessities." 75echo "" 76echo "In case of any question/suggestion please let me know," 77echo "" 78echo "Cheers," 79echo "Account Managers" 80echo "------" 81echo "" Where if you save this script in a file called: new_user.sh and execute the following line: > source new_user.sh Jon\ Doe jondoe 12345 ...it will give you as a result the printing below ( printing that you can redirect into a text file for bookkeeping and posterior use, like to really send the email ) Bongiorno, The account for the user: Jon Doe into the ATLAS Napoli User Interface have been created. Your username is: jondoe Your password is: 12345 You can access using the ssh connection from a terminal: > ssh -YX -p5022 jondoe@atlasui.na.infn.it Or for a specific UI use the number 01, 02 or 03 as below: > ssh -YX -p5022 jondoe@atlasui03.na.infn.it As soon you enter the password, you will get something as below: [jondoe@atlasui03 ~]$

Where you can check where you are:

[jondoe@atlasui03 ~]$pwd /home/jondoe Please, change your password as soon as possible, using the command: > yppasswd You will get something like (to introduce your current pass and the new one): [jondoe@atlasui03 ~]$ yppasswd

The UI has space dedicates to big files that the users could need to store there.
The path of such a space is:

/data/jondoe

Your /home/jondoe folder has a quota of 10GB.

If you have an estimation of the possible space that you will need in /data/ and/or /home/, we will appreciate that information in order to setup your quota according to your work necessities.

In case of any question/suggestion please let me know,

Cheers,
Account Managers
------

Where the Name, Last Name, nickname and password of the new user were written in the correct lines and all the email now has a useful meaning.
This scripts use two fundamental concepts for scripts that we use in a day-by-day basis: inputs and variables. Those and other concepts are review later in this tutorial but we can already review them here for completeness:
#!/bin/bash

## This is a Variable that is filled with the input #1
firstWord=$1 ## Other variable defined inside the script secondWord="World" echo "Hello$1 $secondWord" ## End  ## Q&A Do you have any question so far? For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL ( please remove SPAMNOT from the email address ) # Exercises Part 1 # Part 2 # Review of concepts with examples: ## Script concepts using bash Remember, scripts are used to put to work the computers for you. They are a set of instructions that allows to perform repetitive operations in an automatic way. So, let's continue the tutorial creating some useful bash scripts that can be used for repetitive activities relative to your analysis chain: ### “Hello” World #!/bin/bash ## This is a comment: we will print Hello World again :) echo "Hello World" ### Variables Lets use some variables to get a feeling of the different use. As in previous sections, they are some of the many ways to use variables but we consider the next examples a representative sample for the majority of the scripts that you may will developed: • Define variables coming from the user input: • Strings #!/bin/bash ## Variables coming from "outside", e.g. included as parameters in the script line execution are taken by order using hte character "$"

first_input=$1 second_input=$2

## etc...

echo "I am the first input \" $first_input \" and the second input \" second_input\" " • Integers and floats Variables different the strings are very useful for operations inside a script: counters, maths, iterators, etc... #!/bin/bash ## this is a variable taken from outside energy_in_tev=$1

## but we always work in MeV of GeV, so a transformation is needed:

energy_gev=$(($energy_in_tev * 1000))

energy_in_mev=$[energy_ingev * 1000] echo ''The energy for the humans is$energy


• Define variables inside the scripts

As in programming languages like C++, we can have global and local variables in our scripts:

• Global variables
#!/bin/bash

##


• Local variables
#!/bin/bash

HELLO=Hello
function hello {
local HELLO=World
echo $HELLO } echo$HELLO
hello
echo $HELLO  • Using variables in operations: printing, assigning, concatenating, arithmetic operations: #!/bin/bash ##  ### Loops The for loop is a little bit different from other programming languages. Basically, it let's you iterate over a series of 'words' within a string. The while executes a piece of code if the control expression is true, and only stops when it is false (or a explicit break is found within the executed code. The until loop is almost equal to the while loop, except that the code is executed while the control expression evaluates to false. For sample #!/bin/bash for i in$( ls ); do
echo item: $i done  On the second line, we declare i to be the variable that will take the different values contained in$( ls ).
The third line could be longer if needed, or there could be more lines before the done (4).
'done' (4) indicates that the code that used the value of $i has finished and$i can take a new value.

• C-like for

It's a for loop more similar to C/perl... for. (perl?)

#!/bin/bash

for i in seq 1 10;
do
echo $i done  • While sample #!/bin/bash COUNTER=0 while [$COUNTER -lt 10 ]; do
echo The counter is $COUNTER let COUNTER=COUNTER+1 done This script 'emulates' the well known (C, Pascal, perl, etc) 'for' structure • Until sample #!/bin/bash COUNTER=20 until [$COUNTER -lt 10 ]; do
echo COUNTER $COUNTER let COUNTER-=1 done  • Note: Loop in python We will talk about this later as well...  1#!/bin/sh 2languages=enscript --help-highlight | grep 'Name:' | cut -d ' ' -f 2 3for l in$languages; do
4    cat << EOF
c1->SaveAs("$1.eps") EOF # The ATLAS Ecosystem and Tools Part I # The User Interfaces (lxplus and atlasui) # The GRID ## What is a Job? ## What I need? ### Code ### Samples There are several ways to look for samples into the ATLAS Tools, here below we start with a very simple views of the official samples: You can browse samples with more details below: ### PROXY ## Q&A Do you have any question so far? For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL ( please remove SPAMNOT from the email address ) # The ATLAS Ecosystem and Tools Part II ## PRUN The ATLAS official Documentation can be found here: ATLAS Tutorial Script of the examples jobs: #!/bin/bash ## Example 1 prun --exec " echo %IN> input.txt; source Sroot.sh input.txt mini lep_pt lep_n\ \>\ 1" --outDS user.arturos.test.tutorial.$name.job01.V$version --inDS user.arturos.MC.outreach.v2 --nFilesPerJob 1 --outputs output.root --rootVer=6.02/12 --cmtConfig=x86_64-slc6-gcc48-opt ## Example 2 prun --exec " echo %IN> input.txt; source ScriptMultipleDraw.sh input.txt mini lep_pt lep_n\ \>\ 0" --outDS user.arturos.test.tutorial.$name.job02.V$version --inDS user.arturos.MC.outreach.v2 --nFilesPerJob 1 --outputs output.root --rootVer=6.02/12 --cmtConfig=x86_64-slc6-gcc48-opt echo "Done!" ## End  ## BigPanda The ATLAS official Documentation can be found here: PanDA You can browse official, user tasks using the new BigPanda portal This is an example of an user account ## RUCIO The ATLAS official Documentation can be found here: RUCIO client As well, more documentation in http://rucio.cern.ch/ ## Example 1 rucio list-dids user.arturos.*.outreach* ## Example 2 rucio list-files user.arturos.MC.outreach.v2  ## AMI The ATLAS official Documentation can be found here: AMI database As well, more documentation in the AMI portal and its CERN mirror ## PBOOK The ATLAS official Documentation can be found here: Panda Book ## CERNBOX The ATLAS official Documentation can be found here: http://cernbox.web.cern.ch/ or this dedicate page at CERN as well. ## JIRA The official Documentation can be found here and the Official CERN Jira website is https://its.cern.ch Note: Not commertial of publicitary intention, but the video is very ilustrative of the potencialities of the use of JIRA, as is used in ATLAS collaboration: ## SVN (Vs) GitLab ## Latest ATLAS “Style” guidelines ## Q&A Do you have any question so far? For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL ( please remove SPAMNOT from the email address ) # Second set of examples in Python Vs BASH (Examples of the use in scripts for analysers are included in each point) ## One to One equivalent examples for generic use ## Q&A Do you have any question so far? For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL ( please remove SPAMNOT from the email address ) # Local developed Tools ## Existing scripts ## Websites and Online ## Tier2 # Practical Exercises with each the mentioned Tools (BASH and Python) ## Initial Examples (review concepts) ## HTML fast view Old HTML Scripts ## LaTeX Text.... \documentclass[a4paper]{article} \usepackage[english]{babel} \usepackage[utf8x]{inputenc} \usepackage{amsmath} \usepackage{graphicx} \usepackage[colorinlistoftodos]{todonotes} \title{Your Paper} \author{You} \begin{document} \maketitle \begin{abstract} Your abstract. \end{abstract} \section{Introduction} Your introduction goes here! Some examples of commonly used commands and features are listed below, to help you get started. If you have a question, please use the help menu (?'') on the top bar to search for help or ask us a question. \section{Some examples to get started} \subsection{How to add Comments} Comments can be added to your project by clicking on the comment icon in the toolbar above. % * <john.hammersley@gmail.com> 2014-09-03T09:54:16.211Z: % % Here's an example comment! % To reply to a comment, simply click the reply button in the lower right corner of the comment, and you can close them when you're done. \subsection{How to include Figures} First you have to upload the image file from your computer using the upload link the project menu. Then use the includegraphics command to include it in your document. Use the figure environment and the caption command to add a number and a caption to your figure. See the code for Figure \ref{fig:frog} in this section for an example. \begin{figure} \centering \includegraphics[width=0.3\textwidth]{frog.jpg} \caption{\label{fig:frog}This frog was uploaded via the project menu.} \end{figure} \subsection{How to add Tables} Use the table and tabular commands for basic tables --- see Table~\ref{tab:widgets}, for example. \begin{table} \centering \begin{tabular}{l|r} Item & Quantity \\\hline Widgets & 42 \\ Gadgets & 13 \end{tabular} \caption{\label{tab:widgets}An example table.} \end{table} \subsection{How to write Mathematics} \LaTeX{} is great at typesetting mathematics. Let$X_1, X_2, \ldots, X_n$be a sequence of independent and identically distributed random variables with$\text{E}[X_i] = \mu$and$\text{Var}[X_i] = \sigma^2 < \infty$, and let $$S_n = \frac{X_1 + X_2 + \cdots + X_n}{n} = \frac{1}{n}\sum_{i}^{n} X_i$$ denote their mean. Then as$n$approaches infinity, the random variables$\sqrt{n}(S_n - \mu)$converge in distribution to a normal$\mathcal{N}(0, \sigma^2)\$.

\subsection{How to create Sections and Subsections}

Use section and subsections to organize your document. Simply use the section and subsection buttons in the toolbar to create them, and we'll handle all the formatting and numbering automatically.

You can make lists with automatic numbering \dots

\begin{enumerate}
\item Like this,
\item and like this.
\end{enumerate}
\dots or bullet points \dots
\begin{itemize}
\item Like this,
\item and like this.
\end{itemize}

We hope you find Overleaf useful, and please let us know if you have any feedback using the help menu above.

\end{document}

That result in a document like this below:

## Intermediate Examples

(combinations of two or more tools using pipes)

## Advance Scenarios and Scripts in ATLAS analysis

(review and test of more complex scripts for automation of tasks)

## Q&A

Do you have any question so far?