ScriptingTutorial

The objective of this page is to deliver a tutorial in scripting with focus in analysis of LHC data into the ATLAS Collaboration.

ATLAS computing/analysis Book!!

ATLAS_Script_Setup






Table of Content











Day 1





test_automation-1024x837.jpg

Introduction: Automation


Picking into the large amount of different concepts with can find, an useful definition can be: The use of tools, techniques and ingenuous in other to create algorithms that allow you to request to the computer to perform a series of complex tasks, in a sequential mode, for an single execution or with a deterministic frequency, in an automatic way with a single execution o command line.


What is a script?


It's the translation to a particular language of the set of instructions resulted of the automation algorithm's development for what you need and/or want this automatic execution(s),

## Hi!, this is a set of commands into a script:  

setup some variables

## This is a comment

source a set of other scripts

## This is another comment

set your input and output files and locations

## Finally

execute your main program of set of applications

## Done


Why to use scripts?


Scripts are used to put to work the computers for you.
They are a set of instructions that allows to perform repetitive operations in an automatic way. So, just your imagination can define the limits of a script. Starting for simple checks into your current working area, passing for the setup of simple of complex environments, or for backup creation to interactions with web-based (internet) resources, including the integration of databases, online monitoring of resources to... malware... the reason to have scripts is basically to get something done as far and human-independent as possible. And as we will see, we can wrap several different applications and tools into a single set of instructions, so, it is like to mix many different software into a single box:

An pictorial view of scripts in an analysis


Grphical_view_scripts.png

Scripting Languages and the OS as environment

Scripts are not restricted to a single computer language. In fact we will see that we can write scripts in several languages (and for several OS) as soon as we can reach the tools we need.
At this point, we will focus the attention in bash and python scripts since their use in Linux OS is very well known and in particular for HEP-ATLAS analysis, they are supported by a large community. The tools that we use in ATLAS for management of data samples, coding, sharing and final publications production pass through C++ and other several open source applications that have command lines interface support.


A list of useful (but far to be completed) of kernels for scripts:

Language/Kernel Covered here? Use in HEP?
bash yes yes
Python yes yes
JavaScript not yes
C++ a bit yes
PowerShell not not?
Google Script not not?
... ... ...


Q&A


smile ?? Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )



Anatomy of a first script (bash)

#!/bin/bash          

## The line above helps to let the system know that you are asking for the bash shell

## As you can see, using #'s allows you to introduce comments.
## Comments are EXTREMELY useful for self documentation of the code!
## Even if you never share the script (or you code in general) you will thank yourself after few weeks/months and you come back to check what you actually did :)
## And if you share your code, is a nice way to let your user or college know that you follow one of the Good Practices in Programming and Computer Science in general

## A first sugestion is to add to every script you plan to keep for more that few hours:
## - Date of creation: XX/YY/ZZ
## - Date of last update: xx/yy/zz
## - Contact email
 
## Saying all that  (excuse me) lets create a first script following the traditional "Hello World"
## "echo" is a function that will print in a command-shell -terminal- what you put after the "echo" word:

echo "Hello World"

## End


Now, lets use some LINUX/UNIX terminal commands into a script. Here you can start to see what I mean with: Automation. We are "replacing" your fingers (the human) for a scripts (a robot) that can do exactly the same:

    1#!/bin/bash          
    2
    3## Asking where you are:
    4pwd
    5
    6## Listing the files in the current directory
    7
    8ls
    9
   10## Listing the files in your home directory
   11
   12ls $HOME
   13
   14## Listing the files in the temporary directory
   15
   16ls /tmp/
   17
   18## End
   19


Where as you can see, the lines 4,8,12,16 are the only lines that perform an action in this script.

Linux (UNIX) command refreshing review


lego.jpg

LINUX has an almost infinite number of commands and shell apps that can do virtually any contemporary operation in a computer or network of computers. some of them are:

cat

> cat      ## Concatenate and print (display) the content of files

cd

> cd       ## Change Directory

chmod

> chmod    ## Change access permissions

clear

> clear    ## Clear terminal screen

comm

> comm     ## Compare two sorted files line by line

cp

> cp       ## Copy one or more files to another location

cut

> cut      ## Divide a file into several parts

date

> date     ## Display or change the date & time

diff

> diff     ## Display the differences between two files

du

> du       ## Estimate file space usage

echo

> echo     ## Display message on screen •

egrep

> egrep    ## Search file(s) for lines that match an extended expression

exit

> exit     ## Exit the shell

export

> export   ## Set an environment variable

find

> find     ## Search for files that meet a desired criteria

grep

> grep     ## Search file(s) for lines that match a given pattern

gzip

> gzip     ## Compress or decompress named file(s)

history

> history  ## Command History

kill

> kill     ## Kill a process by specifying its PID

make

> make     ## Recompile a group of programs

mkdir

> mkdir    ## Create new folder(s)

more

> more     ## Display output one screen at a time

mv

> mv       ## Move or rename files or directories

nohup

> nohup    ## Run a command immune to hangups

passwd

> passwd   ## Modify a user password

paste

> paste    ## Merge lines of files

ping

> ping     ## Test a network connection

ps

> ps       ## Process status

pwd

> pwd      ## Print Working Directory

rcp

> rcp      ## Copy files between two machines

rm

> rm       ## Remove files

scp

> scp      ## Secure copy (remote file copy)

sed

> sed      ## Stream Editor

sleep

> sleep    ## Delay for a specified time

sort

> sort     ## Sort text files

source

> source   ## Run commands from a file '.'

su

> su       ## Substitute user identity

tail

> tail     ## Output the last part of file

time

> time     ## Measure Program running time

uniq

> uniq     ## Uniquify files

vi

> vi       ## Text Editor
> vim       ## Text Editor

wait

> wait     ## Wait for a process to complete •

wc

> wc       ## Print byte, word, and line counts

wget

> wget     ## Retrieve web pages or files via HTTP, HTTPS or FTP

Another interesting use is the inclusion of special characters as is the case of a blank space:

## to include a black space into a command line, use an "\" and add the needed space, for example: 
>  echo Ciao Mondo
Ciao Mondo

> echo Ciao\ Mundo
Ciao Mondo

> echo Ciao     Mondo
Ciao Mondo

> echo Ciao\ \ \ \ \  Mondo
Ciao     Mondo

Some scripts in action

puzzle-3d-1.jpg

Generic automatic Email for Account "management"


An example of the creation of a generic email template for users into an (computer) User Interface:
    1#!/bin/bash          
    2###################################################################################################
    3## Author: Arturo Sanchez (arturos@cern.ch)                                                      ##
    4## Created: 07th May 2014                                                                        ##
    5## Updated: 28th October 2015                                                                    ##
    6##                                                                                               ##
    7## How to use this script:                                                                       ##
    8## > ./new_user.sh Name\ LastName nickname 12345 > email_new_users/email_nickname.txt            ##
    9## Example:                                                                                      ##
   10## > ./new_user.sh <Name\ LastName> <username> <password> > email_new_users/email_<username>.txt ##
   11###################################################################################################
   12
   13
   14echo "Bongiorno,"
   15echo ""
   16echo "The account for the user: $1 into the ATLAS Napoli User Interface have been created."
   17echo ""
   18echo ""
   19echo "Your username is:  $2"
   20echo "Your password is:  $3"
   21echo ""
   22echo ""
   23echo "You can access using the ssh connection from a terminal:"
   24echo ""
   25echo "> ssh -YX -p5022 $2@atlasui.na.infn.it"
   26echo ""
   27echo ""
   28echo "Or for a specific UI use the number 01, 02 or 03 as below:" 
   29echo ""
   30echo "> ssh -YX -p5022 $2@atlasui03.na.infn.it"
   31echo ""
   32echo ""
   33echo ""
   34echo "As soon you enter the password, you will get something as below:"
   35echo ""
   36echo "[$2@atlasui03 ~]$"
   37echo ""
   38echo ""
   39echo "Where you can check where you are:"
   40echo ""
   41echo "[$2@atlasui03 ~]$ pwd"
   42echo "/home/$2"
   43echo ""
   44echo ""
   45echo "Please, change your password as soon as possible, using the command:"
   46echo "> yppasswd"
   47echo ""
   48## echo "> passwd"
   49## echo "IMPORTANT! (Please, use the atlasui03 to do this procedure)"
   50## echo ""
   51echo "You will get something like (to introduce your current pass and the new one):"
   52echo ""
   53## echo "[$2@atlasui03 ~]$ passwd"
   54echo "[$2@atlasui03 ~]$ yppasswd"
   55echo "Changing password for user $2."
   56echo "Changing password for $2."
   57echo "(current) UNIX password:"
   58echo "New password: "
   59echo ""
   60echo ""
   61echo ""
   62echo "The UI has space dedicates to big files that the users could need to store there."
   63echo "The path of such a space is: "
   64echo ""
   65echo "/data/<username>"
   66echo ""
   67echo "So, in your case is:"
   68echo ""
   69echo "/data/$2"
   70echo ""
   71echo ""
   72echo "Your /home/$2 folder has a quota of 10GB."
   73echo ""
   74echo "If you have an estimation of the possible space that you will need in /data/ and/or /home/, we will appreciate that information in order to setup your quota according to your work necessities."
   75echo ""
   76echo "In case of any question/suggestion please let me know,"
   77echo ""
   78echo "Cheers,"
   79echo "Account Managers"
   80echo "------"
   81echo ""

Where if you save this script in a file called: new_user.sh and execute the following line:

>  source new_user.sh Jon\ Doe jondoe 12345

...it will give you as a result the printing below ( printing that you can redirect into a text file for bookkeeping and posterior use, like to really send the email smile )

Bongiorno,

The account for the user: Jon Doe into the ATLAS Napoli User Interface have been created.


Your username is:  jondoe
Your password is:  12345


You can access using the ssh connection from a terminal:

> ssh -YX -p5022 jondoe@atlasui.na.infn.it


Or for a specific UI use the number 01, 02 or 03 as below:

> ssh -YX -p5022 jondoe@atlasui03.na.infn.it



As soon you enter the password, you will get something as below:

[jondoe@atlasui03 ~]$


Where you can check where you are:

[jondoe@atlasui03 ~]$ pwd
/home/jondoe


Please, change your password as soon as possible, using the command:
> yppasswd

You will get something like (to introduce your current pass and the new one):

[jondoe@atlasui03 ~]$ yppasswd
Changing password for user jondoe.
Changing password for jondoe.
(current) UNIX password:
New password: 



The UI has space dedicates to big files that the users could need to store there.
The path of such a space is: 

/data/<username>

So, in your case is:

/data/jondoe


Your /home/jondoe folder has a quota of 10GB.

If you have an estimation of the possible space that you will need in /data/ and/or /home/, we will appreciate that information in order to setup your quota according to your work necessities.

In case of any question/suggestion please let me know,

Cheers,
Account Managers
------

Where the Name, Last Name, nickname and password of the new user were written in the correct lines and all the email now has a useful meaning.
This scripts use two fundamental concepts for scripts that we use in a day-by-day basis: inputs and variables. Those and other concepts are review later in this tutorial but we can already review them here for completeness:
#!/bin/bash          

## This is a Variable that is filled with the input #1
firstWord=$1

## Other variable defined inside the script
secondWord="World"

echo "Hello $1 $secondWord"

## End

Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )



Exercises Day 1









Day 2





Review of concepts with examples:

marketing-tools-for-automation.jpg

Script concepts using bash


Remember, scripts are used to put to work the computers for you.
They are a set of instructions that allows to perform repetitive operations in an automatic way. So, let's continue the tutorial creating some useful bash scripts that can be used for repetitive activities relative to your analysis chain:

“Hello” World

#!/bin/bash          

## This is a comment: we will print Hello World again :)

echo "Hello World"

Variables

Lets use some variables to get a feeling of the different use. As in previous sections, they are some of the many ways to use variables but we consider the next examples a representative sample for the majority of the scripts that you may will developed:

  • Define variables coming from the user input:
    • Strings
#!/bin/bash          

## Variables coming from "outside", e.g. included as parameters in the script line execution are taken by order using hte character "$"

first_input=$1 
second_input=$2

## etc...

echo "I am the first input \" $first_input \" and the second input \" second_input\" "

    • Integers and floats
Variables different the strings are very useful for operations inside a script: counters, maths, iterators, etc...

#!/bin/bash          

## this is a variable taken from outside
energy_in_tev=$1

## but we always work in MeV of GeV, so a transformation is needed:

energy_gev=$(($energy_in_tev * 1000))

energy_in_mev=$[energy_ingev * 1000]

echo ''The energy for the humans is $energy

  • Define variables inside the scripts

As in programming languages like C++, we can have global and local variables in our scripts:

    • Global variables
#!/bin/bash          

## 

    • Local variables
#!/bin/bash          

                    HELLO=Hello 
                    function hello {
                            local HELLO=World
                            echo $HELLO
                    }
                    echo $HELLO
                    hello
                    echo $HELLO

    • Using variables in operations: printing, assigning, concatenating, arithmetic operations:

#!/bin/bash          

## 

Loops

The for loop is a little bit different from other programming languages. Basically, it let's you iterate over a series of 'words' within a string.
The while executes a piece of code if the control expression is true, and only stops when it is false (or a explicit break is found within the executed code.
The until loop is almost equal to the while loop, except that the code is executed while the control expression evaluates to false.

For sample

#!/bin/bash          

            for i in $( ls ); do
                echo item: $i
            done
     

On the second line, we declare i to be the variable that will take the different values contained in $( ls ).
The third line could be longer if needed, or there could be more lines before the done (4).
'done' (4) indicates that the code that used the value of $i has finished and $i can take a new value.

    • C-like for

It's a for loop more similar to C/perl... for. (perl?)

#!/bin/bash          

            for i in `seq 1 10`;
            do
                    echo $i
            done    
 

    • While sample

#!/bin/bash  

             COUNTER=0
             while [  $COUNTER -lt 10 ]; do
                 echo The counter is $COUNTER
                 let COUNTER=COUNTER+1 
             done

This script 'emulates' the well known (C, Pascal, perl, etc) 'for' structure

    • Until sample

#!/bin/bash  

             COUNTER=20
             until [  $COUNTER -lt 10 ]; do
                 echo COUNTER $COUNTER
                 let COUNTER-=1
             done
 

  • Note: Loop in python smile We will talk about this later as well...
    1#!/bin/sh
    2languages=`enscript --help-highlight | grep 'Name:' | cut -d ' ' -f 2`
    3for l in $languages; do
    4    cat << EOF
    5   * $l
    6EOF
    7done

Conditionals

Conditionals have many forms. The most basic form is: if expression then statement where 'statement' is only executed if 'expression' evaluates to true. '2<1' is an expresion that evaluates to false, while '2>1' evaluates to true. Conditionals have other forms such as: if expression then statement1 else statement2. Here 'statement1' is executed if 'expression' is true,otherwise 'statement2' is executed.

- using numerical values
- using strings values
- conditions for break, as before in loops
- conditions in “if”
- if, else if, else

A word about syntax:
The base for the 'if' constructions in bash is this:

if [expression];

then

code if 'expression' is true.

fi

Basic conditional example if .. then

#!/bin/bash     
                if [ "foo" = "foo" ]; then
                   echo expression evaluated as true
                fi
  

The code to be executed if the expression within braces is true can be found after the 'then' word and before 'fi' which indicates the end of the conditionally executed code.

  • Basic conditional example if .. then ... else

#!/bin/bash     
                if [ "foo" = "foo" ]; then
                   echo expression evaluated as true
                else
                   echo expression evaluated as false
                fi

  • Conditionals with variables

#!/bin/bash          
                T1="foo"
                T2="bar"
                if [ "$T1" = "$T2" ]; then
                    echo expression evaluated as true
                else
                    echo expression evaluated as false
                fi
                

Pipes


- echo, grep (i, v, c), cat, comm, tail -n X, head -n Y, wc (l,…) as inputs for redirection to files, sed, wc, grep, sort, uniq,...

#!/bin/bash          

## This is a comment

echo "Hello World"

Inputs and Outputs

#!/bin/bash          

## This is a comment

echo "Hello World"

Functions (Moved to Day 3)

- sleep and wait
- define a function
- call functions
- remember of local variables
- use of the .bash_profile and .bash_login file for define functions and scripts in user system/login

#!/bin/bash          

## This is a comment

Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )



Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )




Exercises Day 2









Day 3





ROOT Reminder

Review of concepts


Old tutorial MakeClass and MakeSelector !!

Example macros for script proposes

Key commands and executions of ROOT into bash and python scripts (How to call and interact with root and root files)

#!/bin/bash

root -l file.root << EOF
$1->Draw()
c1->SaveAs("$1.eps")
EOF

The ATLAS Ecosystem and Tools Part I

The User Interfaces (lxplus and atlasui)

The GRID

What is a Job?

What I need?

Code

Samples

PROXY

Test of the Tutorial ATLASUI account! (INTERNAL)

Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )



The ATLAS Ecosystem and Tools Part II

PRUN

(Concept and use) ATLAS Tutorial

Script of the examples jobs:

#!/bin/bash          

## Example 1
prun --exec " echo %IN> input.txt; source Sroot.sh input.txt mini lep_pt lep_n\ \>\ 1" --outDS user.arturos.test.tutorial.$name.job01.V$version --inDS user.arturos.MC.outreach.v2 --nFilesPerJob 1 --outputs output.root --rootVer=6.02/12 --cmtConfig=x86_64-slc6-gcc48-opt

## Example 2
prun --exec " echo %IN> input.txt; source ScriptMultipleDraw.sh input.txt mini lep_pt lep_n\ \>\ 0" --outDS user.arturos.test.tutorial.$name.job02.V$version --inDS user.arturos.MC.outreach.v2 --nFilesPerJob 1 --outputs output.root --rootVer=6.02/12 --cmtConfig=x86_64-slc6-gcc48-opt

echo "Done!"
## End

BigPanda

(Concept and use)
BigPanda Monitor for our jobs (look for your name wink )

RUCIO ( dq2 ? depreciate but still useful )

(Concept and use)

## Example 1
rucio list-dids user.arturos.*.outreach*


## Example 2
rucio list-files user.arturos.MC.outreach.v2

AMI

(Concept and use)

PBOOK

(Concept and use)

CERNBOX

JIRA

SVN (Vs) GitLab

Latest ATLAS “Style” guidelines

The ATLASUI

What is inside

Properties

Services

Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )



The LXPLUS <---> ATLASUI integration

(Examples and case of use)

Second set of examples in Python Vs BASH

(Examples of the use in scripts for analysers are included in each point)

poseswithbrick.png

One to One equivalent examples for generic use

Online resources and references

Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )



Local developed Tools

Existing scripts

Websites and Online

Tier2

Practical Exercises with each the mentioned Tools

(BASH and Python)

Initial Examples

(review concepts)

HTML fast view

Old HTML

Scripts

LaTeX

Text....


\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[colorinlistoftodos]{todonotes}

\title{Your Paper}
\author{You}

\begin{document}
\maketitle

\begin{abstract}
Your abstract.
\end{abstract}

\section{Introduction}

Your introduction goes here! Some examples of commonly used commands and features are listed below, to help you get started. If you have a question, please use the help menu (``?'') on the top bar to search for help or ask us a question. 

\section{Some examples to get started}

\subsection{How to add Comments}

Comments can be added to your project by clicking on the comment icon in the toolbar above. % * <john.hammersley@gmail.com> 2014-09-03T09:54:16.211Z:
%
% Here's an example comment!
%
To reply to a comment, simply click the reply button in the lower right corner of the comment, and you can close them when you're done.

\subsection{How to include Figures}

First you have to upload the image file from your computer using the upload link the project menu. Then use the includegraphics command to include it in your document. Use the figure environment and the caption command to add a number and a caption to your figure. See the code for Figure \ref{fig:frog} in this section for an example.

\begin{figure}
\centering
\includegraphics[width=0.3\textwidth]{frog.jpg}
\caption{\label{fig:frog}This frog was uploaded via the project menu.}
\end{figure}

\subsection{How to add Tables}

Use the table and tabular commands for basic tables --- see Table~\ref{tab:widgets}, for example. 

\begin{table}
\centering
\begin{tabular}{l|r}
Item & Quantity \\\hline
Widgets & 42 \\
Gadgets & 13
\end{tabular}
\caption{\label{tab:widgets}An example table.}
\end{table}

\subsection{How to write Mathematics}

\LaTeX{} is great at typesetting mathematics. Let $X_1, X_2, \ldots, X_n$ be a sequence of independent and identically distributed random variables with $\text{E}[X_i] = \mu$ and $\text{Var}[X_i] = \sigma^2 < \infty$, and let
$$S_n = \frac{X_1 + X_2 + \cdots + X_n}{n}
      = \frac{1}{n}\sum_{i}^{n} X_i$$
denote their mean. Then as $n$ approaches infinity, the random variables $\sqrt{n}(S_n - \mu)$ converge in distribution to a normal $\mathcal{N}(0, \sigma^2)$.


\subsection{How to create Sections and Subsections}

Use section and subsections to organize your document. Simply use the section and subsection buttons in the toolbar to create them, and we'll handle all the formatting and numbering automatically.

\subsection{How to add Lists}

You can make lists with automatic numbering \dots

\begin{enumerate}
\item Like this,
\item and like this.
\end{enumerate}
\dots or bullet points \dots
\begin{itemize}
\item Like this,
\item and like this.
\end{itemize}

We hope you find Overleaf useful, and please let us know if you have any feedback using the help menu above.

\end{document}

That result in a document like this below:



Intermediate Examples

(combinations of two or more tools using pipes)

Advance Scenarios and Scripts in ATLAS analysis

(review and test of more complex scripts for automation of tasks)

Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )




Exercises Day 3





Major updates:
-- ArturoS - 2015-10-21

Responsible: ArturoS
Subject: public

Edit | Attach | Watch | Print version | History: r22 < r21 < r20 < r19 < r18 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r22 - 2015-11-03 - ArturoS
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Atlas All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback