ScriptingInHEP

Scripting In HEP - LHC


NEW NOTE To get all the functionality of the TWIKI, please remove the http:// protection of your browser! (shown here below for Firefox and Chrome)
Screenshot_2015-05-07_13.48.29.png





Table of Content






Important Note: To use the Table links, please expand the days below before smile


Go to Part 1

Go to Part 2

Go to Part 3

Go to Part 4


The objective of this page is to deliver a tutorial in scripting with focus in analysis of LHC data into the ATLAS Collaboration.

ATLAS computing/analysis Book!!

ATLAS_Script_Setup



Part 1


test_automation-1024x837.jpg

Introduction: Automation


Picking into the large amount of different concepts with can find, an useful definition can be: The use of tools, techniques and ingenuous in other to create algorithms that allow you to request to the computer to perform a series of complex tasks, in a sequential mode, for an single execution or with a deterministic frequency, in an automatic way with a single execution o command line.


What is a script?


It's the translation to a particular language of the set of instructions resulted of the automation algorithm's development for what you need and/or want this automatic execution(s),

## Hi!, this is a set of commands into a script:  

setup some variables

## This is a comment

source a set of other scripts

## This is another comment

set your input and output files and locations

## Finally

execute your main program of set of applications

## Done


Why to use scripts?


Scripts are used to put to work the computers for you.
They are a set of instructions that allows to perform repetitive operations in an automatic way. So, just your imagination can define the limits of a script. Starting for simple checks into your current working area, passing for the setup of simple of complex environments, or for backup creation to interactions with web-based (internet) resources, including the integration of databases, online monitoring of resources to... malware... the reason to have scripts is basically to get something done as far and human-independent as possible. And as we will see, we can wrap several different applications and tools into a single set of instructions, so, it is like to mix many different software into a single box:

An pictorial view of scripts in an analysis


Grphical_view_scripts.png

Scripting Languages and the OS as environment

Scripts are not restricted to a single computer language. In fact we will see that we can write scripts in several languages (and for several OS) as soon as we can reach the tools we need.
At this point, we will focus the attention in bash and python scripts since their use in Linux OS is very well known and in particular for HEP-ATLAS analysis, they are supported by a large community. The tools that we use in ATLAS for management of data samples, coding, sharing and final publications production pass through C++ and other several open source applications that have command lines interface support.


A list of useful (but far to be completed) of kernels for scripts:

Language/Kernel Covered here? Use in HEP?
bash yes yes
Python yes yes
JavaScript not yes
C++ a bit yes
PowerShell not not?
Google Script not not?
... ... ...


Q&A


smile ?? Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )



Anatomy of a first script (bash)

#!/bin/bash          

## The line above helps to let the system know that you are asking for the bash shell

## As you can see, using #'s allows you to introduce comments.
## Comments are EXTREMELY useful for self documentation of the code!
## Even if you never share the script (or you code in general) you will thank yourself after few weeks/months and you come back to check what you actually did :)
## And if you share your code, is a nice way to let your user or college know that you follow one of the Good Practices in Programming and Computer Science in general

## A first sugestion is to add to every script you plan to keep for more that few hours:
## - Date of creation: XX/YY/ZZ
## - Date of last update: xx/yy/zz
## - Contact email
 
## Saying all that  (excuse me) lets create a first script following the traditional "Hello World"
## "echo" is a function that will print in a command-shell -terminal- what you put after the "echo" word:

echo "Hello World"

## End


Now, lets use some LINUX/UNIX terminal commands into a script. Here you can start to see what I mean with: Automation. We are "replacing" your fingers (the human) for a scripts (a robot) that can do exactly the same:

    1#!/bin/bash          
    2
    3## Asking where you are:
    4pwd
    5
    6## Listing the files in the current directory
    7
    8ls
    9
   10## Listing the files in your home directory
   11
   12ls $HOME
   13
   14## Listing the files in the temporary directory
   15
   16ls /tmp/
   17
   18## End
   19


Where as you can see, the lines 4,8,12,16 are the only lines that perform an action in this script.

Linux (UNIX) command refreshing review


lego.jpg

LINUX has an almost infinite number of commands and shell apps that can do virtually any contemporary operation in a computer or network of computers. some of them are:

cat

> cat      ## Concatenate and print (display) the content of files

cd

> cd       ## Change Directory

chmod

> chmod    ## Change access permissions

clear

> clear    ## Clear terminal screen

comm

> comm     ## Compare two sorted files line by line

cp

> cp       ## Copy one or more files to another location

cut

> cut      ## Divide a file into several parts

date

> date     ## Display or change the date & time

diff

> diff     ## Display the differences between two files

du

> du       ## Estimate file space usage

echo

> echo     ## Display message on screen •

egrep

> egrep    ## Search file(s) for lines that match an extended expression

exit

> exit     ## Exit the shell

export

> export   ## Set an environment variable

find

> find     ## Search for files that meet a desired criteria

grep

> grep     ## Search file(s) for lines that match a given pattern

gzip

> gzip     ## Compress or decompress named file(s)

history

> history  ## Command History

kill

> kill     ## Kill a process by specifying its PID

make

> make     ## Recompile a group of programs

mkdir

> mkdir    ## Create new folder(s)

more

> more     ## Display output one screen at a time

mv

> mv       ## Move or rename files or directories

nohup

> nohup    ## Run a command immune to hangups

passwd

> passwd   ## Modify a user password

paste

> paste    ## Merge lines of files

ping

> ping     ## Test a network connection

ps

> ps       ## Process status

pwd

> pwd      ## Print Working Directory

rcp

> rcp      ## Copy files between two machines

rm

> rm       ## Remove files

scp

> scp      ## Secure copy (remote file copy)

sed

> sed      ## Stream Editor

sleep

> sleep    ## Delay for a specified time

sort

> sort     ## Sort text files

source

> source   ## Run commands from a file '.'

su

> su       ## Substitute user identity

tail

> tail     ## Output the last part of file

time

> time     ## Measure Program running time

uniq

> uniq     ## Uniquify files

vi

> vi       ## Text Editor
> vim       ## Text Editor

wait

> wait     ## Wait for a process to complete •

wc

> wc       ## Print byte, word, and line counts

wget

> wget     ## Retrieve web pages or files via HTTP, HTTPS or FTP

Another interesting use is the inclusion of special characters as is the case of a blank space:

## to include a black space into a command line, use an "\" and add the needed space, for example: 
>  echo Ciao Mondo
Ciao Mondo

> echo Ciao\ Mundo
Ciao Mondo

> echo Ciao     Mondo
Ciao Mondo

> echo Ciao\ \ \ \ \  Mondo
Ciao     Mondo

Some scripts in action

puzzle-3d-1.jpg

Generic automatic Email for Account "management"


An example of the creation of a generic email template for users into an (computer) User Interface:
    1#!/bin/bash          
    2###################################################################################################
    3## Author: Arturo Sanchez (arturos@cern.ch)                                                      ##
    4## Created: 07th May 2014                                                                        ##
    5## Updated: 28th October 2015                                                                    ##
    6##                                                                                               ##
    7## How to use this script:                                                                       ##
    8## > ./new_user.sh Name\ LastName nickname 12345 > email_new_users/email_nickname.txt            ##
    9## Example:                                                                                      ##
   10## > ./new_user.sh <Name\ LastName> <username> <password> > email_new_users/email_<username>.txt ##
   11###################################################################################################
   12
   13
   14echo "Bongiorno,"
   15echo ""
   16echo "The account for the user: $1 into the ATLAS Napoli User Interface have been created."
   17echo ""
   18echo ""
   19echo "Your username is:  $2"
   20echo "Your password is:  $3"
   21echo ""
   22echo ""
   23echo "You can access using the ssh connection from a terminal:"
   24echo ""
   25echo "> ssh -YX -p5022 $2@atlasui.na.infn.it"
   26echo ""
   27echo ""
   28echo "Or for a specific UI use the number 01, 02 or 03 as below:" 
   29echo ""
   30echo "> ssh -YX -p5022 $2@atlasui03.na.infn.it"
   31echo ""
   32echo ""
   33echo ""
   34echo "As soon you enter the password, you will get something as below:"
   35echo ""
   36echo "[$2@atlasui03 ~]$"
   37echo ""
   38echo ""
   39echo "Where you can check where you are:"
   40echo ""
   41echo "[$2@atlasui03 ~]$ pwd"
   42echo "/home/$2"
   43echo ""
   44echo ""
   45echo "Please, change your password as soon as possible, using the command:"
   46echo "> yppasswd"
   47echo ""
   48## echo "> passwd"
   49## echo "IMPORTANT! (Please, use the atlasui03 to do this procedure)"
   50## echo ""
   51echo "You will get something like (to introduce your current pass and the new one):"
   52echo ""
   53## echo "[$2@atlasui03 ~]$ passwd"
   54echo "[$2@atlasui03 ~]$ yppasswd"
   55echo "Changing password for user $2."
   56echo "Changing password for $2."
   57echo "(current) UNIX password:"
   58echo "New password: "
   59echo ""
   60echo ""
   61echo ""
   62echo "The UI has space dedicates to big files that the users could need to store there."
   63echo "The path of such a space is: "
   64echo ""
   65echo "/data/<username>"
   66echo ""
   67echo "So, in your case is:"
   68echo ""
   69echo "/data/$2"
   70echo ""
   71echo ""
   72echo "Your /home/$2 folder has a quota of 10GB."
   73echo ""
   74echo "If you have an estimation of the possible space that you will need in /data/ and/or /home/, we will appreciate that information in order to setup your quota according to your work necessities."
   75echo ""
   76echo "In case of any question/suggestion please let me know,"
   77echo ""
   78echo "Cheers,"
   79echo "Account Managers"
   80echo "------"
   81echo ""

Where if you save this script in a file called: new_user.sh and execute the following line:

>  source new_user.sh Jon\ Doe jondoe 12345

...it will give you as a result the printing below ( printing that you can redirect into a text file for bookkeeping and posterior use, like to really send the email smile )

Bongiorno,

The account for the user: Jon Doe into the ATLAS Napoli User Interface have been created.


Your username is:  jondoe
Your password is:  12345


You can access using the ssh connection from a terminal:

> ssh -YX -p5022 jondoe@atlasui.na.infn.it


Or for a specific UI use the number 01, 02 or 03 as below:

> ssh -YX -p5022 jondoe@atlasui03.na.infn.it



As soon you enter the password, you will get something as below:

[jondoe@atlasui03 ~]$


Where you can check where you are:

[jondoe@atlasui03 ~]$ pwd
/home/jondoe


Please, change your password as soon as possible, using the command:
> yppasswd

You will get something like (to introduce your current pass and the new one):

[jondoe@atlasui03 ~]$ yppasswd
Changing password for user jondoe.
Changing password for jondoe.
(current) UNIX password:
New password: 



The UI has space dedicates to big files that the users could need to store there.
The path of such a space is: 

/data/<username>

So, in your case is:

/data/jondoe


Your /home/jondoe folder has a quota of 10GB.

If you have an estimation of the possible space that you will need in /data/ and/or /home/, we will appreciate that information in order to setup your quota according to your work necessities.

In case of any question/suggestion please let me know,

Cheers,
Account Managers
------

Where the Name, Last Name, nickname and password of the new user were written in the correct lines and all the email now has a useful meaning.
This scripts use two fundamental concepts for scripts that we use in a day-by-day basis: inputs and variables. Those and other concepts are review later in this tutorial but we can already review them here for completeness:
#!/bin/bash          

## This is a Variable that is filled with the input #1
firstWord=$1

## Other variable defined inside the script
secondWord="World"

echo "Hello $1 $secondWord"

## End

Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )



Exercises Part 1








Part 2


Review of concepts with examples:

marketing-tools-for-automation.jpg

Script concepts using bash


Remember, scripts are used to put to work the computers for you.
They are a set of instructions that allows to perform repetitive operations in an automatic way. So, let's continue the tutorial creating some useful bash scripts that can be used for repetitive activities relative to your analysis chain:

“Hello” World

#!/bin/bash          

## This is a comment: we will print Hello World again :)

echo "Hello World"

Variables

Lets use some variables to get a feeling of the different use. As in previous sections, they are some of the many ways to use variables but we consider the next examples a representative sample for the majority of the scripts that you may will developed:

  • Define variables coming from the user input:
    • Strings
#!/bin/bash          

## Variables coming from "outside", e.g. included as parameters in the script line execution are taken by order using hte character "$"

first_input=$1 
second_input=$2

## etc...

echo "I am the first input \" $first_input \" and the second input \" second_input\" "

    • Integers and floats
Variables different the strings are very useful for operations inside a script: counters, maths, iterators, etc...

#!/bin/bash          

## this is a variable taken from outside
energy_in_tev=$1

## but we always work in MeV of GeV, so a transformation is needed:

energy_gev=$(($energy_in_tev * 1000))

energy_in_mev=$[energy_ingev * 1000]

echo ''The energy for the humans is $energy

  • Define variables inside the scripts

As in programming languages like C++, we can have global and local variables in our scripts:

    • Global variables
#!/bin/bash          

## 

    • Local variables
#!/bin/bash          

                    HELLO=Hello 
                    function hello {
                            local HELLO=World
                            echo $HELLO
                    }
                    echo $HELLO
                    hello
                    echo $HELLO

    • Using variables in operations: printing, assigning, concatenating, arithmetic operations:

#!/bin/bash          

## 

Loops

The for loop is a little bit different from other programming languages. Basically, it let's you iterate over a series of 'words' within a string.
The while executes a piece of code if the control expression is true, and only stops when it is false (or a explicit break is found within the executed code.
The until loop is almost equal to the while loop, except that the code is executed while the control expression evaluates to false.

For sample

#!/bin/bash          

            for i in $( ls ); do
                echo item: $i
            done
     

On the second line, we declare i to be the variable that will take the different values contained in $( ls ).
The third line could be longer if needed, or there could be more lines before the done (4).
'done' (4) indicates that the code that used the value of $i has finished and $i can take a new value.

    • C-like for

It's a for loop more similar to C/perl... for. (perl?)

#!/bin/bash          

            for i in `seq 1 10`;
            do
                    echo $i
            done    
 

    • While sample

#!/bin/bash  

             COUNTER=0
             while [  $COUNTER -lt 10 ]; do
                 echo The counter is $COUNTER
                 let COUNTER=COUNTER+1 
             done

This script 'emulates' the well known (C, Pascal, perl, etc) 'for' structure

    • Until sample

#!/bin/bash  

             COUNTER=20
             until [  $COUNTER -lt 10 ]; do
                 echo COUNTER $COUNTER
                 let COUNTER-=1
             done
 

  • Note: Loop in python smile We will talk about this later as well...
    1#!/bin/sh
    2languages=`enscript --help-highlight | grep 'Name:' | cut -d ' ' -f 2`
    3for l in $languages; do
    4    cat << EOF
    5   * $l
    6EOF
    7done

Conditionals

Conditionals have many forms. The most basic form is: if expression then statement where 'statement' is only executed if 'expression' evaluates to true. '2<1' is an expresion that evaluates to false, while '2>1' evaluates to true. Conditionals have other forms such as: if expression then statement1 else statement2. Here 'statement1' is executed if 'expression' is true,otherwise 'statement2' is executed.

- using numerical values
- using strings values
- conditions for break, as before in loops
- conditions in “if”
- if, else if, else

A word about syntax:
The base for the 'if' constructions in bash is this:

if [expression];

then

code if 'expression' is true.

fi

Basic conditional example if .. then

#!/bin/bash     
                if [ "foo" = "foo" ]; then
                   echo expression evaluated as true
                fi
  

The code to be executed if the expression within braces is true can be found after the 'then' word and before 'fi' which indicates the end of the conditionally executed code.

  • Basic conditional example if .. then ... else

#!/bin/bash     
                if [ "foo" = "foo" ]; then
                   echo expression evaluated as true
                else
                   echo expression evaluated as false
                fi

  • Conditionals with variables

#!/bin/bash          
                T1="foo"
                T2="bar"
                if [ "$T1" = "$T2" ]; then
                    echo expression evaluated as true
                else
                    echo expression evaluated as false
                fi
                

Pipes


- echo, grep (i, v, c), cat, comm, tail -n X, head -n Y, wc (l,…) as inputs for redirection to files, sed, wc, grep, sort, uniq,...

#!/bin/bash          

## This is a comment

echo "Hello World"

Inputs and Outputs

#!/bin/bash          

## This is a comment

echo "Hello World"

Functions

- sleep and wait
- define a function
- call functions
- remember of local variables
- use of the .bash_profile and .bash_login file for define functions and scripts in user system/login

#!/bin/bash          

## This is a comment

Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )




Exercises Part 2




The objective of this page is to deliver extra help and resources to the *X Workshop ATLAS Cosenza-Italia 2015

ATLAS Detector



Part 3


Introduction to ATLAS + Run II

We will work using the ATLAS Analysis-Computing Run II Model, where the key point is the use of the computer facilities as a services.
Where we get the necessary Software and Tools to perform our Physics Analysis.
But at the same time, in this very early stage of the so-calleGenerid Run II, it will be necessary a lot of development.
This means: to create new codes and frameworks where the constant loop programing-compilation-correction-test will take a lot of the analyzers (e.g. your) time.
For that reason we are presenting here some ideas on how to carry your work with you and be ready to code and tests with or without connection to the usual ATLAS Computing Facilities.

ATLAS_Analysis_Computing_Model_RunI_vs_RunII.png


ROOT Installation and First Macros

Using the most standard ROOT instructions ( available in its website and here below ) we will proceed to show you how to install any version of ROOT in three different Operative Systems (OS):

  1. Scientific Linux 6.6.
  2. MAC OS 10.9 (also useful for 10.10.2).
  3. Ubuntu 14.04.




Installation of ROOT in a Scientific Linux OS distribution (6.6)

Video Tutorial with the explanation ( for the moment in Spanish ) on how to get installed a ROOT version in Scientific Linux 6.6 (SL6).



The next lines refer to the commands explained and used in the video-installation

## Getting ready before to install ROOT: You need to be root (administrator)
     su
##  (Insert the admin password)

## Installing updates and useful packages
     yum update
     yum install yum-priorities
     yum install kernel-devel-2.6.32-431.el6.x86_64
     yum install kernel-devel
     yum install gcc
     yum install git

## Getting the source of ROOT
   git clone http://root.cern.ch/git/root.git
   cd root/
   git checkout -b v5-34-23 v5-34-23
   ./configure --help
   ./configure             ## ( You can include the arch if you want: linuxx8664gcc )    Note: you can include  "--all" too, e.g. "./configure --al"

## Installing the missing pre-requeriments od ROOT
   yum install gcc-c++
   yum install libX11-devel
   yum install libX11-xpm
   yum install libXpm-devel
   yum install libXft-devel
   yum install libXext-devel

## Now the configuration should be OK and is printted that we can do "make"
   ./configure --all
   make
   . bin/thisroot.sh 

## Finally do:
   root

Installation of ROOT in a MAC OS distribution (10.9 but also useful for 10.10.2 )

Video Tutorial with the explanation ( for the moment in Spanish ) on how to get installed a ROOT version in MAC OS 10.9.


Installation of ROOT in a Ubuntu distribution (14.04)

Instructions in order to install ROOT in a bright new version of Ubuntu distribution 14.04. The next lines have been execute in a new VM:
Note: can look pedantic but we can learn a bit more of how to get ROOT for futures occasions.

## - 1) You must install a New version of Ubuntu 14.04 ( _or any 14's stable version_ )

## - 2) Do the Update of the OS using the statard "Update" tool/application, e.g. look for "Automatic/Available Updates" and install them.

## - 3) Open a new *Terminal*

## - 4) Execute the command: ( _it will request the administration password, that one used while the Ubuntu installation..._ )

   sudo apt-get update

## - 5) Execute the command

   sudo apt-get install git

## - 6) Go to the directory where you would like to install *ROOT*

   cd <your_installation_path>

## - 7) Execute the necessary command to obtain *ROOT* ( _in this example, one of the most recent-stable ROOT5 versions_ )

   git clone http://root.cern.ch/git/root.git
   cd root

   git tag -l
   git checkout -b v5-34-24 v5-34-24

## - 8) Let's do the configuration: here the *KEY* is to install *just* what *ROOT* will miss every time we run the configuration command, So:

   ./configure 

## - 9) The first error get solved doing:

   sudo apt-get install dpkg-dev

## - 10) And again, the configuration command

   ./configure 

## - 11) The next error get solved doing:

   sudo apt-get install libx11-dev

## - 12) One more time:

   ./configure 

## - 13) The next error get solved doing:

   sudo apt-get install libxpm-dev

## - 14) One more time:

   ./configure 

## - 15) And the last error get solved doing:

   sudo apt-get install libxft-dev

## - 16) At this point the configuration should finish successfully, showing that you can do " *make* " 
## (  Optional: if just by chance you get another error, you just need to execute ---->   sudo apt-get install libxext-dev     )

## - 17) Execute the command (_ and wait... this can take between 30min to 3 hours..._ ) 

   sudo make

  • You should obtain something like this below

Screenshot_2015-02-01_19.31.07.png

## - 18) After the compilation, souce the code

   source bin/thisroot.sh

## - 19) And run *ROOT*

   root

Screenshot_2015-02-01_19.47.38.png


You can check this video where a dynamic installation is perform too ( in English )





Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )



Generic tutorial(s) of two different macros for ROOT analysis

The Standard and useful ROOT documentation

We can always refer to the ROOT online Tutorial recommendations. Here below it has been included, so, you can directly browse the different examples that they have, take a look!!




Two Simple ( and fully working ) ROOT - macro production

Small zip package prepared to understand the internal structure of the 2 different macros for analysis of ROOT ntuple

This will help to start to work very easy and with two different methods. (take ~27min setup the 2 “analysis examples")

Files can be download from here or directly get the zip file clicking here!

Or use this "window" to take the mentioned file




The next is a Video-Tutorial ( in English ) to create the macros step by step and to use the input files:




NEW Getting the Complete AnalysisBase environment in a local machine using a Virtual Machine (VM)

Getting the Virtualization Software (Hypervisor) VirtualBox

Using the website VirtualBox or simple use the "window" below to browse the website ( and download the software )




Some Steps:




Screenshot_2015-02-02_22.34.19.png

Screenshot_2015-02-02_22.36.23.png

Screenshot_2015-02-02_22.37.05.png



  • Look for the icon into your Applications (folder) and get the main interface of VirtualBox:

Screenshot_2015-02-02_22.38.01.png


  • Now let's create a new VM entity by clicking New on the left corner:

Screenshot_2015-02-02_22.43.56.png


  • For Memory Size I strongly suggest you to use 1024MB of RAM

Screenshot_2015-02-02_22.44.39.png


  • For the Hard Drive creation, take the middle's option: " Create a virtual hard drive now "

Screenshot_2015-02-02_22.44.51.png


  • For Hard drive file type take the default option " VDI "

Screenshot_2015-02-02_22.45.04.png


  • For Storage on physical hard drive select " Dynamicaly allocated " (This is an smart option for developing)

Screenshot_2015-02-02_22.45.13.png


  • For File location and size you will need to select a maximum size fro the virtual HD of at least 26GB

Screenshot_2015-02-02_23.21.44.png


  • After those steps, your VM is created (well, an empty VM)
    Good VM-SL6 Configuration
    Screenshot_2015-02-03_14.42.36.png



Getting the Virtual-Machine-Scientific-Linux-6.6 ( VM-SL6 ) with the ATLAS Software

In order to obtain the VDI file of our VM-SL6, we need to be sure that our host system have space for that, so remember, at least 26GB of space should be reserved for this.
To download the - compressed - file just click here.

You can use the "window" below to get the VM-SL6




  • You can directly select the option of Open the file is you have it:

Screenshot_2015-02-05_02.43.32.png

  • Unzip the VM-SL6

Screenshot_2015-02-03_15.08.26.png Screenshot_2015-02-03_15.28.18.png


  • Adding the VDI file of the VM-SL6 to the VirtualBox machine: You must click Settings on the up-left position of your VirtualBox window (the " Ingranaggio " icon)

Screenshot_2015-02-02_22.46.35.png


  • Now in the new pop-up window, you must go to Storage ( 4th left to the right )

Screenshot_2015-02-02_22.46.50.png


  • You will see on the right the Hard Disk and a small icon of a hard disk where if you click there, the system will give you two options. You have to select: Choose a virtual hard disk file...

Screenshot_2015-02-02_22.48.11.png


  • And browse for the recent downloaded (and uncompressed) VDI file of the VM-SL6 :

Screenshot_2015-02-03_15.31.55.png


  • Now say OK to the Settings window and Start (Avvia ) your VM-SL6 ( you will see something like this below )

Screenshot_2015-02-03_15.33.12.png
Screenshot_2015-02-03_15.34.04.png



IMPORTANT NOTE: To login the VM-SL6 use as username = adminuser and as password = adminuser
(This is your root password if you plan/need to do su and/or sudo ).

  • If you open a Firefox window, you will wet this presentation below, but where the only bookmark is the Tutorial that you are doing now! wink

Screenshot_2015-02-03_15.35.23.png


- Open a new Terminal ( the icon is at the top... ) and do a ls command right there, you will get the into the list: (picture below)

  1. The two scripts to setup the AnalysisBase 2.0.24 and AnalysisBase 2.1.24 :
        AnalysisBase_2.0.24.sh
        AnalysisBase_2.1.24.sh
  2. The ATLAS folder where is the software and xAOD input we will use
  3. Other Standard Linux folders

Screenshot_2015-02-03_15.36.28.png


  • Going into the ATLAS folder you will find the next relevant for us structure:
    ATLAS
      |
      ---- AnalysisBase     ## Our workspace place with the ATLAS Analysis software
            |
            ---- 2.0.24
            |
            ---- 2.1.24
      |
      ---- roots              ## Our two possible ROOT versions, very up-to-date (February 2015) They are use to run 2.0.24 (ROOT5) and 2.1.24 (ROOT6) 
            |
            ---- root-v5-34-24
            |
            ---- root-v6-02-04
      |
      ---- samples          ## The xAOD sample useful for tutorial and development, as you see (picture below) it's a real sample with 3GB size.
            |
            ---- mc14_8TeV.160009.*ggH125_gamma*/
                   |
                    ---- AOD.01512140._000094.pool.root.1
    
    Screenshot_2015-02-05_16.31.54.png


  • You can see the behaivor of ROOT using the two versions (ROOT5 and ROOT6 here below) smile

Screenshot_2015-02-03_15.38.22.png

Screenshot_2015-02-03_15.39.45.png


  • Taking a look into the two AnalysisBase setup scripts mentioned before (using command more ) Yo can see how ROOT is sourced and RootCore (+ boost that is necessary to compile...)

Screenshot_2015-02-03_15.41.40.png


  • An internal view of one of the AnalysisBase releases:

Screenshot_2015-02-03_15.53.00.png


  • At this point you can start to work with the instructions in the Tutorial (copying and pasting instructions...)

Screenshot_2015-02-03_15.56.19.png
Screenshot_2015-02-03_16.17.42.png


  • You can take a look to the xAOD file too as it is mention in the tutorial, but browsing the file locally!

Screenshot_2015-02-03_01.00.00.png


Note: You can install a Cloud client to keep a real time contact with your two machines.

  • Dropbox or Google Drive Client to share files between the VM-SL6 and your host OS:
    Screenshot_2015-02-03_16.32.37.png Screenshot_2015-02-03_16.34.04.png


Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )




Exercises Part 3







Part 4


ROOT Reminder

Example macros for script proposes

Key commands and executions of ROOT into bash and python scripts (How to call and interact with root and root files)

#!/bin/bash

root -l file.root << EOF
$1->Draw()
c1->SaveAs("$1.eps")
EOF

The ATLAS Ecosystem and Tools Part I

The User Interfaces (lxplus and atlasui)

The GRID

What is a Job?

What I need?

Code

Samples


There are several ways to look for samples into the ATLAS Tools, here below we start with a very simple views of the official samples:


You can browse samples with more details below:

PROXY

Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )



The ATLAS Ecosystem and Tools Part II

PRUN


The ATLAS official Documentation can be found here: ATLAS Tutorial

Script of the examples jobs:

#!/bin/bash          

## Example 1
prun --exec " echo %IN> input.txt; source Sroot.sh input.txt mini lep_pt lep_n\ \>\ 1" --outDS user.arturos.test.tutorial.$name.job01.V$version --inDS user.arturos.MC.outreach.v2 --nFilesPerJob 1 --outputs output.root --rootVer=6.02/12 --cmtConfig=x86_64-slc6-gcc48-opt

## Example 2
prun --exec " echo %IN> input.txt; source ScriptMultipleDraw.sh input.txt mini lep_pt lep_n\ \>\ 0" --outDS user.arturos.test.tutorial.$name.job02.V$version --inDS user.arturos.MC.outreach.v2 --nFilesPerJob 1 --outputs output.root --rootVer=6.02/12 --cmtConfig=x86_64-slc6-gcc48-opt

echo "Done!"
## End

BigPanda

The ATLAS official Documentation can be found here: PanDA

You can browse official, user tasks using the new BigPanda portal
This is an example of an user account

RUCIO

The ATLAS official Documentation can be found here: RUCIO client
As well, more documentation in http://rucio.cern.ch/

## Example 1
rucio list-dids user.arturos.*.outreach*


## Example 2
rucio list-files user.arturos.MC.outreach.v2

AMI

The ATLAS official Documentation can be found here: AMI database
As well, more documentation in the AMI portal and its CERN mirror

PBOOK


The ATLAS official Documentation can be found here: Panda Book

CERNBOX


The ATLAS official Documentation can be found here: http://cernbox.web.cern.ch/ or this dedicate page at CERN as well.

JIRA


The official Documentation can be found here and the Official CERN Jira website is https://its.cern.ch


Note: Not commertial of publicitary intention, but the video is very ilustrative of the potencialities of the use of JIRA, as is used in ATLAS collaboration:


SVN (Vs) GitLab

Latest ATLAS “Style” guidelines

Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )



Second set of examples in Python Vs BASH

(Examples of the use in scripts for analysers are included in each point)

poseswithbrick.png

One to One equivalent examples for generic use

Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )



Local developed Tools

Existing scripts

Websites and Online

Tier2

Practical Exercises with each the mentioned Tools

(BASH and Python)

Initial Examples

(review concepts)

HTML fast view

Old HTML

Scripts

LaTeX

Text....


\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[colorinlistoftodos]{todonotes}

\title{Your Paper}
\author{You}

\begin{document}
\maketitle

\begin{abstract}
Your abstract.
\end{abstract}

\section{Introduction}

Your introduction goes here! Some examples of commonly used commands and features are listed below, to help you get started. If you have a question, please use the help menu (``?'') on the top bar to search for help or ask us a question. 

\section{Some examples to get started}

\subsection{How to add Comments}

Comments can be added to your project by clicking on the comment icon in the toolbar above. % * <john.hammersley@gmail.com> 2014-09-03T09:54:16.211Z:
%
% Here's an example comment!
%
To reply to a comment, simply click the reply button in the lower right corner of the comment, and you can close them when you're done.

\subsection{How to include Figures}

First you have to upload the image file from your computer using the upload link the project menu. Then use the includegraphics command to include it in your document. Use the figure environment and the caption command to add a number and a caption to your figure. See the code for Figure \ref{fig:frog} in this section for an example.

\begin{figure}
\centering
\includegraphics[width=0.3\textwidth]{frog.jpg}
\caption{\label{fig:frog}This frog was uploaded via the project menu.}
\end{figure}

\subsection{How to add Tables}

Use the table and tabular commands for basic tables --- see Table~\ref{tab:widgets}, for example. 

\begin{table}
\centering
\begin{tabular}{l|r}
Item & Quantity \\\hline
Widgets & 42 \\
Gadgets & 13
\end{tabular}
\caption{\label{tab:widgets}An example table.}
\end{table}

\subsection{How to write Mathematics}

\LaTeX{} is great at typesetting mathematics. Let $X_1, X_2, \ldots, X_n$ be a sequence of independent and identically distributed random variables with $\text{E}[X_i] = \mu$ and $\text{Var}[X_i] = \sigma^2 < \infty$, and let
$$S_n = \frac{X_1 + X_2 + \cdots + X_n}{n}
      = \frac{1}{n}\sum_{i}^{n} X_i$$
denote their mean. Then as $n$ approaches infinity, the random variables $\sqrt{n}(S_n - \mu)$ converge in distribution to a normal $\mathcal{N}(0, \sigma^2)$.


\subsection{How to create Sections and Subsections}

Use section and subsections to organize your document. Simply use the section and subsection buttons in the toolbar to create them, and we'll handle all the formatting and numbering automatically.

\subsection{How to add Lists}

You can make lists with automatic numbering \dots

\begin{enumerate}
\item Like this,
\item and like this.
\end{enumerate}
\dots or bullet points \dots
\begin{itemize}
\item Like this,
\item and like this.
\end{itemize}

We hope you find Overleaf useful, and please let us know if you have any feedback using the help menu above.

\end{document}

That result in a document like this below:



Intermediate Examples

(combinations of two or more tools using pipes)

Advance Scenarios and Scripts in ATLAS analysis

(review and test of more complex scripts for automation of tasks)

Q&A


smile Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

( please remove SPAMNOT from the email address wink )




Exercises Part 4





Major updates:
-- ArturoS - 2015-01-30

Responsible: ArturoS
Last reviewed by: ArturoS @ cern.ch

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r1 - 2015-12-04 - ArturoS
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Atlas All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback