Difference: ScriptingTutorial (1 vs. 22)

Revision 222015-11-03 - ArturoS

Line: 1 to 1
 
META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

ScriptingTutorial

Line: 18 to 17
 
Deleted:
<
<
ATLASUI integration
  • Second set of examples in Python Vs BASH
  • Local developed Tools
  • Practical Exercises with each the mentioned Tools
  • Exercises Day 3
  • <!-- this line is optional -->
     

    <!-- ---------------------------------------------------------------------------------- -->
    <!-- ---------------------------------------------------------------------------------- -->
    Added:
    >
    >





    Table of Content





    <!--/twistyPlugin twikiMakeVisibleInline-->
    ATLASUI integration
  • Second set of examples in Python Vs BASH
  • Local developed Tools
  • Practical Exercises with each the mentioned Tools
  • Exercises Day 3
  • <!-- this line is optional -->
    <!-- ********************************************************* -->
    <!--/twistyPlugin-->

     



    Changed:
    <
    <
    >
    >
     
    Line: 706 to 744
     

    Changed:
    <
    <
    >
    >
     
    Line: 738 to 776
     
    marketing-tools-for-automation.jpg

    Script concepts using bash

    Added:
    >
    >

    Remember, scripts are used to put to work the computers for you.
    They are a set of instructions that allows to perform repetitive operations in an automatic way. So, let's continue the tutorial creating some useful bash scripts that can be used for repetitive activities relative to your analysis chain:

     

    “Hello” World

    %SYNTAX{ syntax="bash" }%
    Line: 1102 to 1151
     

    Changed:
    <
    <
    >
    >
     

    Revision 212015-11-03 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 223 to 223
     


    Changed:
    <
    <
    Where as you can see, the lines 4,8,12,16 and the only lines that perform an action in this script.
    >
    >
    Where as you can see, the lines 4,8,12,16 are the only lines that perform an action in this script.
     

    Revision 202015-11-02 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 1195 to 1195
     

    The ATLAS Ecosystem and Tools Part II

    PRUN

    Changed:
    <
    <
    (Concept and use)
    >
    >
    (Concept and use) ATLAS Tutorial

    Script of the examples jobs:

    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## Example 1
    prun --exec " echo %IN> input.txt; source Sroot.sh input.txt mini lep_pt lep_n\ \>\ 1" --outDS user.arturos.test.tutorial.$name.job01.V$version --inDS user.arturos.MC.outreach.v2 --nFilesPerJob 1 --outputs output.root --rootVer=6.02/12 --cmtConfig=x86_64-slc6-gcc48-opt
    
    ## Example 2
    prun --exec " echo %IN> input.txt; source ScriptMultipleDraw.sh input.txt mini lep_pt lep_n\ \>\ 0" --outDS user.arturos.test.tutorial.$name.job02.V$version --inDS user.arturos.MC.outreach.v2 --nFilesPerJob 1 --outputs output.root --rootVer=6.02/12 --cmtConfig=x86_64-slc6-gcc48-opt
    
    echo "Done!"
    ## End
    
    <!-- end SyntaxHighlightingPlugin -->
     

    BigPanda

    (Concept and use)
    Added:
    >
    >

    BigPanda Monitor for our jobs (look for your name wink )
     

    RUCIO ( dq2 ? depreciate but still useful )

    (Concept and use)
    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    ## Example 1
    rucio list-dids user.arturos.*.outreach*
    
    
    ## Example 2
    rucio list-files user.arturos.MC.outreach.v2
    
    <!-- end SyntaxHighlightingPlugin -->
     

    AMI

    (Concept and use)

    Revision 192015-11-02 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 1159 to 1159
     

    The GRID

    Changed:
    <
    <
    >
    >
     

    What is a Job?

    Revision 182015-11-01 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 1158 to 1158
     

    The GRID

    Added:
    >
    >
     

    What is a Job?

    What I need?

    Revision 172015-11-01 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 1058 to 1058
     
    Deleted:
    <
    <

    ROOT Reminder

    Review of concepts


    Old tutorial MakeClass and MakeSelector !!

    Example macros for script proposes

    Key commands and executions of ROOT into bash and python scripts (How to call and interact with root and root files)

    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash
    
    root -l file.root << EOF
    $1->Draw()
    c1->SaveAs("$1.eps")
    EOF
    <!-- end SyntaxHighlightingPlugin -->

     

    Q&A


    Line: 1098 to 1075
     
    Deleted:
    <
    <

    The ATLAS Ecosystem and Tools Part I

    The User Interfaces (lxplus and atlasui)

    The GRID

    What is a Job?

    What I need?

    Code

    Samples

    PROXY

    Test of the Tutorial ATLASUI account! (INTERNAL)

    Q&A


    smile Do you have any question so far?
    For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

    ( please remove SPAMNOT from the email address wink )



     
    <!-- ********************************************************* -->
    </>
    <!--/twistyPlugin-->
    Line: 1185 to 1128
     hideimgright="/twiki/pub/TWiki/TWikiDocGraphics/toggleclose-small.gif" }%
    Added:
    >
    >

    ROOT Reminder

    Review of concepts


    Old tutorial MakeClass and MakeSelector !!

    Example macros for script proposes

    Key commands and executions of ROOT into bash and python scripts (How to call and interact with root and root files)

    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash
    
    root -l file.root << EOF
    $1->Draw()
    c1->SaveAs("$1.eps")
    EOF
    <!-- end SyntaxHighlightingPlugin -->

    The ATLAS Ecosystem and Tools Part I

    The User Interfaces (lxplus and atlasui)

    The GRID

    What is a Job?

    What I need?

    Code

    Samples

    PROXY

    Test of the Tutorial ATLASUI account! (INTERNAL)

    Q&A


    smile Do you have any question so far?
    For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

    ( please remove SPAMNOT from the email address wink )



     

    The ATLAS Ecosystem and Tools Part II

    PRUN

    Revision 162015-10-30 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 743 to 743
     %SYNTAX{ syntax="bash" }% #!/bin/bash
    Changed:
    <
    <
    ## This is a comment
    >
    >
    ## This is a comment: we will print Hello World again smile
      echo "Hello World" %ENDSYNTAX%
    Line: 758 to 758
     %SYNTAX{ syntax="bash" }% #!/bin/bash
    Changed:
    <
    <
    ##
    >
    >
    ## Variables coming from "outside", e.g. included as parameters in the script line execution are taken by order using hte character "$"

    first_input=$1 second_input=$2

     
    Added:
    >
    >
    ## etc...

    echo "I am the first input \" $first_input \" and the second input \" second_input\" "

     %ENDSYNTAX%

      • Integers and floats
    Added:
    >
    >
    Variables different the strings are very useful for operations inside a script: counters, maths, iterators, etc...
     %SYNTAX{ syntax="bash" }% #!/bin/bash
    Changed:
    <
    <
    ##
    >
    >
    ## this is a variable taken from outside energy_in_tev=$1

    ## but we always work in MeV of GeV, so a transformation is needed:

    energy_gev=$(($energy_in_tev * 1000))

    energy_in_mev=$[energy_ingev * 1000]

    echo ''The energy for the humans is $energy

      %ENDSYNTAX%
    Line: 787 to 804
     %SYNTAX{ syntax="bash" }% #!/bin/bash
    Changed:
    <
    <
    ##
    >
    >
    HELLO=Hello function hello { local HELLO=World echo $HELLO } echo $HELLO hello echo $HELLO
      %ENDSYNTAX%
    Line: 806 to 830
     
    <!-- ------------------------------------------------------------------------- -->

    Loops

    Added:
    >
    >
    The for loop is a little bit different from other programming languages. Basically, it let's you iterate over a series of 'words' within a string.
    The while executes a piece of code if the control expression is true, and only stops when it is false (or a explicit break is found within the executed code.
    The until loop is almost equal to the while loop, except that the code is executed while the control expression evaluates to false.

    For sample

    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
                for i in $( ls ); do
                    echo item: $i
                done
         
    <!-- end SyntaxHighlightingPlugin -->

    On the second line, we declare i to be the variable that will take the different values contained in $( ls ).
    The third line could be longer if needed, or there could be more lines before the done (4).
    'done' (4) indicates that the code that used the value of $i has finished and $i can take a new value.

      • C-like for

    It's a for loop more similar to C/perl... for. (perl?)

    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
                for i in `seq 1 10`;
                do
                        echo $i
                done    
     
    <!-- end SyntaxHighlightingPlugin -->

      • While sample

    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash  
    
                 COUNTER=0
                 while [  $COUNTER -lt 10 ]; do
                     echo The counter is $COUNTER
                     let COUNTER=COUNTER+1 
                 done
    <!-- end SyntaxHighlightingPlugin -->

    This script 'emulates' the well known (C, Pascal, perl, etc) 'for' structure

      • Until sample

    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash  
    
                 COUNTER=20
                 until [  $COUNTER -lt 10 ]; do
                     echo COUNTER $COUNTER
                     let COUNTER-=1
                 done
     
    <!-- end SyntaxHighlightingPlugin -->

    • Note: Loop in python smile We will talk about this later as well...
      %CODE{"sh" num="1"}% #!/bin/sh
    Line: 819 to 920
     
    Deleted:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    ## This is a comment
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->
     

    Conditionals

    Added:
    >
    >
    Conditionals have many forms. The most basic form is: if expression then statement where 'statement' is only executed if 'expression' evaluates to true. '2<1' is an expresion that evaluates to false, while '2>1' evaluates to true. Conditionals have other forms such as: if expression then statement1 else statement2. Here 'statement1' is executed if 'expression' is true,otherwise 'statement2' is executed.

    - using numerical values
    - using strings values
    - conditions for break, as before in loops
    - conditions in “if”
    - if, else if, else

    A word about syntax:
    The base for the 'if' constructions in bash is this:

    if [expression];
    
    then
    
    code if 'expression' is true.
    
    fi
    

    Basic conditional example if .. then

     %SYNTAX{ syntax="bash" }%
    Deleted:
    <
    <
    #!/bin/bash
     
    Changed:
    <
    <
    ## This is a comment
    >
    >
    #!/bin/bash if [ "foo" = "foo" ]; then echo expression evaluated as true fi
     
    Deleted:
    <
    <
    echo "Hello World"
     %ENDSYNTAX%
    Changed:
    <
    <

    Pipes

    >
    >
    The code to be executed if the expression within braces is true can be found after the 'then' word and before 'fi' which indicates the end of the conditionally executed code.

    • Basic conditional example if .. then ... else
     %SYNTAX{ syntax="bash" }% #!/bin/bash
    Added:
    >
    >
    if [ "foo" = "foo" ]; then echo expression evaluated as true else echo expression evaluated as false fi
     
    Changed:
    <
    <
    ## This is a comment
    >
    >
    %ENDSYNTAX%

    • Conditionals with variables

    %SYNTAX{ syntax="bash" }% #!/bin/bash T1="foo" T2="bar" if [ "$T1" = "$T2" ]; then echo expression evaluated as true else echo expression evaluated as false fi

     
    Deleted:
    <
    <
    echo "Hello World"
     %ENDSYNTAX%
    Changed:
    <
    <

    Inputs and Outputs

    >
    >

    Pipes


    - echo, grep (i, v, c), cat, comm, tail -n X, head -n Y, wc (l,…) as inputs for redirection to files, sed, wc, grep, sort, uniq,...

     %SYNTAX{ syntax="bash" }% #!/bin/bash
    Line: 851 to 1011
     echo "Hello World" %ENDSYNTAX%
    Changed:
    <
    <

    Functions

    >
    >

    Inputs and Outputs

     %SYNTAX{ syntax="bash" }% #!/bin/bash
    Line: 860 to 1020
     echo "Hello World" %ENDSYNTAX%
    Changed:
    <
    <

    Working with text files

    >
    >

    Functions (Moved to Day 3)

    - sleep and wait
    - define a function
    - call functions
    - remember of local variables
    - use of the .bash_profile and .bash_login file for define functions and scripts in user system/login

     %SYNTAX{ syntax="bash" }% #!/bin/bash

    ## This is a comment

    Deleted:
    <
    <
    echo "Hello World"
     %ENDSYNTAX%
    Line: 890 to 1061
     

    ROOT Reminder

    Review of concepts

    Added:
    >
    >

    Old tutorial MakeClass and MakeSelector !!
     

    Example macros for script proposes

    Added:
    >
    >
    Key commands and executions of ROOT into bash and python scripts (How to call and interact with root and root files)
      %SYNTAX{ syntax="bash" }% #!/bin/bash
    Line: 901 to 1078
     c1->SaveAs("$1.eps") EOF %ENDSYNTAX%
    Added:
    >
    >

     

    Q&A

    Line: 920 to 1098
     
    Changed:
    <
    <

    First set of examples in Python Vs BASH

    %SYNTAX{ syntax="python" }% #!/usr/bin/python

    >
    >

    The ATLAS Ecosystem and Tools Part I

     
    Changed:
    <
    <
    import os, math, sys
    >
    >

    The User Interfaces (lxplus and atlasui)

     
    Changed:
    <
    <
    ############################# # # Script for # ############################# %ENDSYNTAX%
    >
    >

    The GRID

     
    Added:
    >
    >

    What is a Job?

     
    Added:
    >
    >

    What I need?

     
    Changed:
    <
    <

    One to One equivalent examples for generic use

    >
    >

    Code

     
    Added:
    >
    >

    Samples

     
    Changed:
    <
    <

    Online resources and references

    >
    >

    PROXY

     
    Added:
    >
    >

    Test of the Tutorial ATLASUI account! (INTERNAL)

     

    Q&A


    Line: 1010 to 1185
     hideimgright="/twiki/pub/TWiki/TWikiDocGraphics/toggleclose-small.gif" }%
    Changed:
    <
    <

    Introduction to ATLAS Ecosystem and Tools

    The User Interfaces (lxplus and atlasui)

    The GRID

    What is a Job?

    What I need?

    Code

    Samples

    PROXY

    ATLAS practical Tools

    >
    >

    The ATLAS Ecosystem and Tools Part II

     

    PRUN

    (Concept and use)

    Revision 152015-10-30 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 893 to 893
     

    Example macros for script proposes

    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash
    
    root -l file.root << EOF
    $1->Draw()
    c1->SaveAs("$1.eps")
    EOF
    <!-- end SyntaxHighlightingPlugin -->
     

    Q&A


    Revision 142015-10-30 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 749 to 749
     %ENDSYNTAX%

    Variables

    Added:
    >
    >
    Lets use some variables to get a feeling of the different use. As in previous sections, they are some of the many ways to use variables but we consider the next examples a representative sample for the majority of the scripts that you may will developed:

    • Define variables coming from the user input:
      • Strings
     %SYNTAX{ syntax="bash" }% #!/bin/bash
    Changed:
    <
    <
    ## This is a comment
    >
    >
    ##

    %ENDSYNTAX%

      • Integers and floats
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## 
    
    <!-- end SyntaxHighlightingPlugin -->

    • Define variables inside the scripts

    As in programming languages like C++, we can have global and local variables in our scripts:

      • Global variables
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## 
    
    <!-- end SyntaxHighlightingPlugin -->

      • Local variables
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## 
    
    <!-- end SyntaxHighlightingPlugin -->

      • Using variables in operations: printing, assigning, concatenating, arithmetic operations:

    %SYNTAX{ syntax="bash" }% #!/bin/bash

    ##

     
    Deleted:
    <
    <
    echo "Hello World"
     %ENDSYNTAX%
    Added:
    >
    >

     
    Added:
    >
    >
    <!-- ------------------------------------------------------------------------- -->
     

    Loops

    Revision 132015-10-28 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 55 to 55
     }%
    Changed:
    <
    <
    test_automation-1024x837.jpg
    >
    >
    test_automation-1024x837.jpg
     

    Introduction: Automation

    Line: 63 to 63
     
    Changed:
    <
    <
    Picking into the large amount of different concepts with can find, an useful definition is: The use of tools, techniques and ingenuous in other to create algorithms that allow you to request to the computer to perform a serie of complex tasks, in a sequential mode, with a deterministic frequency in an automatic way with a single execution o command.
    >
    >
    Picking into the large amount of different concepts with can find, an useful definition can be: The use of tools, techniques and ingenuous in other to create algorithms that allow you to request to the computer to perform a series of complex tasks, in a sequential mode, for an single execution or with a deterministic frequency, in an automatic way with a single execution o command line.
     
    Line: 75 to 75
     
    Changed:
    <
    <
    Is the translation to a particular language the set of instructions result of the automation algorithm's development for what you need/want this automatic execution(s),
    >
    >
    It's the translation to a particular language of the set of instructions resulted of the automation algorithm's development for what you need and/or want this automatic execution(s),
     
    Line: 83 to 83
      %SYNTAX{ syntax="bash" }%
    Changed:
    <
    <
    ## Hi, this is a set of commands into a script:
    >
    >
    ## Hi!, this is a set of commands into a script:
      setup some variables
    Line: 111 to 111
     
    Changed:
    <
    <
    Scripts are used to put to work the computers for you. They are a set of instructions that allows to perform repetitive operations in an automatic way, so, just your imagination can define the limits of an script. Starting for simple checks to your current working area, passing for setup of simple of complex environments, or for backup creations to interactions with web-based (internet) resources, to integration of databases, online monitoring of resources to... malware... the reasons to have scripts is basically to get something done as far and human-independent as possible. And as we will see, we can wrap several different applications and tools into a single set of instructions, so, it is like to mix many different software into a single box:
    >
    >
    Scripts are used to put to work the computers for you.
    They are a set of instructions that allows to perform repetitive operations in an automatic way. So, just your imagination can define the limits of a script. Starting for simple checks into your current working area, passing for the setup of simple of complex environments, or for backup creation to interactions with web-based (internet) resources, including the integration of databases, online monitoring of resources to... malware... the reason to have scripts is basically to get something done as far and human-independent as possible. And as we will see, we can wrap several different applications and tools into a single set of instructions, so, it is like to mix many different software into a single box:
     
    Line: 169 to 170
     
    Changed:
    <
    <

    Pre-review of bash scripts

    >
    >

    Anatomy of a first script (bash)

      %SYNTAX{ syntax="bash" }% #!/bin/bash
    Line: 225 to 226
     Where as you can see, the lines 4,8,12,16 and the only lines that perform an action in this script.
    Added:
    >
    >
     

    Linux (UNIX) command refreshing review


    Added:
    >
    >
    lego.jpg
      LINUX has an almost infinite number of commands and shell apps that can do virtually any contemporary operation in a computer or network of computers. some of them are:
    Deleted:
    <
    <
    lego.jpg
     cat %SYNTAX{ syntax="zsh" }% > cat ## Concatenate and print (display) the content of files
    Line: 663 to 665
     %ENDSYNTAX%
    Deleted:
    <
    <

    HTML fast view

    <!-- SyntaxHighlightingPlugin -->
    Old HTML
    <!-- end SyntaxHighlightingPlugin -->

    <!-- SyntaxHighlightingPlugin -->
    Scripts
    <!-- end SyntaxHighlightingPlugin -->

    LaTeX

    <!-- SyntaxHighlightingPlugin -->
    \documentclass[a4paper]{article}
    
    \usepackage[english]{babel}
    \usepackage[utf8x]{inputenc}
    \usepackage{amsmath}
    \usepackage{graphicx}
    \usepackage[colorinlistoftodos]{todonotes}
    
    \title{Your Paper}
    \author{You}
    
    \begin{document}
    \maketitle
    
    \begin{abstract}
    Your abstract.
    \end{abstract}
    
    \section{Introduction}
    
    Your introduction goes here! Some examples of commonly used commands and features are listed below, to help you get started. If you have a question, please use the help menu (``?'') on the top bar to search for help or ask us a question. 
    
    \section{Some examples to get started}
    
    \subsection{How to add Comments}
    
    Comments can be added to your project by clicking on the comment icon in the toolbar above. % * <john.hammersley@gmail.com> 2014-09-03T09:54:16.211Z:
    %
    % Here's an example comment!
    %
    To reply to a comment, simply click the reply button in the lower right corner of the comment, and you can close them when you're done.
    
    \subsection{How to include Figures}
    
    First you have to upload the image file from your computer using the upload link the project menu. Then use the includegraphics command to include it in your document. Use the figure environment and the caption command to add a number and a caption to your figure. See the code for Figure \ref{fig:frog} in this section for an example.
    
    \begin{figure}
    \centering
    \includegraphics[width=0.3\textwidth]{frog.jpg}
    \caption{\label{fig:frog}This frog was uploaded via the project menu.}
    \end{figure}
    
    \subsection{How to add Tables}
    
    Use the table and tabular commands for basic tables --- see Table~\ref{tab:widgets}, for example. 
    
    \begin{table}
    \centering
    \begin{tabular}{l|r}
    Item & Quantity \\\hline
    Widgets & 42 \\
    Gadgets & 13
    \end{tabular}
    \caption{\label{tab:widgets}An example table.}
    \end{table}
    
    \subsection{How to write Mathematics}
    
    \LaTeX{} is great at typesetting mathematics. Let $X_1, X_2, \ldots, X_n$ be a sequence of independent and identically distributed random variables with $\text{E}[X_i] = \mu$ and $\text{Var}[X_i] = \sigma^2 < \infty$, and let
    $$S_n = \frac{X_1 + X_2 + \cdots + X_n}{n}
          = \frac{1}{n}\sum_{i}^{n} X_i$$
    denote their mean. Then as $n$ approaches infinity, the random variables $\sqrt{n}(S_n - \mu)$ converge in distribution to a normal $\mathcal{N}(0, \sigma^2)$.
    
    
    \subsection{How to create Sections and Subsections}
    
    Use section and subsections to organize your document. Simply use the section and subsection buttons in the toolbar to create them, and we'll handle all the formatting and numbering automatically.
    
    \subsection{How to add Lists}
    
    You can make lists with automatic numbering \dots
    
    \begin{enumerate}
    \item Like this,
    \item and like this.
    \end{enumerate}
    \dots or bullet points \dots
    \begin{itemize}
    \item Like this,
    \item and like this.
    \end{itemize}
    
    We hope you find Overleaf useful, and please let us know if you have any feedback using the help menu above.
    
    \end{document}
    <!-- end SyntaxHighlightingPlugin -->

    That result in a document like this below:


     

    Q&A


    Line: 840 to 735
     

    Review of concepts with examples:

    Deleted:
    <
    <

    Script concepts using bash

     
    marketing-tools-for-automation.jpg
    Added:
    >
    >

    Script concepts using bash

     

    “Hello” World

    %SYNTAX{ syntax="bash" }% #!/bin/bash
    Line: 1171 to 1067
     

    Initial Examples

    (review concepts)
    Added:
    >
    >

    HTML fast view

    <!-- SyntaxHighlightingPlugin -->
    Old HTML
    <!-- end SyntaxHighlightingPlugin -->

    <!-- SyntaxHighlightingPlugin -->
    Scripts
    <!-- end SyntaxHighlightingPlugin -->

    LaTeX

    Text....

    <!--/twistyPlugin twikiMakeVisibleInline-->

    <!-- SyntaxHighlightingPlugin -->
    \documentclass[a4paper]{article}
    
    \usepackage[english]{babel}
    \usepackage[utf8x]{inputenc}
    \usepackage{amsmath}
    \usepackage{graphicx}
    \usepackage[colorinlistoftodos]{todonotes}
    
    \title{Your Paper}
    \author{You}
    
    \begin{document}
    \maketitle
    
    \begin{abstract}
    Your abstract.
    \end{abstract}
    
    \section{Introduction}
    
    Your introduction goes here! Some examples of commonly used commands and features are listed below, to help you get started. If you have a question, please use the help menu (``?'') on the top bar to search for help or ask us a question. 
    
    \section{Some examples to get started}
    
    \subsection{How to add Comments}
    
    Comments can be added to your project by clicking on the comment icon in the toolbar above. % * <john.hammersley@gmail.com> 2014-09-03T09:54:16.211Z:
    %
    % Here's an example comment!
    %
    To reply to a comment, simply click the reply button in the lower right corner of the comment, and you can close them when you're done.
    
    \subsection{How to include Figures}
    
    First you have to upload the image file from your computer using the upload link the project menu. Then use the includegraphics command to include it in your document. Use the figure environment and the caption command to add a number and a caption to your figure. See the code for Figure \ref{fig:frog} in this section for an example.
    
    \begin{figure}
    \centering
    \includegraphics[width=0.3\textwidth]{frog.jpg}
    \caption{\label{fig:frog}This frog was uploaded via the project menu.}
    \end{figure}
    
    \subsection{How to add Tables}
    
    Use the table and tabular commands for basic tables --- see Table~\ref{tab:widgets}, for example. 
    
    \begin{table}
    \centering
    \begin{tabular}{l|r}
    Item & Quantity \\\hline
    Widgets & 42 \\
    Gadgets & 13
    \end{tabular}
    \caption{\label{tab:widgets}An example table.}
    \end{table}
    
    \subsection{How to write Mathematics}
    
    \LaTeX{} is great at typesetting mathematics. Let $X_1, X_2, \ldots, X_n$ be a sequence of independent and identically distributed random variables with $\text{E}[X_i] = \mu$ and $\text{Var}[X_i] = \sigma^2 < \infty$, and let
    $$S_n = \frac{X_1 + X_2 + \cdots + X_n}{n}
          = \frac{1}{n}\sum_{i}^{n} X_i$$
    denote their mean. Then as $n$ approaches infinity, the random variables $\sqrt{n}(S_n - \mu)$ converge in distribution to a normal $\mathcal{N}(0, \sigma^2)$.
    
    
    \subsection{How to create Sections and Subsections}
    
    Use section and subsections to organize your document. Simply use the section and subsection buttons in the toolbar to create them, and we'll handle all the formatting and numbering automatically.
    
    \subsection{How to add Lists}
    
    You can make lists with automatic numbering \dots
    
    \begin{enumerate}
    \item Like this,
    \item and like this.
    \end{enumerate}
    \dots or bullet points \dots
    \begin{itemize}
    \item Like this,
    \item and like this.
    \end{itemize}
    
    We hope you find Overleaf useful, and please let us know if you have any feedback using the help menu above.
    
    \end{document}
    <!-- end SyntaxHighlightingPlugin -->

    That result in a document like this below:


    <!--/twistyPlugin-->

     

    Intermediate Examples

    (combinations of two or more tools using pipes)

    Revision 122015-10-28 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 784 to 784
     ( please remove SPAMNOT from the email address wink )

    Added:
    >
    >
    <!-- ********************************************************* -->
    </>
    <!--/twistyPlugin-->
     
    Added:
    >
    >

    Exercises Day 1

    <!--/twistyPlugin twikiMakeVisibleInline-->


     
    <!-- ********************************************************* -->
    <!--/twistyPlugin-->

    Line: 991 to 1009
     </>
    <!--/twistyPlugin-->

    Added:
    >
    >

    Exercises Day 2

    <!--/twistyPlugin twikiMakeVisibleInline-->


    <!-- ********************************************************* -->
    <!--/twistyPlugin-->

     

    Line: 1160 to 1197
     
    <!-- ********************************************************* -->
    </>
    <!--/twistyPlugin-->

    Added:
    >
    >

    Exercises Day 3

    <!--/twistyPlugin twikiMakeVisibleInline-->


    <!-- ********************************************************* -->
    <!--/twistyPlugin-->

     

    Revision 112015-10-28 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 54 to 54
     hideimgright="/twiki/pub/TWiki/TWikiDocGraphics/toggleclose-small.gif" }%
    Deleted:
    <
    <

    Introduction

     
    Added:
    >
    >
    test_automation-1024x837.jpg
     
    Changed:
    <
    <

    Automatisation

    >
    >

    Introduction: Automation

     
    Line: 74 to 75
     
    Changed:
    <
    <
    Is the translation to a particular language the set of instructions result of the automatisation algorithm's development for what you need/want this automatic execution(s),
    >
    >
    Is the translation to a particular language the set of instructions result of the automation algorithm's development for what you need/want this automatic execution(s),
     
    Line: 119 to 120
     

    An pictorial view of scripts in an analysis


    Changed:
    <
    <
    Grphical_view_scripts.png
    >
    >
    Grphical_view_scripts.png
     
    Line: 194 to 195
     %ENDSYNTAX%


    Changed:
    <
    <
    Now, lets use some LINUX/UNIX terminal commands into a script. Here you can start to see what I mean with: Automatisation. We are "replacing" your fingers (the human) for a scripts (a robot) that can do exactly the same:
    >
    >
    Now, lets use some LINUX/UNIX terminal commands into a script. Here you can start to see what I mean with: Automation. We are "replacing" your fingers (the human) for a scripts (a robot) that can do exactly the same:
     

    Line: 225 to 226
     

    Linux (UNIX) command refreshing review

    Deleted:
    <
    <
     
    Changed:
    <
    <

    >
    >
    LINUX has an almost infinite number of commands and shell apps that can do virtually any contemporary operation in a computer or network of computers. some of them are:

    lego.jpg

    cat

     %SYNTAX{ syntax="zsh" }%
    Changed:
    <
    <
    > command n
    >
    >
    > cat ## Concatenate and print (display) the content of files
     %ENDSYNTAX%
    Deleted:
    <
    <

     
    Changed:
    <
    <

    >
    >
    cd
    <!-- SyntaxHighlightingPlugin -->
    > cd       ## Change Directory
    
    <!-- end SyntaxHighlightingPlugin -->
     
    Changed:
    <
    <
    %SYNTAX{ syntax="bash" }% Scripts
    >
    >
    chmod %SYNTAX{ syntax="zsh" }% > chmod ## Change access permissions
     %ENDSYNTAX%
    Deleted:
    <
    <

     
    Changed:
    <
    <

    >
    >
    clear
     %SYNTAX{ syntax="zsh" }%
    Changed:
    <
    <
    ## to include a black space into a command line, use an "\" and add the needed space, for example: > echo Ciao Mondo
    Ciao Mondo
    >
    >
    > clear ## Clear terminal screen %ENDSYNTAX%
     
    Changed:
    <
    <
    > echo Ciao\ Mundo
    Ciao Mondo
    >
    >
    comm
    <!-- SyntaxHighlightingPlugin -->
    > comm     ## Compare two sorted files line by line
    
    <!-- end SyntaxHighlightingPlugin -->
     
    Changed:
    <
    <
    > echo Ciao Mondo
    Ciao Mondo
    >
    >
    cp
    <!-- SyntaxHighlightingPlugin -->
    > cp       ## Copy one or more files to another location
    
    <!-- end SyntaxHighlightingPlugin -->
     
    Changed:
    <
    <
    > echo Ciao\ \ \ \ \ Mondo
    Ciao Mondo
    >
    >
    cut
    <!-- SyntaxHighlightingPlugin -->
    > cut      ## Divide a file into several parts
    
    <!-- end SyntaxHighlightingPlugin -->
     
    Added:
    >
    >
    date %SYNTAX{ syntax="zsh" }% > date ## Display or change the date & time
     %ENDSYNTAX%
    Deleted:
    <
    <

     
    Changed:
    <
    <


    >
    >
    diff
    <!-- SyntaxHighlightingPlugin -->
    > diff     ## Display the differences between two files
    
    <!-- end SyntaxHighlightingPlugin -->
     
    Changed:
    <
    <

    >
    >
    du
     %SYNTAX{ syntax="zsh" }%
    Changed:
    <
    <
    > command n
    >
    >
    > du ## Estimate file space usage
     %ENDSYNTAX%
    Deleted:
    <
    <

     
    Changed:
    <
    <


    >
    >
    echo
    <!-- SyntaxHighlightingPlugin -->
    > echo     ## Display message on screen •
    
    <!-- end SyntaxHighlightingPlugin -->
     
    Changed:
    <
    <

    >
    >
    egrep
    <!-- SyntaxHighlightingPlugin -->
    > egrep    ## Search file(s) for lines that match an extended expression
    
    <!-- end SyntaxHighlightingPlugin -->

    exit

    <!-- SyntaxHighlightingPlugin -->
    > exit     ## Exit the shell
    
    <!-- end SyntaxHighlightingPlugin -->

    export

    <!-- SyntaxHighlightingPlugin -->
    > export   ## Set an environment variable
    
    <!-- end SyntaxHighlightingPlugin -->

    find

    <!-- SyntaxHighlightingPlugin -->
    > find     ## Search for files that meet a desired criteria
    
    <!-- end SyntaxHighlightingPlugin -->

    grep

    <!-- SyntaxHighlightingPlugin -->
    > grep     ## Search file(s) for lines that match a given pattern
    
    <!-- end SyntaxHighlightingPlugin -->

    gzip

    <!-- SyntaxHighlightingPlugin -->
    > gzip     ## Compress or decompress named file(s)
    
    <!-- end SyntaxHighlightingPlugin -->

    history

    <!-- SyntaxHighlightingPlugin -->
    > history  ## Command History
    
    <!-- end SyntaxHighlightingPlugin -->

    kill

    <!-- SyntaxHighlightingPlugin -->
    > kill     ## Kill a process by specifying its PID
    
    <!-- end SyntaxHighlightingPlugin -->

    make

    <!-- SyntaxHighlightingPlugin -->
    > make     ## Recompile a group of programs
    
    <!-- end SyntaxHighlightingPlugin -->

    mkdir

    <!-- SyntaxHighlightingPlugin -->
    > mkdir    ## Create new folder(s)
    
    <!-- end SyntaxHighlightingPlugin -->

    more

    <!-- SyntaxHighlightingPlugin -->
    > more     ## Display output one screen at a time
    
    <!-- end SyntaxHighlightingPlugin -->

    mv

    <!-- SyntaxHighlightingPlugin -->
    > mv       ## Move or rename files or directories
    
    <!-- end SyntaxHighlightingPlugin -->

    nohup

    <!-- SyntaxHighlightingPlugin -->
    > nohup    ## Run a command immune to hangups
    
    <!-- end SyntaxHighlightingPlugin -->

    passwd

    <!-- SyntaxHighlightingPlugin -->
    > passwd   ## Modify a user password
    
    <!-- end SyntaxHighlightingPlugin -->

    paste

    <!-- SyntaxHighlightingPlugin -->
    > paste    ## Merge lines of files
    
    <!-- end SyntaxHighlightingPlugin -->

    ping

    <!-- SyntaxHighlightingPlugin -->
    > ping     ## Test a network connection
    
    <!-- end SyntaxHighlightingPlugin -->

    ps

    <!-- SyntaxHighlightingPlugin -->
    > ps       ## Process status
    
    <!-- end SyntaxHighlightingPlugin -->

    pwd

    <!-- SyntaxHighlightingPlugin -->
    > pwd      ## Print Working Directory
    
    <!-- end SyntaxHighlightingPlugin -->

    rcp

     %SYNTAX{ syntax="zsh" }%
    Changed:
    <
    <
    > command n
    >
    >
    > rcp ## Copy files between two machines %ENDSYNTAX%

    rm

    <!-- SyntaxHighlightingPlugin -->
    > rm       ## Remove files
    
    <!-- end SyntaxHighlightingPlugin -->

    scp

    <!-- SyntaxHighlightingPlugin -->
    > scp      ## Secure copy (remote file copy)
    
    <!-- end SyntaxHighlightingPlugin -->

    sed

    <!-- SyntaxHighlightingPlugin -->
    > sed      ## Stream Editor
    
    <!-- end SyntaxHighlightingPlugin -->

    sleep

    <!-- SyntaxHighlightingPlugin -->
    > sleep    ## Delay for a specified time
    
    <!-- end SyntaxHighlightingPlugin -->

    sort

    <!-- SyntaxHighlightingPlugin -->
    > sort     ## Sort text files
    
    <!-- end SyntaxHighlightingPlugin -->

    source

    <!-- SyntaxHighlightingPlugin -->
    > source   ## Run commands from a file '.'
    
    <!-- end SyntaxHighlightingPlugin -->

    su

    <!-- SyntaxHighlightingPlugin -->
    > su       ## Substitute user identity
    
    <!-- end SyntaxHighlightingPlugin -->

    tail

    <!-- SyntaxHighlightingPlugin -->
    > tail     ## Output the last part of file
    
    <!-- end SyntaxHighlightingPlugin -->

    time

    <!-- SyntaxHighlightingPlugin -->
    > time     ## Measure Program running time
    
    <!-- end SyntaxHighlightingPlugin -->

    uniq

    <!-- SyntaxHighlightingPlugin -->
    > uniq     ## Uniquify files
    
    <!-- end SyntaxHighlightingPlugin -->

    vi

    <!-- SyntaxHighlightingPlugin -->
    > vi       ## Text Editor
    > vim       ## Text Editor
    
    <!-- end SyntaxHighlightingPlugin -->

    wait

    <!-- SyntaxHighlightingPlugin -->
    > wait     ## Wait for a process to complete •
    
    <!-- end SyntaxHighlightingPlugin -->

    wc

    <!-- SyntaxHighlightingPlugin -->
    > wc       ## Print byte, word, and line counts
    
    <!-- end SyntaxHighlightingPlugin -->

    wget %SYNTAX{ syntax="zsh" }% > wget ## Retrieve web pages or files via HTTP, HTTPS or FTP

     %ENDSYNTAX%
    Added:
    >
    >
    Another interesting use is the inclusion of special characters as is the case of a blank space:
     
    Added:
    >
    >
    %SYNTAX{ syntax="zsh" }% ## to include a black space into a command line, use an "\" and add the needed space, for example: > echo Ciao Mondo
    Ciao Mondo
     
    Added:
    >
    >
    > echo Ciao\ Mundo
    Ciao Mondo
     
    Added:
    >
    >
    > echo Ciao Mondo
    Ciao Mondo
     
    Added:
    >
    >
    > echo Ciao\ \ \ \ \ Mondo
    Ciao Mondo
     
    Added:
    >
    >
    %ENDSYNTAX%
     
    Deleted:
    <
    <

    Some scripts in action

     
    Added:
    >
    >

    Some scripts in action

    puzzle-3d-1.jpg
     

    Generic automatic Email for Account "management"


    Line: 299 to 491
     ## Updated: 28th October 2015 ## ## ## ## How to use this script: ##
    Changed:
    <
    <
    ## > ./new_user.sh Arturo\ Sanchez arturos 1234 > email_new_users/email_arturos.txt ##
    >
    >
    ## > ./new_user.sh Name\ LastName nickname 12345 > email_new_users/email_nickname.txt ##
     ## Example: ## ## > ./new_user.sh <Name\ LastName> > email_new_users/email_.txt ## ###################################################################################################
    Line: 370 to 562
     echo "In case of any question/suggestion please let me know," echo "" echo "Cheers,"
    Changed:
    <
    <
    echo "Arturo"
    >
    >
    echo "Account Managers"
     echo "------" echo "" %ENDCODE%
    Line: 448 to 640
     In case of any question/suggestion please let me know,

    Cheers,

    Changed:
    <
    <
    Arturo
    >
    >
    Account Managers
     
    %ENDSYNTAX%
    Line: 459 to 651
     %SYNTAX{ syntax="bash" }% #!/bin/bash
    Changed:
    <
    <
    ## This is a comment
    >
    >
    ## This is a Variable that is filled with the input #1 firstWord=$1
     
    Changed:
    <
    <
    echo "Hello World"
    >
    >
    ## Other variable defined inside the script secondWord="World"

    echo "Hello $1 $secondWord"

    ## End

     %ENDSYNTAX%
    Line: 477 to 675
     %ENDSYNTAX%
    Added:
    >
    >

    LaTeX

    %SYNTAX{ syntax="tex" }% \documentclass[a4paper]{article}
     
    Changed:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    New HTML
    <!-- end SyntaxHighlightingPlugin -->
    >
    >
    \usepackage[english]{babel} \usepackage[utf8x]{inputenc} \usepackage{amsmath} \usepackage{graphicx} \usepackage[colorinlistoftodos]{todonotes}

    \title{Your Paper} \author{You}

    \begin{document} \maketitle

    \begin{abstract} Your abstract. \end{abstract}

    \section{Introduction}

    Your introduction goes here! Some examples of commonly used commands and features are listed below, to help you get started. If you have a question, please use the help menu (``?'') on the top bar to search for help or ask us a question.

    \section{Some examples to get started}

    \subsection{How to add Comments}

    Comments can be added to your project by clicking on the comment icon in the toolbar above. % * <john.hammersley@gmail.com> 2014-09-03T09:54:16.211Z: % % Here's an example comment! % To reply to a comment, simply click the reply button in the lower right corner of the comment, and you can close them when you're done.

    \subsection{How to include Figures}

     
    Added:
    >
    >
    First you have to upload the image file from your computer using the upload link the project menu. Then use the includegraphics command to include it in your document. Use the figure environment and the caption command to add a number and a caption to your figure. See the code for Figure \ref{fig:frog} in this section for an example.
     
    Added:
    >
    >
    \begin{figure} \centering \includegraphics[width=0.3\textwidth]{frog.jpg} \caption{\label{fig:frog}This frog was uploaded via the project menu.} \end{figure}

    \subsection{How to add Tables}

    Use the table and tabular commands for basic tables --- see Table~\ref{tab:widgets}, for example.

    \begin{table} \centering \begin{tabular}{l|r} Item & Quantity \\\hline Widgets & 42 \Gadgets & 13 \end{tabular} \caption{\label{tab:widgets}An example table.} \end{table}

    \subsection{How to write Mathematics}

    \LaTeX{} is great at typesetting mathematics. Let $X_1, X_2, \ldots, X_n$ be a sequence of independent and identically distributed random variables with $\text{E}[X_i] = \mu$ and $\text{Var}[X_i] = \sigma^2 < \infty$, and let $$S_n = \frac{X_1 + X_2 + \cdots + X_n}{n} = \frac{1}{n}\sum_{i}^{n} X_i$$ denote their mean. Then as $n$ approaches infinity, the random variables $\sqrt{n}(S_n - \mu)$ converge in distribution to a normal $\mathcal{N}(0, \sigma^2)$.

    \subsection{How to create Sections and Subsections}

    Use section and subsections to organize your document. Simply use the section and subsection buttons in the toolbar to create them, and we'll handle all the formatting and numbering automatically.

    \subsection{How to add Lists}

    You can make lists with automatic numbering \dots

    \begin{enumerate} \item Like this, \item and like this. \end{enumerate} \dots or bullet points \dots \begin{itemize} \item Like this, \item and like this. \end{itemize}

    We hope you find Overleaf useful, and please let us know if you have any feedback using the help menu above.

    \end{document} %ENDSYNTAX%

    That result in a document like this below:


     

    Q&A


    Line: 538 to 824
     

    Script concepts using bash

    Added:
    >
    >
    marketing-tools-for-automation.jpg
     

    “Hello” World

    %SYNTAX{ syntax="bash" }% #!/bin/bash
    Line: 807 to 1094
     

    Second set of examples in Python Vs BASH

    (Examples of the use in scripts for analysers are included in each point)
    Added:
    >
    >
    poseswithbrick.png
     

    One to One equivalent examples for generic use

    Online resources and references

    Line: 849 to 1139
      (combinations of two or more tools using pipes)

    Advance Scenarios and Scripts in ATLAS analysis

    Changed:
    <
    <
    (review and test of more complex scripts for automatisation of tasks)
    >
    >
    (review and test of more complex scripts for automation of tasks)
     

    Q&A


    Line: 892 to 1182
     
    Added:
    >
    >
     
    META FILEATTACHMENT attachment="Screenshot_2015-10-21_17.34.46.png" attr="h" comment="" date="1445892756" name="Screenshot_2015-10-21_17.34.46.png" path="Screenshot_2015-10-21_17.34.46.png" size="88312" user="arturos" version="1"
    META FILEATTACHMENT attachment="Grphical_view_scripts.png" attr="h" comment="graphical view script working" date="1445970045" name="Grphical_view_scripts.png" path="Grphical_view_scripts.png" size="163482" user="arturos" version="1"
    Added:
    >
    >
    META FILEATTACHMENT attachment="test_script.pdf" attr="h" comment="" date="1445989209" name="test_script.pdf" path="test_script.pdf" size="229560" user="arturos" version="1"
    META FILEATTACHMENT attachment="puzzle-3d-1.jpg" attr="h" comment="" date="1445994141" name="puzzle-3d-1.jpg" path="puzzle-3d-1.jpg" size="421694" user="arturos" version="1"
    META FILEATTACHMENT attachment="marketing-tools-for-automation.jpg" attr="h" comment="" date="1445994771" name="marketing-tools-for-automation.jpg" path="marketing-tools-for-automation.jpg" size="173675" user="arturos" version="1"
    META FILEATTACHMENT attachment="test_automation-1024x837.jpg" attr="h" comment="" date="1445994772" name="test_automation-1024x837.jpg" path="test_automation-1024x837.jpg" size="165881" user="arturos" version="1"
    META FILEATTACHMENT attachment="poseswithbrick.png" attr="h" comment="" date="1445994772" name="poseswithbrick.png" path="poseswithbrick.png" size="208106" user="arturos" version="1"
    META FILEATTACHMENT attachment="lego.jpg" attr="h" comment="" date="1445994773" name="lego.jpg" path="lego.jpg" size="306649" user="arturos" version="1"

    Revision 102015-10-27 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 164 to 164
     ( please remove SPAMNOT from the email address wink )

    Added:
    >
    >
     
    Changed:
    <
    <

    Pre-Review of bash scripts

    >
    >

    Pre-review of bash scripts

      %SYNTAX{ syntax="bash" }% #!/bin/bash
    Changed:
    <
    <
    ## This is a comment
    >
    >
    ## The line above helps to let the system know that you are asking for the bash shell

    ## As you can see, using #'s allows you to introduce comments. ## Comments are EXTREMELY useful for self documentation of the code! ## Even if you never share the script (or you code in general) you will thank yourself after few weeks/months and you come back to check what you actually did smile ## And if you share your code, is a nice way to let your user or college know that you follow one of the Good Practices in Programming and Computer Science in general

    ## A first sugestion is to add to every script you plan to keep for more that few hours: ## - Date of creation: XX/YY/ZZ ## - Date of last update: xx/yy/zz ## - Contact email

    ## Saying all that (excuse me) lets create a first script following the traditional "Hello World" ## "echo" is a function that will print in a command-shell -terminal- what you put after the "echo" word:

      echo "Hello World"
    Added:
    >
    >
    ## End
     %ENDSYNTAX%
    Added:
    >
    >

    Now, lets use some LINUX/UNIX terminal commands into a script. Here you can start to see what I mean with: Automatisation. We are "replacing" your fingers (the human) for a scripts (a robot) that can do exactly the same:
     
    Changed:
    <
    <
    %SYNTAX{ syntax="bash" }%
    >
    >
    %CODE{"bash" num="1"}%
     #!/bin/bash
    Changed:
    <
    <
    ## This is a comment
    >
    >
    ## Asking where you are: pwd
     
    Changed:
    <
    <
    echo "Hello World" %ENDSYNTAX%
    >
    >
    ## Listing the files in the current directory

    ls

    ## Listing the files in your home directory

    ls $HOME

     
    Added:
    >
    >
    ## Listing the files in the temporary directory
     
    Added:
    >
    >
    ls /tmp/

    ## End %ENDCODE%


    Where as you can see, the lines 4,8,12,16 and the only lines that perform an action in this script.

     

    Linux (UNIX) command refreshing review

    Added:
    >
    >


    <!-- SyntaxHighlightingPlugin -->
    >  command n
    <!-- end SyntaxHighlightingPlugin -->


     
    <!-- SyntaxHighlightingPlugin -->
    Scripts
    <!-- end SyntaxHighlightingPlugin -->
    Added:
    >
    >

     
    Added:
    >
    >

    %SYNTAX{ syntax="zsh" }% ## to include a black space into a command line, use an "\" and add the needed space, for example: > echo Ciao Mondo
    Ciao Mondo
     
    Changed:
    <
    <
    %SYNTAX{ syntax="bash" }% #!/bin/bash
    >
    >
    > echo Ciao\ Mundo
    Ciao Mondo
     
    Changed:
    <
    <
    ## This is a comment
    >
    >
    > echo Ciao Mondo
    Ciao Mondo

    > echo Ciao\ \ \ \ \ Mondo
    Ciao Mondo

     
    Deleted:
    <
    <
    echo "Hello World"
     %ENDSYNTAX%
    Added:
    >
    >

     
    Added:
    >
    >


     
    Changed:
    <
    <
    %SYNTAX{ syntax="bash" }% Scripts
    >
    >

    %SYNTAX{ syntax="zsh" }% > command n
     %ENDSYNTAX%
    Added:
    >
    >




    <!-- SyntaxHighlightingPlugin -->
    >  command n
    <!-- end SyntaxHighlightingPlugin -->


     
    Line: 217 to 290
     
    An example of the creation of a generic email template for users into an (computer) User Interface:
    Changed:
    <
    <
    %SYNTAX{ syntax="bash" }%
    >
    >
    %CODE{"bash" num="1"}%
     #!/bin/bash ###################################################################################################
    Changed:
    <
    <
    ## Author: Arturo Sanchez ## ## Create: 07 May 2014 ## ## Modify: 07 May 2014 ##
    >
    >
    ## Author: Arturo Sanchez (arturos@cernNOSPAMPLEASE.ch) ## ## Created: 07th May 2014 ## ## Updated: 28th October 2015 ##
     ## ## ## How to use this script: ## ## > ./new_user.sh Arturo\ Sanchez arturos 1234 > email_new_users/email_arturos.txt ##
    Line: 299 to 373
     echo "Arturo" echo "------" echo ""
    Added:
    >
    >
    %ENDCODE%

    Where if you save this script in a file called: new_user.sh and execute the following line:
    %SYNTAX{ syntax="zsh" }% > source new_user.sh Jon\ Doe jondoe 12345

     %ENDSYNTAX%
    Added:
    >
    >

     
    Changed:
    <
    <
    %SYNTAX{ syntax="bash" }% #!/bin/bash
    >
    >
    ...it will give you as a result the printing below ( printing that you can redirect into a text file for bookkeeping and posterior use, like to really send the email smile )
    %SYNTAX{ syntax="tex" }% Bongiorno,
     
    Changed:
    <
    <
    ## This is a comment
    >
    >
    The account for the user: Jon Doe into the ATLAS Napoli User Interface have been created.
     
    Deleted:
    <
    <
    echo "Hello World" %ENDSYNTAX%
     
    Added:
    >
    >
    Your username is: jondoe Your password is: 12345

    You can access using the ssh connection from a terminal:

    > ssh -YX -p5022 jondoe@atlasuiNOSPAMPLEASE.na.infn.it

    Or for a specific UI use the number 01, 02 or 03 as below:

    > ssh -YX -p5022 jondoe@atlasui03NOSPAMPLEASE.na.infn.it

    As soon you enter the password, you will get something as below:

    [jondoe@atlasui03 ~]$

    Where you can check where you are:

    [jondoe@atlasui03 ~]$ pwd /home/jondoe

    Please, change your password as soon as possible, using the command: > yppasswd

    You will get something like (to introduce your current pass and the new one):

    [jondoe@atlasui03 ~]$ yppasswd Changing password for user jondoe. Changing password for jondoe. (current) UNIX password: New password:

    The UI has space dedicates to big files that the users could need to store there. The path of such a space is:

    /data/

    So, in your case is:

    /data/jondoe

    Your /home/jondoe folder has a quota of 10GB.

    If you have an estimation of the possible space that you will need in /data/ and/or /home/, we will appreciate that information in order to setup your quota according to your work necessities.

    In case of any question/suggestion please let me know,

    Cheers, Arturo


    %ENDSYNTAX%
    Where the Name, Last Name, nickname and password of the new user were written in the correct lines and all the email now has a useful meaning.
    This scripts use two fundamental concepts for scripts that we use in a day-by-day basis: inputs and variables. Those and other concepts are review later in this tutorial but we can already review them here for completeness:
     %SYNTAX{ syntax="bash" }% #!/bin/bash
    Line: 318 to 465
     %ENDSYNTAX%
    Changed:
    <
    <

    HTML fast view

    >
    >

    HTML fast view

      %SYNTAX{ syntax="html" }%
    Changed:
    <
    <
    Scripts
    >
    >
    Old HTML
     %ENDSYNTAX%
    Line: 332 to 479
     

    %SYNTAX{ syntax="html" }%

    Changed:
    <
    <
    Scripts
    >
    >
    New HTML
     %ENDSYNTAX%
    Line: 410 to 557
     %ENDSYNTAX%

    Loops

    Deleted:
    <
    <
    %SYNTAX{ syntax="bash" }% #!/bin/bash
     
    Changed:
    <
    <
    ## This is a comment
    >
    >
    <!-- SyntaxHighlightingPlugin -->
        1#!/bin/sh
        2languages=`enscript --help-highlight | grep 'Name:' | cut -d ' ' -f 2`
        3for l in $languages; do
        4    cat << EOF
        5   * $l
        6EOF
        7done
    <!-- end SyntaxHighlightingPlugin -->
     
    Added:
    >
    >
    %SYNTAX{ syntax="bash" }%#!/bin/bash ## This is a comment
     echo "Hello World" %ENDSYNTAX%

    Revision 92015-10-27 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 46 to 46
     
    Changed:
    <
    <

    Introduction (NEW October 27th, 2015)

    >
    >
    <!--/twistyPlugin twikiMakeVisibleInline-->

    Introduction

     

    Automatisation

    Line: 108 to 116
     
    Added:
    >
    >
     

    An pictorial view of scripts in an analysis

    Added:
    >
    >

    Grphical_view_scripts.png

     

    Scripting Languages and the OS as environment

    Scripts are not restricted to a single computer language. In fact we will see that we can write scripts in several languages (and for several OS) as soon as we can reach the tools we need.
    At this point, we will focus the attention in bash and python scripts since their use in Linux OS is very well known and in particular for HEP-ATLAS analysis, they are supported by a large community. The tools that we use in ATLAS for management of data samples, coding, sharing and final publications production pass through C++ and other several open source applications that have command lines interface support.
    Added:
    >
    >



    A list of useful (but far to be completed) of kernels for scripts:

    Language/Kernel Covered here? Use in HEP?
    bash yes yes
    Python yes yes
    JavaScript not yes
    C++ a bit yes
    PowerShell not not?
    Google Script not not?
    ... ... ...


     
    Deleted:
    <
    <
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash  
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->
     

    Q&A

    Added:
    >
    >

    smile ?? Do you have any question so far?
    For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

    ( please remove SPAMNOT from the email address wink )



     

    Pre-Review of bash scripts

    Line: 203 to 245
     echo "> ssh -YX -p5022 $2@atlasui.na.infn.it" echo "" echo ""
    Changed:
    <
    <
    echo "Or for a specific UI use the number 01, 02 or 03 as bellow:"
    >
    >
    echo "Or for a specific UI use the number 01, 02 or 03 as below:"
     echo "" echo "> ssh -YX -p5022 $2@atlasui03.na.infn.it" echo "" echo "" echo ""
    Changed:
    <
    <
    echo "As soon you enter the password, you will get something as bellow:"
    >
    >
    echo "As soon you enter the password, you will get something as below:"
     echo "" echo "[$2@atlasui03 ~]$" echo ""
    Line: 296 to 338
     

    Q&A

    Changed:
    <
    <
    ?? smile
    >
    >

    smile Do you have any question so far?
    For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

    ( please remove SPAMNOT from the email address wink )



    <!-- ********************************************************* -->
    <!--/twistyPlugin-->

     


    Line: 320 to 379
     
    Added:
    >
    >
    <!--/twistyPlugin twikiMakeVisibleInline-->
     

    Review of concepts with examples:

    Script concepts using bash

    Line: 396 to 463
     echo "Hello World" %ENDSYNTAX%
    Added:
    >
    >
     

    Q&A

    Changed:
    <
    <
    ?? smile
    >
    >

    smile Do you have any question so far?
    For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

    ( please remove SPAMNOT from the email address wink )



     

    ROOT Reminder

    Line: 407 to 488
     

    Example macros for script proposes

    Q&A

    Changed:
    <
    <
    >
    >

    smile Do you have any question so far?
    For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

    ( please remove SPAMNOT from the email address wink )



     

    First set of examples in Python Vs BASH

    Line: 429 to 523
     

    One to One equivalent examples for generic use

    Online resources and references

    Added:
    >
    >
     

    Q&A

    Added:
    >
    >

    smile Do you have any question so far?
    For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

    ( please remove SPAMNOT from the email address wink )



    <!-- ********************************************************* -->
    <!--/twistyPlugin-->

     


    Line: 452 to 566
     

    Added:
    >
    >
    <!--/twistyPlugin twikiMakeVisibleInline-->
     

    Introduction to ATLAS Ecosystem and Tools

    Line: 504 to 625
     

    Services

    Q&A

    Added:
    >
    >

    smile Do you have any question so far?
    For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

    ( please remove SPAMNOT from the email address wink )



     

    The LXPLUS <--->

    ATLASUI integration
    Line: 519 to 654
     

    Online resources and references

    Q&A

    Added:
    >
    >

    smile Do you have any question so far?
    For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

    ( please remove SPAMNOT from the email address wink )



     
    Changed:
    <
    <

    Neapolitan Tools

    >
    >

    Local developed Tools

     

    Existing scripts

    Line: 545 to 694
      (review and test of more complex scripts for automatisation of tasks)

    Q&A

    Added:
    >
    >

    smile Do you have any question so far?
    For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL

    ( please remove SPAMNOT from the email address wink )



     
    Changed:
    <
    <

    >
    >
    <!-- ********************************************************* -->
    <!--/twistyPlugin-->

     
    Line: 571 to 735
     

    META FILEATTACHMENT attachment="Screenshot_2015-10-21_17.34.46.png" attr="h" comment="" date="1445892756" name="Screenshot_2015-10-21_17.34.46.png" path="Screenshot_2015-10-21_17.34.46.png" size="88312" user="arturos" version="1"
    Added:
    >
    >
    META FILEATTACHMENT attachment="Grphical_view_scripts.png" attr="h" comment="graphical view script working" date="1445970045" name="Grphical_view_scripts.png" path="Grphical_view_scripts.png" size="163482" user="arturos" version="1"

    Revision 82015-10-27 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 170 to 170
     

    Some scripts in action

    Added:
    >
    >

    Generic automatic Email for Account "management"


    An example of the creation of a generic email template for users into an (computer) User Interface:
     %SYNTAX{ syntax="bash" }% #!/bin/bash
    Changed:
    <
    <
    ## This is a comment

    echo "Hello World"

    >
    >
    ################################################################################################### ## Author: Arturo Sanchez ## ## Create: 07 May 2014 ## ## Modify: 07 May 2014 ## ## ## ## How to use this script: ## ## > ./new_user.sh Arturo\ Sanchez arturos 1234 > email_new_users/email_arturos.txt ## ## Example: ## ## > ./new_user.sh <Name\ LastName> > email_new_users/email_.txt ## ###################################################################################################

    echo "Bongiorno," echo "" echo "The account for the user: $1 into the ATLAS Napoli User Interface have been created." echo "" echo "" echo "Your username is: $2" echo "Your password is: $3" echo "" echo "" echo "You can access using the ssh connection from a terminal:" echo "" echo "> ssh -YX -p5022 $2@atlasui.na.infn.it" echo "" echo "" echo "Or for a specific UI use the number 01, 02 or 03 as bellow:" echo "" echo "> ssh -YX -p5022 $2@atlasui03.na.infn.it" echo "" echo "" echo "" echo "As soon you enter the password, you will get something as bellow:" echo "" echo "[$2@atlasui03 ~]$" echo "" echo "" echo "Where you can check where you are:" echo "" echo "[$2@atlasui03 ~]$ pwd" echo "/home/$2" echo "" echo "" echo "Please, change your password as soon as possible, using the command:" echo "> yppasswd" echo "" ## echo "> passwd" ## echo "IMPORTANT! (Please, use the atlasui03 to do this procedure)" ## echo "" echo "You will get something like (to introduce your current pass and the new one):" echo "" ## echo "[$2@atlasui03 ~]$ passwd" echo "[$2@atlasui03 ~]$ yppasswd" echo "Changing password for user $2." echo "Changing password for $2." echo "(current) UNIX password:" echo "New password: " echo "" echo "" echo "" echo "The UI has space dedicates to big files that the users could need to store there." echo "The path of such a space is: " echo "" echo "/data/" echo "" echo "So, in your case is:" echo "" echo "/data/$2" echo "" echo "" echo "Your /home/$2 folder has a quota of 10GB." echo "" echo "If you have an estimation of the possible space that you will need in /data/ and/or /home/, we will appreciate that information in order to setup your quota according to your work necessities." echo "" echo "In case of any question/suggestion please let me know," echo "" echo "Cheers," echo "Arturo" echo "------" echo ""

     %ENDSYNTAX%

    %SYNTAX{ syntax="bash" }%

    Revision 72015-10-27 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Revision 62015-10-27 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 50 to 50
     

    Automatisation

    Added:
    >
    >

     
    Line: 57 to 58
     
    Added:
    >
    >


     

    What is a script?

    Added:
    >
    >

     
    Changed:
    <
    <
    Is the translation to a particular language the set of instructions result of the automatisation algorithm's development
    >
    >
    Is the translation to a particular language the set of instructions result of the automatisation algorithm's development for what you need/want this automatic execution(s),
     
    Added:
    >
    >

    <!-- SyntaxHighlightingPlugin -->
    ## !Hi, this is a set of commands into a script:  
    
    setup some variables
    
    ## This is a comment
    
    source a set of other scripts
    
    ## This is another comment
    
    set your input and output files and locations
    
    ## Finally
    
    execute your main program of set of applications
    
    ## Done
    
    <!-- end SyntaxHighlightingPlugin -->


     

    Why to use scripts?

    Added:
    >
    >

    Scripts are used to put to work the computers for you. They are a set of instructions that allows to perform repetitive operations in an automatic way, so, just your imagination can define the limits of an script. Starting for simple checks to your current working area, passing for setup of simple of complex environments, or for backup creations to interactions with web-based (internet) resources, to integration of databases, online monitoring of resources to... malware... the reasons to have scripts is basically to get something done as far and human-independent as possible. And as we will see, we can wrap several different applications and tools into a single set of instructions, so, it is like to mix many different software into a single box:

     

    An pictorial view of scripts in an analysis

    Scripting Languages and the OS as environment

    Changed:
    <
    <
    >
    >
    Scripts are not restricted to a single computer language. In fact we will see that we can write scripts in several languages (and for several OS) as soon as we can reach the tools we need.
    At this point, we will focus the attention in bash and python scripts since their use in Linux OS is very well known and in particular for HEP-ATLAS analysis, they are supported by a large community. The tools that we use in ATLAS for management of data samples, coding, sharing and final publications production pass through C++ and other several open source applications that have command lines interface support.
      %SYNTAX{ syntax="bash" }% #!/bin/bash
    Line: 291 to 331
     

    First set of examples in Python Vs BASH

    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    #!/usr/bin/python
    
    import os, math, sys
    
    #############################
    #
    # Script for 
    #
    #############################
    
    <!-- end SyntaxHighlightingPlugin -->

     

    One to One equivalent examples for generic use

    Online resources and references

    Revision 52015-10-27 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Line: 50 to 50
     

    Automatisation

    Added:
    >
    >
      Picking into the large amount of different concepts with can find, an useful definition is: The use of tools, techniques and ingenuous in other to create algorithms that allow you to request to the computer to perform a serie of complex tasks, in a sequential mode, with a deterministic frequency in an automatic way with a single execution o command.
    Added:
    >
    >
     

    What is a script?

    Added:
    >
    >
    Is the translation to a particular language the set of instructions result of the automatisation algorithm's development
     
    Deleted:
    <
    <
    Is the translation to a particular language the set of instructions that the automatisation algorithm that was developed during s
     

    Why to use scripts?

    Line: 64 to 76
     

    %SYNTAX{ syntax="bash" }%

    Changed:
    <
    <
    Scripts
    >
    >
    #!/bin/bash

    ## This is a comment

    echo "Hello World"

     %ENDSYNTAX%

    Q&A

    Pre-Review of bash scripts

    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->

    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->

     

    Linux (UNIX) command refreshing review

    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    Scripts
    <!-- end SyntaxHighlightingPlugin -->

    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->

    <!-- SyntaxHighlightingPlugin -->
    Scripts
    <!-- end SyntaxHighlightingPlugin -->

     

    Some scripts in action

    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->

    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->

    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->
     

    HTML fast view

    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    Scripts
    <!-- end SyntaxHighlightingPlugin -->

    <!-- SyntaxHighlightingPlugin -->
    Scripts
    <!-- end SyntaxHighlightingPlugin -->

    <!-- SyntaxHighlightingPlugin -->
    Scripts
    <!-- end SyntaxHighlightingPlugin -->

     

    Q&A

    Added:
    >
    >
    ?? smile
     


    Line: 106 to 204
     

    Script concepts using bash

    “Hello” World

    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->
     

    Variables

    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->
     

    Loops

    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->
     

    Conditionals

    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->
     

    Pipes

    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->
     

    Inputs and Outputs

    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->
     

    Functions

    Added:
    >
    >
    <!-- SyntaxHighlightingPlugin -->
    #!/bin/bash          
    
    ## This is a comment
    
    echo "Hello World"
    <!-- end SyntaxHighlightingPlugin -->
     

    Working with text files

    Added:
    >
    >
    %SYNTAX{ syntax="bash" }% #!/bin/bash
     
    Changed:
    <
    <

    Q&A

    >
    >
    ## This is a comment
     
    Added:
    >
    >
    echo "Hello World" %ENDSYNTAX%

    Q&A

    ?? smile
     

    ROOT Reminder

    Revision 42015-10-27 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    ScriptingTutorial

    Revision 32015-10-26 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"
    Changed:
    <
    <

    Introduction

    >
    >

    ScriptingTutorial

     
    Deleted:
    <
    <
    <!-- Add an introduction here, describing the purpose of this topic. -->
     
    Changed:
    <
    <

    Day 1:
    >
    >
     
    Changed:
    <
    <
    1.a) Introduction - Automatisation Picking into the large amount of different concepts with can find, an useful definition is: The use of tools, techniques and ingenuous in other to create algoritms that allow you to request to the computer to perform a serie of complex tasks -in a sequential mode- with a deterministic frequency in an automatic way with a single execution o command.

    - What is a script?

    Is the translation to a particular language the set of instructions that the automatization algorithm that was developed during s

    - Why to use scripts? - An pictorial view of scripts in an analysis - Scripting Languages and the OS as environment - Q&A

    1.b) Pre-review of bash scripts - Linux (UNIX) command refreshing review - Some scripts in action from the twiki - HTML fast view - Q&A


    Day 2:

    2.a) Review of concepts with examples: (Mainly BASH) -“Hello” World - Variables - Loops - Conditionals - Pipes - Inputs and Outputs - Functions - Working with text files - Q&A

    2.b) ROOT Reminder - Review of concepts - Example macros for script proposes - Q&A

    2.c) First set of examples in Python Vs BASH - One to One equivalent examples for generic use - Online resources and references - Q&A


    Day 3:

    3.a) Introduction to ATLAS Ecosystem and Tools

    - The User Interfaces (lxplus and atlasui) - The GRID - What is a Job? - What I need? - Code - Samples - PROXY - PRUN

    >
    >
    The objective of this page is to deliver a tutorial in scripting with focus in analysis of LHC data into the ATLAS Collaboration.

    ATLAS computing/analysis Book!!

    ATLAS_Script_Setup

    ATLASUI integration
  • Second set of examples in Python Vs BASH
  • Local developed Tools
  • Practical Exercises with each the mentioned Tools
  • Exercises Day 3
  • <!-- this line is optional -->

    <!-- ---------------------------------------------------------------------------------- -->
    <!-- ---------------------------------------------------------------------------------- -->






    Day 1





    Introduction (NEW October 27th, 2015)

    Automatisation

    Picking into the large amount of different concepts with can find, an useful definition is: The use of tools, techniques and ingenuous in other to create algorithms that allow you to request to the computer to perform a serie of complex tasks, in a sequential mode, with a deterministic frequency in an automatic way with a single execution o command.

    What is a script?

    Is the translation to a particular language the set of instructions that the automatisation algorithm that was developed during s

    Why to use scripts?

    An pictorial view of scripts in an analysis

    Scripting Languages and the OS as environment

    <!-- SyntaxHighlightingPlugin -->
    Scripts
    <!-- end SyntaxHighlightingPlugin -->

    Q&A

    Pre-Review of bash scripts

    Linux (UNIX) command refreshing review

    Some scripts in action

    HTML fast view

    Q&A






    Day 2





    Review of concepts with examples:

    Script concepts using bash

    “Hello” World

    Variables

    Loops

    Conditionals

    Pipes

    Inputs and Outputs

    Functions

    Working with text files

    Q&A

    ROOT Reminder

    Review of concepts

    Example macros for script proposes

    Q&A

    First set of examples in Python Vs BASH

    One to One equivalent examples for generic use

    Online resources and references

    Q&A






    Day 3





    Introduction to ATLAS Ecosystem and Tools

    The User Interfaces (lxplus and atlasui)

    The GRID

    What is a Job?

    What I need?

    Code

    Samples

    PROXY

    ATLAS practical Tools

    PRUN

      (Concept and use)
    Changed:
    <
    <
    - BigPanda
    >
    >

    BigPanda

      (Concept and use)
    Changed:
    <
    <
    - RUCIO ( dq2 ? depreciate but still useful )
    >
    >

    RUCIO ( dq2 ? depreciate but still useful )

      (Concept and use)
    Changed:
    <
    <
    - AMI
    >
    >

    AMI

      (Concept and use)
    Changed:
    <
    <
    - PBOOK
    >
    >

    PBOOK

      (Concept and use)
    Deleted:
    <
    <
    - CERNBOX - JIRA - SVN (Vs) GitLab - Latest ATLAS “Style” guidelines ….

    - The ATLASUI - What is inside - Properties - Services - Q&A

     
    Changed:
    <
    <
    3.b) The LXPLUS <--->
    ATLASUI integration
    >
    >

    CERNBOX

    JIRA

    SVN (Vs) GitLab

    Latest ATLAS “Style” guidelines

    The ATLASUI

    What is inside

    Properties

    Services

    Q&A

    The LXPLUS <--->

    ATLASUI integration
      (Examples and case of use)
    Changed:
    <
    <
    3.c) Second set of examples in Python Vs BASH
    >
    >

    Second set of examples in Python Vs BASH

      (Examples of the use in scripts for analysers are included in each point)
    Deleted:
    <
    <
    - One to One equivalent examples for generic use - Online resources and references - Q&A

    3.d) Napoli’s Tools - Existing scripts - Websites and Online - Tier2

     
    Changed:
    <
    <
    3.e) Practical Exercises with each the mentioned Tools (BASH and Python)
    >
    >

    One to One equivalent examples for generic use

     
    Changed:
    <
    <
    - Initial Examples (review concepts)
    >
    >

    Online resources and references

     
    Changed:
    <
    <
    - Intermediate Examples (combinations of two or more tools using pipes)
    >
    >

    Q&A

     
    Deleted:
    <
    <
    - Advance Scenarios and Scripts in ATLAS analysis (review and test of more complex scripts for automatisation of tasks)
     
    Changed:
    <
    <
    - Q&A
    >
    >

    Neapolitan Tools

     
    Changed:
    <
    <


    >
    >

    Existing scripts

     
    Changed:
    <
    <
    Notes:
    >
    >

    Websites and Online

     
    Changed:
    <
    <
    - In each of the sessions we can have time for Q&A - It will be included reference to external links and ATLAS resources and not just at the end. - The TWIKI will be used to include “cut and paste” all the examples in the tutorial https://twiki.cern.ch/twiki/bin/view/AtlasPublic/ScriptingTutorial
    >
    >

    Tier2

     
    Deleted:
    <
    <
    - Practical exercises:
     
    Deleted:
    <
    <
    - I would like to teach them several very useful linux command to use directly in the terminal and later how they are combine like LEGOS to create an algorithm to search, sort, mix, compare files, directories, list of samples,… Very real examples for an analyser.
     
    Changed:
    <
    <
    - Details about ATLAS setups a nd other more super simple but important like the use of absolute paths and options in their scripts. "nohup" and "screen" - At the same time I want to use all the files-database with have in Naples (that I use for my websites) to look for samples and information like their status, x-sections.
    >
    >

    Practical Exercises with each the mentioned Tools

    (BASH and Python)
     
    Deleted:
    <
    <
    - Introduce of course RUCIO for search, download data-sets official and private production, book to recover, DQ2, pyAMI examples as well.
     
    Changed:
    <
    <
    - For the “analysis” examples I will use ROOT. again, the idea is to read root files, trying to move them, sort them, etc.. using scripts.
    >
    >

    Initial Examples

    (review concepts)
     
    Changed:
    <
    <
    - I go to prepare some examples of grid jobs, not using frameworks, but ROOT macros running on grid. but we can include some real jobs as well.
    >
    >

    Intermediate Examples

    (combinations of two or more tools using pipes)
     
    Changed:
    <
    <
    - Sure smile they are starting, and I think this is why I want to go in useful details and prepare the TWIKI so they can come back there to get refresh of help when they start to work in their analysis.
    >
    >

    Advance Scenarios and Scripts in ATLAS analysis

    (review and test of more complex scripts for automatisation of tasks)

    Q&A

     
    Deleted:
    <
    <
    - About the use of a certificate for them, we can have the alternative solution to let them login in a generic account in ATLASUI, with my certificate installed there and the proxy done (I have an script that keep the proxy alive…) so, they can tests their scripts and using a tail in the name of the job to identify each of them I think it will be fine. At the same time this will allow us to monitor the jobs and to tests book commands to retry, kill jobs.
     
    Line: 164 to 280
     
    <!-- Once this page has been reviewed, please add the name and the date e.g. StephenHaywood - 31 Oct 2006 -->

    \ No newline at end of file

    Added:
    >
    >
    META FILEATTACHMENT attachment="Screenshot_2015-10-21_17.34.46.png" attr="h" comment="" date="1445892756" name="Screenshot_2015-10-21_17.34.46.png" path="Screenshot_2015-10-21_17.34.46.png" size="88312" user="arturos" version="1"

    Revision 22015-10-26 - ArturoS

    Line: 1 to 1
     
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"
    Deleted:
    <
    <
     

    Introduction

    <!-- Add an introduction here, describing the purpose of this topic. -->
    Line: 5 to 4
     
    <!-- Add an introduction here, describing the purpose of this topic. -->
    Changed:
    <
    <
    To be modified soon...
    >
    >

    Day 1:

    1.a) Introduction - Automatisation Picking into the large amount of different concepts with can find, an useful definition is: The use of tools, techniques and ingenuous in other to create algoritms that allow you to request to the computer to perform a serie of complex tasks -in a sequential mode- with a deterministic frequency in an automatic way with a single execution o command.

    - What is a script?

    Is the translation to a particular language the set of instructions that the automatization algorithm that was developed during s

    - Why to use scripts? - An pictorial view of scripts in an analysis - Scripting Languages and the OS as environment - Q&A

    1.b) Pre-review of bash scripts - Linux (UNIX) command refreshing review - Some scripts in action from the twiki - HTML fast view - Q&A


    Day 2:

    2.a) Review of concepts with examples: (Mainly BASH) -“Hello” World - Variables - Loops - Conditionals - Pipes - Inputs and Outputs - Functions - Working with text files - Q&A

    2.b) ROOT Reminder - Review of concepts - Example macros for script proposes - Q&A

    2.c) First set of examples in Python Vs BASH - One to One equivalent examples for generic use - Online resources and references - Q&A


    Day 3:

    3.a) Introduction to ATLAS Ecosystem and Tools

    - The User Interfaces (lxplus and atlasui) - The GRID - What is a Job? - What I need? - Code - Samples - PROXY - PRUN (Concept and use) - BigPanda (Concept and use) - RUCIO ( dq2 ? depreciate but still useful ) (Concept and use) - AMI (Concept and use) - PBOOK (Concept and use) - CERNBOX - JIRA - SVN (Vs) GitLab - Latest ATLAS “Style” guidelines ….

    - The ATLASUI - What is inside - Properties - Services - Q&A

    3.b) The LXPLUS <--->

    ATLASUI integration (Examples and case of use)

    3.c) Second set of examples in Python Vs BASH (Examples of the use in scripts for analysers are included in each point) - One to One equivalent examples for generic use - Online resources and references - Q&A

    3.d) Napoli’s Tools - Existing scripts - Websites and Online - Tier2

    3.e) Practical Exercises with each the mentioned Tools (BASH and Python)

    - Initial Examples (review concepts)

    - Intermediate Examples (combinations of two or more tools using pipes)

    - Advance Scenarios and Scripts in ATLAS analysis (review and test of more complex scripts for automatisation of tasks)

    - Q&A



    Notes:

    - In each of the sessions we can have time for Q&A - It will be included reference to external links and ATLAS resources and not just at the end. - The TWIKI will be used to include “cut and paste” all the examples in the tutorial https://twiki.cern.ch/twiki/bin/view/AtlasPublic/ScriptingTutorial

    - Practical exercises:

    - I would like to teach them several very useful linux command to use directly in the terminal and later how they are combine like LEGOS to create an algorithm to search, sort, mix, compare files, directories, list of samples,… Very real examples for an analyser.

    - Details about ATLAS setups a nd other more super simple but important like the use of absolute paths and options in their scripts. "nohup" and "screen" - At the same time I want to use all the files-database with have in Naples (that I use for my websites) to look for samples and information like their status, x-sections.

    - Introduce of course RUCIO for search, download data-sets official and private production, book to recover, DQ2, pyAMI examples as well.

    - For the “analysis” examples I will use ROOT. again, the idea is to read root files, trying to move them, sort them, etc.. using scripts.

    - I go to prepare some examples of grid jobs, not using frameworks, but ROOT macros running on grid. but we can include some real jobs as well.

    - Sure smile they are starting, and I think this is why I want to go in useful details and prepare the TWIKI so they can come back there to get refresh of help when they start to work in their analysis.

    - About the use of a certificate for them, we can have the alternative solution to let them login in a generic account in ATLASUI, with my certificate installed there and the proxy done (I have an script that keep the proxy alive…) so, they can tests their scripts and using a tail in the name of the job to identify each of them I think it will be fine. At the same time this will allow us to monitor the jobs and to tests book commands to retry, kill jobs.


     

    Revision 12015-10-21 - ArturoS

    Line: 1 to 1
    Added:
    >
    >
    META TOPICPARENT name="https://twiki/cern/ch/twiki/bin/view/AtlasPublic.WebAnalysis"

    Introduction

    <!-- Add an introduction here, describing the purpose of this topic. -->

    To be modified soon...

    <!-- ********************************************************* -->
    <!-- Do NOT remove the remaining lines, but add requested info as appropriate-->
    <!-- ********************************************************* -->


    <!-- For significant updates to the topic, consider adding your 'signature' (beneath this editing box) -->
    Major updates:
    -- ArturoS - 2015-10-21

    <!-- Person responsible for the page: 
    Either leave as is - the creator's name will be inserted; 
    Or replace the complete REVINFO tag (including percentages symbols) with a name in the form TwikiUsersName -->
    Responsible: ArturoS
    Subject: public
    <!-- Once this page has been reviewed, please add the name and the date e.g. StephenHaywood - 31 Oct 2006 -->
     
    This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
    Ideas, requests, problems regarding TWiki? Send feedback