Linux notes and commands

Redhat

Linux command library How to Use Lxplus at CERN Linux installation at CERN

shell script
slides

#!/bin/bash
# My first script

echo "Hello World!"
The first line of the script is important. This is a special clue, called a shebang, given to the shell indicating what program is used to interpret the script. In this case, it is /bin/bash. Other scripting languages such as Perl, awk, tcl, Tk, and python also use this mechanism.

The second line is a comment. Everything that appears after a "#" symbol is ignored by bash. As your scripts become bigger and more complicated, comments become vital. They are used by programmers to explain what is going on so that others can figure it out. The last line is the echo command. This command simply prints its arguments on the display.

date
https://stackoverflow.com/questions/20361982/get-current-time-in-hours-and-minutes
date +%H:%M gives: 14:22
date; gives: Mon Oct 22 14:23:38 CEST 2018

cat command
merge hints

Append the contents of file2.txt to the end of file1.txt. 
The content of file3.txt is appended to the end of merged contents of file1.txt and file2.txt and so on.
The entire merged file is saved with the name mergedfile.txt in the current working directory:

bash$ cat file1.txt file2.txt file3.txt file4.txt > ./mergedfile.txt

The cat command accepts regular expressions as input file names, 
which means you can use them to reduce the number of arguments:

bash$ cat file*.txt my*.txt > mergedfile.txt
This will merge all the files in the current directory that start with the name file 
and has a txt extension followed by the files that start with my and has a txt 
extension. You have to be careful about using regular expressions, if you want 
to preserve the order of files. If you get the regular expression wrong, it will affect 
the exact order in which the files are merged.


A quick and easy way to make sure the files get merged in the exact order you want
is to use the output of another file listing program such as ls or find and pipe it to the 
cat command. First execute the find command with the regular expression and verify 
the file order…

bash$ find . -name "file*.txt" -o -name "my*.txt"
This will print the files in order such that you can verify it to be correct or modify it 
to match what you want. You can then pipe that output into the cat command.

bash$ find . -name "file*.txt" -o -name "my*.txt" | xargs cat > ./mergedfile.txt
When you merge multiple files into one file using regular expressions to match them, 
especially when it is piped and where the output file is not very obvious, make sure that 
the regular expression does not match the filename of the merged file. In the case 
that it does match, usually the cat command is pretty good at error-ing out with the 
message “input file is output file”. But it helps to be careful to start with.

RAL linux

  • linux.pp.rl.ac.uk
  • same usrn as at CERN

ssh -Y davec@linux.pp.rl.ac.uk
firefox &

Linux beginners guide Linux Support Page Printing from Linux ssh faqs

Availabe printers

  • type> /usr/sbin/lpadmincern --list --building XX
  • alias lprpretty 'lpr -P 40-1B-HPCOR -oprettyprint'
  • lprpretty fn to print

Using the convert command for animated gifs etc

  • link, jpg to video
  • convert -delay 20 -loop 0 evt*.gif animatedevts.gif
  • NOTE: does not work on .png files to one animated png - need to convert to gif - can be done on the same command line:
convert -delay 10 -loop 0 *.png animation.gif

importing animated gifs into pdf
link

linux command.org 50 most used commands List of Unix programmes

cd tricks

grep More grep xargs Unix and Linux shortcut keys

find

PATH, CMSSW PATH info

Linux updates

System->Administration->Software update
  • May get errors - try deselting item, or all items (rt click - drop down)
  • cat /etc/redhat-release for release version
  • /usr/bin/yum update ---skip-broken to try to avoid broken updates

Browsing files on host machine

Access file browser windows in the following ways:
  • Click on the file browser window launcher on the top edge panel
  • Choose Applications -> Browse Filesystem
  • Select a folder, then choose File-> Browse Folder

SCP

Example syntax for Secure Copy (scp)

Copy the file "foobar.txt" from a remote host to the local host
$ scp your_username@remotehost.edu:foobar.txt /some/local/directory

Copy the file "foobar.txt" from the local host to a remote host
$ scp foobar.txt your_username@remotehost.edu:/some/remote/directory

WinSCP

Trouble with WinSCP from home - big block SFTP transfer error.
Remove print statement (echo ROOTSYS = $ROOTSYS) from .tcshrc. All OK now.
Put all print statements into .login file !!!!!!!!!!

Combine files

cat file2 >> file1
The >> operator appends the output to the named file or creates the named file if it does not exist.

cat file1 file2 > file3
This concatenates two or more files to one. You can have as many source files as you need. For example,

cat *.txt >> newfile.txt

Linux commands

Command Action
./ . denotes the current directory. Since you want to run a file in your current directory and that directory is not in your $PATH, you need the ./ bit to tell the shell where the executable is. So, ./foo means run the executable called foo that is in this directory.
source source is a bash shell built-in command that executes the content of the file passed as argument, in the current shell. It has a synonym in '.' (period).
  source filename [arguments], or writing with its synonym, '.' (period) :
  . filename [arguments]
  ie source .tcshrc to make changes come into effect during the current session
   
sh files do chmod 700 fn.sh to make executable
  do $bash ~davec/fn.sh to execute
   
g++ to compile with C++, ie g++ map-test.C will generate an executable, by default a.out
  g++ map-test.C -o map.out
  g++ filename -o outputfile
  g++ filename -Wall -ansi -o outputfile
  g++ main.cpp f1.cpp f2.cpp f3.cpp for multiple files, keep header files in same folder or explicitly address
  In order to compile with all warnings enabled and to produce standards-compatible C++ code
  If you want to have the compiler treat warnings as errors--meaning you don't even get an executable, you can use the -Werror flag. This will make sure you don't miss an error.
   
  Do ./a.out to run
  Also see g++
   
file manipulate file names and attributes
display ie display fn.png will show the image on any workstation running X
gv ie gv file.pdf to display a pdf or eps file
 
cp -R copy folders with files/folders
  cp -R /afs/cern.ch/user/d/davec/CMSSW_8_0_8/src/Reco/ /afs/cern.ch/user/d/davec/CMSSW_8_0_17/src
 
  many associated display functions
 
command line tcsh examples
  command-line-format.txt
  set prompt="$HOST "
  note: leaves space to look good
  set prompt = "-- %T %n %~ -- \n$ " gives:
  -- 23:16 davec ~/my/tmp --
  $
\033 \033 commands etc
echo print to standard output
  echo CMSSW_BASE is $CMSSW_BASE
  CMSSW_BASE is /afs/cern.ch/user/d/davec/HZZ/CMSSW_5_3_9
du -h get size of directory, -h option to get output in human readable format i.e. show output in kilobytes (K), megabytes (M) and gigabytes (G)
du -sh get size of directory but does not list size of subdirectories
  du -h /dir1/file2
   
Output stream ofstream myfile;
  myfile.open ("example.txt");
  myfile << "Writing this to a file.\n";
  myfile.close();
   
Redirecting stdout ie: redirect printf output
  freopen ("myfile.txt","w",stdout);
  printf ("This sentence is redirected to a file.");
  fclose (stdout);
  NOTE: "w" creates an empty file for output operations.
  If a file with the same name already exists, its contents are discarded
  and the file is treated as a new empty file.
   
  freopen ("myfile.txt","a",stdout);
  "a" appends: it opens file for output at the end of a file.
  Output operations always write data at the end of the file, expanding it.
  The file is created if it does not exist.
   
setenv All variables set with setenv command are automatically exported to subshell.
set All csh variables set to with set command are NOT automatically exported to subshell.
$ the shell sees a word that begins with a "$", tries to find out what was assigned to the variable and substitutes it.
$(command) The characters "$( )" tell the shell, "substitute the results of the enclosed command."
  Don't put spaces on either side of the equal sign when assigning value to variable. For e.g. In following variable declaration there will be no error
  $ no=10
  But there will be problem for any of the following variable declaration:
  $ no =10
  $ no= 10
  $ no = 10
" Double Quotes "Double Quotes" - Anything enclose in double quotes removed meaning of that characters (except \ and $).
' Single quotes 'Single quotes' - Enclosed in single quotes remains unchanged.
` Back quote `Back quote` - To execute command, ie echo "Today is `date`".
  in bash: right_now = $(date +"%x %r %Z") in tsch: set right_now = `date`
  time_stamp="Updated on $right_now by $USER" in tcsh set time_stamp="Updated on $right_now by $USER"
linux info Linux commands page
shell Learning the shell
  Writing shell scripts
command line convention, UPPERCASE for constants and environment variables
  convention, lowercase names for variables
  you can always type the name of the shell to switch.(bash,csh,sh,tcsh)
  Linux shell scripts
  Working with commands
  How to intrduce arguments on the command line
sed stream editor - to alter file contents etc
  Intro, sed by Bruce Barnett!
  Unix Sed Working methodology
  Called as one execution cycle. Cycle continues till end of file/input is reached:
  Read a entire line from stdin/file. Removes any trailing newline. Places the line, in its pattern buffer.
  Modify the pattern buffer according to the supplied commands. Print the pattern buffer to stdout.
  sed -n '3'p thegeekstuff.txt. Prints the 3rd line in the file thegeekstuff.txt
  sed -n '31445,32889'p thegeekstuff.txt. Prints lines 31445 to 32889 from the file thegeekstuff.txt
  sed '2d' fn > fn2 to delete line 2 in file fn and make the new file fn2
  Other examples
lines in PuTTy change the lines stored in the session
lines
 command_with_lots_of_output | less 
screen screen
cd tricks alias ..="cd .." alias ..2="cd ../.." alias ..3="cd ../../.." alias ..4="cd ../../../.." alias ..5="cd ../../../../.."
cd ../../../.. navigate 4 directories up
cd - toggle between last two directories
ls --color directories are in blue, pictures in purple, and executables in green.
unzip unzip fn.zip
tar -xvwf fn.tar untar a tar file (there is no 'untar' command)
tar -xvwzf fn.tar.gz unzip and untar a tar.gz file
tar -xvwzf fn.tgz for tgz files
csh will execute the top shell command .tcshrc - but this will be a new shell area
ctrl-shift t for a new terminal in same directory !!!!!
rehash sometimes needed to get back CMSSW commands
<ctrl+r> to repeat a command
&!:str repeat command containing str
Scrolling Go into 'Edit' on Terminal window -> Profile Preferences -> Scrolling and set the number of scroll back lines
< file means open a file for reading and associate with STDIN.
<< token Means use the current input stream as STDIN for the program until token is seen. We will ignore this one until we get to scripting.
> file means open "file" for writing and truncate it and associate it with STDOUT.
> any previous "file" will be overwritten
>> file means open a file for writing and seek to the end and associate it with STDOUT. This is how you append to a file using a redirect.
>> file will append to "file"
n>&m means redirect FD n to the same places as FD m. Eg, 2>&1 means send STDERR to the same place that STDOUT is going to.
   
diff compare two files, diff file1 file2
   
  $ some-command >> /tmp/log.log 2>&1
find command ie combine, do echo $PATH to check all the areas
  first directory: /afs/cern.ch/user/d/davec/ANALYSIS/CMSSW_6_1_1/bin/slc5_amd64_gcc472
  find /afs/cern.ch/user/d/davec/ANALYSIS/CMSSW_6_1_1/bin/slc5_amd64_gcc472 -iname combine, to search all subsequent directories
  combine is in /afs/cern.ch/user/d/davec/ANALYSIS/CMSSW_6_1_1/bin/slc5_amd64_gcc472/combine
  listed as an executable as -rwxr-xr-x 1 davec zh 166K Mar 17 19:19 combine
  Likewise for cmsRun
  /afs/cern.ch/cms/slc5_amd64_gcc472/cms/cmssw/CMSSW_6_1_1/bin/slc5_amd64_gcc472/cmsRun, -rwxr-xr-x 1 cmsbuild zh 166K Feb 13 13:20 cmsRun
find+grep
 find -iname '*.*' -print0 | xargs -0 grep "ret" > ret.txt 
 
 find mydir/ -name '*.C' -print0 | xargs -0 grep "examplestring" 
  /afs/cern.ch/user/d/davec -name '*.csv' will search for .csv files in davec and subdirectories
 
 find /afs/cern.ch/user/d/davec//HEEP/CMSSW_5_2_3/src/Reco/RecoAnalyzer/ -exec grep -l "iConfig" '{}' \; 
 
 find /afs/cern.ch/user/d/davec/ -noleaf -name 'reco*.cc' -exec ls -l {} \; 
 
 find /afs/cern.ch/user/d/davec/HEEP/CMSSW_5_2_3/src/Reco/RecoAnalyzer -noleaf -name '*.cc' -exec grep 'iConfig' *.cc {} \; 
 
 find /afs/cern.ch/user/d/davec/HEEP/CMSSW_5_2_3/ -noleaf -exec grep -l 'ESHandle' {} > ESHandle.txt \; 
 
 find /afs/cern.ch/user/d/davec//HEEP/CMSSW_5_2_3/src/Reco/RecoAnalyzer/ -exec grep -l "iConfig\|parameter" '{}' \; 
grep if you want to use shell variables, you need double quotes, ie
 grep "$HOME" file 
searches file for the name of your home directory
 
 grep -r "sevLev" /afs/cern.ch/cms/slc5_amd64_gcc462/cms/cmssw/CMSSW_5_2_3/src/ 
  for a recursive search including all subdirectories
 
 ls |grep blah 
lists all files in the current directory whose names contain the string "blah"
  The canonical wildcard character is the dot "."
  grep b.g file will search going through all SINGLE characters between b and g, ie big, bag but not boogy
 
 grep '$HOME' 
file searches for the string $HOME
 
 grep "cat\|dog" file 
matches lines containing the word "cat" or the word "dog"
 
 grep "I am a \(cat\|dog\)" 
matches lines containing the string "I am a cat" or the string "I am a dog"
  To match a selection of characters, use [].
  [Hh]ello matches lines containing hello or Hello
  [0-3] is the same as [0123], [a-k] is the same as [abcdefghijk], [A-C] is the same as [ABC], [A-Ca-k] is the same as [ABCabcdefghijk]
  an expression consisting of a character followed by an escaped question mark, \?, matches one or zero instances of that character.
  bugg\?y matches all of the following: bugy , buggy but not bugggy
  An expression surrounded by "escaped" parentheses is treated by a single character.
  Fred\(eric\)\? Smith matches Fred Smith or Frederic Smith
Escaped character character preceded by a backslash.
  The preceding backslash does one of the following: (a) removes an implied special meaning from a character (b) adds special meaning to a "non-special" character
  The $ character matches the end of the line.
  The ^ character matches the beginning of the line.
  ie, grep "^From.*mscharmi" /var/spool/mail/davec will match only from the very start
  The [] may be used to search for non-matches. This is done by putting a carat ^ as the first character inside the square brackets.
  ? \ . [ ] ^ $ need to be escaped with \
  grep 'hello.gif' file would find hello-gif , hello1gif , helloagif , etc.
  To stop "." being a wildcard, precede with \ , ie grep 'hello\.gif' file
  $ sign loses its meaning if characters follow it (I think)
  the carat ^ loses its meaning if other characters precede it.
  Square brackets behave a little differently. The rules for square brackets go as follows:
  A closing square bracket loses its special meaning if placed first in a list. for example []12] matches ] , 1, or 2.
  A dash - loses it's usual meaning inside lists if it is placed last.
  A carat ^ loses it's special meaning if it is not placed first
  Most special characters lose their meaning inside square brackets
   
jobs/queues commands
  The simplest command to submit a batch job to the default (8 minutes) queue is:
  bsub myjob.sh. To see the possible parameters, type: man bsub.
bsub –q 1nh ~user1/jobs/job1.csh Submit a job in your home directory to the 1 hour queue
  (Remember: chmod 755 filename to make the file executable)
bsub "/bin/hostname" Submit a command to the default (8 minutes) queue
bsub –M 20000 job2 Submit job2 with a 20 MB memory limit
  (It is recommended to specify a memory limit)
bsub -q 8nh -W 120 job4 Submit job4 to the 8 hour queue with a runtime limit of 120 minutes
  (It is recommended to specify a runtime limit)
bsub –R "type=SLC5_64" job3 Run job3 on a SLC5 host
Also: bjobs, bkill, bqueues, bhosts, bmgroup, bmod, bchkpnt
  brestart, bgadd, bgdel, bjgroup, sh, getrlimit
  sbrk, libckpt.a, lsb.users, lsbqueues, lsb.params
  lsb.hosts, lsb.serviceclasses, mbatchd
   
top see processes which consume most cpu, leave with Q only
kill n the process with pid n will be signalled
kill -n all processes in group n are signalled
   
ctrl alt F1 gives a terminal to login again
ctrl alt F7 to go back to X screen
ssh -X lxplus To allow tunnelin for emacs etc
ssh -Y lxplus ditto
source .bashrc Will execute the .bashrc file
rm -rf directory name or filename Will force removal. Dangerous!
fs lq quota check
fs listquota quota check
fs examine detailed quota check
chmod u+x fn to give user executable permission
chmod 755 fn to give all read and exec permission, I also get write permission
chmod 700 fn Only I get read, write and exec permission
.tcshrc location for aliases etc. Executed at logon
<ctrl><shift> t For a new terminal in the same directory
<ctrl>xs save
./hello to execute file 'hello', '.' for current directory, '/' something in this directory
ps -u davec to list my processes that are running
   
printing  
lpr lpr -P 40-1B-HPCOR -oprettyprint VPT-UVA-Brunel-info.C
  CERN linux printing page
  /usr/sbin/lpadmincern --list --building XX to list available printers
  To install all printers available in building XX run as root:
  /usr/sbin/lpadmincern --add --building XX
  /usr/sbin/system-config-printer to fine tune printer after installing
   
  Copy all files and directories in dev recursively to subdirectory bak:
  $ cp -R dev bak

Examples

~/scratch0 > fs listquota
Volume Name                    Quota       Used %Used   Partition
s.cms.davec.0                2000000    1827235   91%<<       29%    <<WARNING

~ > fs lq
Volume Name                    Quota       Used %Used   Partition
user.davec                    888000     762767   86%         46%

~/scratch0 > fs examine
File . (1933946694.1.1) contained in volume 1933946694
Volume status for vid = 1933946694 named s.cms.davec.0
Current disk quota is 2000000
Current blocks used are 1827235
The partition has 348838756 blocks available out of 488139640

Linux compiler info

Makefiles

Superuser, su

Remember to be on PCRALXP4 - NOT lxplus !!!!!!!

To add users etc to the PC
sudo su
then enter cern usrn and pwd
then addcernusr xxxxxx
May need to check bin for exact add command.

Otherwise, as a superuser:

  • su -
  • enter pc passwd, then:
  • /usr/sbin/addusercern sproston
  • check with:
  • cat /etc/passwd | grep '/afs/cern.ch/user/'

-- DavidCockerill - 07-Apr-2011

Topic attachments
I Attachment History Action Size Date Who Comment
Texttxt Emacs-from-putty.txt r1 manage 2.9 K 2014-08-03 - 11:28 DavidCockerill  
PDFpdf Linux-PATH.pdf r1 manage 42.1 K 2012-05-01 - 11:34 DavidCockerill  
Texttxt command-line-format.txt r1 manage 14.4 K 2014-08-28 - 12:02 DavidCockerill  
PDFpdf path-info-CMSSW.pdf r1 manage 17.5 K 2012-05-01 - 11:33 DavidCockerill  
Edit | Attach | Watch | Print version | History: r78 < r77 < r76 < r75 < r74 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r78 - 2019-09-23 - DavidCockerill
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback