Robert Frazier's TWiki Sandbox

A collection of useful knowledge...

How to securely append the contents of a local file to a remote file, or vice versa

Annoyed with there not being an append option/flag in scp so you can append the contents of a local file onto the end of a remote file? As far as I know, there is no way of doing this via scp (which is a shame), but there is a combination of cat & ssh that allows you to achieve the same secure file append aim:

Append the contents of a local file to the end of a remote file:

Append to contents of a remote file to the end of a local file:

Making and using SSH keys, and using the ssh-agent

Fed up of having to type your SSH password in all the time when logging in to a remote server? Read on if so...

Note: Logging into lxplus.cern.ch with SSH keys is not useful because the SSH password is recycled to also get you a kerberos (i.e. AFS) token in order for you to access your home folder. If you need to store a public key in your lxplus account's ~/.ssh/authorized_keys file, but don't want a normal login to lxplus to make use of it, then see the section on the SSH Config File

First off, create yourself a public/private SSH key pair on your LOCAL machine. The the key generator will by default put the keys in the ~/.ssh/ folder, which is fine, and it will also ask you for a password with which to encrypt/decrypt your private key with. It is - in my opinion - essential that you enter a good password for keeping your private key safe. Anyway, to make the key pair do:

  • ssh-keygen -t rsa

This should result in two files; your public key is in id_rsa.pub, and your private key is in id_rsa.

Now you need to append your public key to the remote server's ~/.ssh/authorized_keys file. If the server does not already have a .ssh/ folder, you should log in and create one. The following command will securely append your public key:

So, if you didn't know this already, if you now ssh to your remote server, it'll ask you for the password to your private key. At first this seems disappointing, as it looks like you still have to type in a password all the time. However, you can make use of an SSH agent to look after your private key. If using a Linux machine, you can do something fairly simple like this:

  • eval `ssh-agent`
  • ssh-add ~/.ssh/id_rsa

And from now on, you won't need to type in the password to access your private key. You can see what keys the agent is looking after like so:

  • ssh-add -l

The downside of using the ssh-agent as shown above is that you can end up with many agents being active, and if you start a second terminal session up on your local machine, it won't pick up on the fact that you already have ssh-agent running. This is annoying. Fix this problem with the following very handy script in your local machine's .bash_profile login file. This will start up a single agent, and once you've put in the password for your private key, any further console sessions will pick it up:

######################################################################
# Cunning SSH-agent script from Joseph M. Reagle as found on Cygwin mailing list,
# with a small host-specific modification from me when working on shared filesystems.
#
# Summary: if there is already an ssh-agent (of yours) running on the host it will use it.  If there
# isn't, it starts one and imports your private key.

SSH_ENV=$HOME/.ssh/environment.`uname -n`

function start_agent {
     echo "Initialising new SSH agent..."
     /usr/bin/ssh-agent | sed 's/^echo/#echo/' > ${SSH_ENV}
     echo succeeded
     chmod 600 ${SSH_ENV}
     . ${SSH_ENV} > /dev/null
     /usr/bin/ssh-add;
}

# Source SSH settings, if applicable

if [ -f "${SSH_ENV}" ]; then
     . ${SSH_ENV} > /dev/null
     #ps ${SSH_AGENT_PID} doesn't work under cywgin
     ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
         start_agent;
     }
else
     start_agent;
fi
######################################################################

Now every time you login, the above script will run. You should have to type the password to your private key rarely from now on.

Note: On Macs, the above may work, but I haven't tested it. Instead, I make use of the excellent SSHKeychain.

SSH Config File

The SSH config file can be used for many things, such as ensuring a certain "quietness" of login, to switching off public-key authentication for certain hosts. Create (or add to) the following file: ~/.ssh/config, and you can add hosts and options as shown in the example below:

host lxplus.cern.ch
PubkeyAuthentication no 
PasswordAuthentication yes
LogLevel QUIET

host cmscvs.cern.ch
ForwardX11 no
LogLevel QUIET

CERN CVS access via "ext" ssh authentication

As well as the instructions in the above links, you can make use the instructions on SSH keys + ssh-agent found HERE

A more useful bash prompt

Coloured prompt, so you can review your history and see commands more easily. It also timestamps commands - which is quite useful sometimes - and gives you your file location relative to your home directory. Put the following in your .bash_profile file:

# Setup nice coloured bash prompt
export PS1="\[\033[1;34m\][\t][\u@\h] \w>\[\033[0m\] "

A more useful bash command history file

# Setup a big history file, prevent duplicate commands in file, ignore ls commands.
export HISTFILESIZE=100000
export HISTSIZE=100000
export HISTCONTROL=ignoreboth
export HISTIGNORE="ls"

Four-vector TrackTrigger Simulation and Fireworks

Fireworks install instructions here: https://twiki.cern.ch/twiki/bin/view/CMS/WorkBookFireworks

Then you need to point cmsShow at the tracks.root output of the KHTriggerStudy/PerformanceAnalysis/test/test_trackParams.py job. (i.e. ./cmsShow tracks.root)

Once fireworks is running, you need to make a couple of alterations to see all the relevant info. In order to make visible the reconstructed tracks from the Four-Vector Track Trigger simulation, go to the Window menu --> Show Add Collection. Then, in the dialog that opens click on "Tracks recoTrackProducer" in the lower window, and then give it a name in the box above, such as "RecoTrackTrigTracks". Now click Add Collection.

Next, get rid of all the GenParticle tracks that our simulation doesn't reconstruct, otherwise you don't get a fair comparison. Right click on the GenParticles tickbox label on the left-hand side and it should open up a "Collection Controller" window. Under the filter tab, replace the existing expression with the following expression and then hit the filter button:

($.pt() > 5 && $.charge() != 0 && $.status() == 1)

This will select only charged genParticles with a p_t greater than 5 GeV that are stable. The four-vector track trigger simulation ignores unstable particles, hence switching these off in the fireworks viewer. Now you should hopefully see purple genParticles and white reco tracks overlaying the genParticle tracks from the innermost barrel layer onwards. Reduce the pt cut if you want to see more genparticles.

An Excellent Python Startup Script

Slightly modified the original... but only very slightly. Gives you command history from file and tab completion:

# Clever trick to add history support and tab completion to cPython.
# This was stolen from a fellow ubuntero's blog and extended to add
# more error checking along with the ability to be ran system-wide.
#
# Save as /etc/pythonrc and put 'export PYTHONSTARTUP=/etc/pythonrc' in
# /etc/bash.bashrc. This makes the functionality available system-wide.
#
# Also works find as a local python startup file:
# Save as ~/.pythonrc and put 'export PYTHONSTARTUP=$HOME/.pythonrc' in
# ~/.bash_profile, or similar.
#
# Cleaned up 7/11/2009 by Jeff Schroeder <jeffschroeder@computer.org>
# Released under the public domain for all to share and share alike.

try:
    import readline
except ImportError:
    pass
else:
    import os
    import atexit
    import rlcompleter

class irlcompleter(rlcompleter.Completer):
    def complete(self, text, state):
        if text == "":
            readline.insert_text('\t')
            return None
        else:
            return rlcompleter.Completer.complete(self,text,state)

# You could change this line to bind another key instead tab.
readline.parse_and_bind("tab: complete")
readline.set_completer(irlcompleter().complete)

# Restore our command-line history, and save it when Python exits.
historyPath = os.path.expanduser("~/.pyhistory")

# Create a blank history file if it doesn't exist already
if not os.path.exists(historyPath) and not os.path.isdir(historyPath):
    try:
        open(historyPath, 'w').close()
    # Gracefully ignore things if historyPath is not writable
    except IOError:
        pass

# Read the history file in for autocompletion and save it on exit
if os.access(historyPath, os.R_OK):
    readline.read_history_file(historyPath)

if os.access(historyPath, os.W_OK):
    atexit.register(lambda x=historyPath: readline.write_history_file(x))

# Some optional deletes at the end to clean things up a bit.
#del os, atexit, historyPath

-- RobertFrazier - 08-Oct-2009
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2010-01-24 - RobertFrazier
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback