In general, you'll do your dune-artdaq / dune-raw-data development in an area which you've installed, on a branch designed for the specific work you're doing (e.g. the "fragment_generator_for_component_X" branch rather than the "develop" branch) and which only you (or people in your subgroup) make code changes in. Once you're confident that your code changes are ready to be used by everyone working with the dune-artdaq package, you'll want to push your branch to the central repo, at which point experts will merge your changes into the develop branch and deploy them to the standard dune-artdaq installations. There are three ways to push your branch to the central repo: the recommended way is shown first.

Standard method

It's now possible to push your code changes on the np04-srv-* machines directly into the central repositories at Fermilab. The instructions on how to get set up to do this are below; as a prerequisite, however, you'll want to have write access to the dune-artdaq (and, if applicable, dune-raw-data) repositories. You can check this by going to https://cdcvs.fnal.gov/redmine/projects/dune-artdaq and/or https://cdcvs.fnal.gov/redmine/projects/dune-raw-data and seeing whether your name is in the gray box near the upper right hand of the screen which says "Members". If not, contact myself (John Freeman, jcfree@fnalNOSPAMPLEASE.gov) or anyone else under the subheading "Manager" (as opposed to "Developer") in the gray box to get write access.

If you have write access, to get set up to push, carefully follow these steps:

From the computer you usually use to log in to lxplus before subsequently logging into the ProtoDUNE cluster, execute

kinit <your username at cern>@CERN.CH # Obviously, replace <your username at cern> with your actual username
ssh -XK <your username at cern>@lxplus7.cern.ch
and once you're logged into the lxplus node, first take note of the node you got logged onto by looking at the $HOSTNAME variable - we'll want to use it a few steps below. Then get into the ProtoDUNE cluster:
ssh -XK <your username at cern>@np04-srv-010 
and once you're onto the ProtoDUNE cluster, if you haven't already done this (i.e., this is your first time going through these instructions), you'll first want to back up your ~/.ssh directory since we're going to make changes to it (e.g., "cp -rp ~/.ssh ~/.ssh.backup). Then you'll want to create what's known as an "ssh key pair"; as a practical matter, this means you execute
cd ~/.ssh
ssh-keygen 
and what you'll see is
Generating public/private rsa key pair.
Enter file in which to save the key (/nfs/home/jofreema/.ssh/id_rsa): 
Hit Enter. Then you'll see:
Enter passphrase (empty for no passphrase):
and enter a not-easy-to-guess but easy-to-remember password - you'll be asked for this password when you try pushing code to the central repository. This command will create a file called ~/.ssh/id_rsa.pub (a "public key") and ~/.ssh/id_rsa (a "private key"). Keep the id_rsa file readable only by you - it's supposed to be private.

Next, you'll want to add the following code snippet to your ~/.ssh/config file (or if ~/.ssh/config doesn't exist, create a file with this name containing the snippet below):

Host cdcvs.fnal.gov
    ProxyJump <your username at cern>@<the lxplus node you used to get onto the cluster, expressed with the ".cern.ch">
    ForwardX11 = no
    ForwardAgent true
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
A word concerning the "<the lxplus node you used to get onto the cluster...>" token in the snippet. You'll want to of course replace this token with what it refers to. But in general, what you want is a working lxplus node listed in your ~/.ssh/config file. In this writer's experience, pushing code failed when the specific node listed became unavailable (down for maintenance, etc.) and started working again when I replaced it with a functioning lxplus node. Keep this in mind going forward.

After adding the snippet, execute the following:

ssh-agent bash
ssh-add $HOME/.ssh/id_rsa 
and after this last command, you'll be asked for the not-easy-to-guess but easy-to-remember password which you created earlier in the instructions.

cd into the dune-artdaq git repository in which you're developing and do the following:

git remote set-url origin ssh://p-dune-artdaq@cdcvs.fnal.gov/cvs/projects/dune-artdaq
If you plan on pushing changes to dune-raw-data, then you just cd into the dune-raw-data repository and follow the previous instructions but replace "dune-artdaq" with "dune-raw-data". Make sure you use the correct repository name -- if, e.g., you set the url for a dune-raw-data repo as "ssh://p-dune-artdaq@cdcvs.fnal.gov/cvs/projects/dune-artdaq", then disaster can occur when you try pushing your changes.

Before you can push changes to the central repository, there's a final step. Contact John Freeman (jcfree@fnalNOSPAMPLEASE.gov) and/or Kurt Biery (biery@fnalNOSPAMPLEASE.gov) and ask them to add the public key you created with the ssh-keygen command to the config file in p-dune-artdaq@cdcvsNOSPAMPLEASE.fnal.gov (and p-dune-raw-data@cdcvsNOSPAMPLEASE.fnal.gov as well, if applicable). Once you've done this, you won't need to repeat all the steps described above, you'll simply be able to push directly to the central repo from your local repo on np04-srv-* as long as you enter the password you created and as long as the particular lxplus node you listed in ~/.ssh/config is working.

Troubleshooting the standard method

Sometimes, you may see a message like the following:

ssh: connect to host lxplus717.cern.ch port 22: Network is unreachable
ssh_exchange_identification: Connection closed by remote host
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

What this means is that the node you listed in ~/.ssh/config isn't working. If this is the case, replace the node (in the example above, lxplus717) with the lxplus node you used to log into the np04 cluster with, since by definition you know it'll be working.

Alternate method

Another technique by which developers can set up to push code to the central repository, created by Phil Rodrigues in 2018, can be found here.

Straightforward-but-cumbersome method

To push, you'll want to be on lxplus rather than np04-srv-XXX. So first, log into an lxplus node:

ssh -l <your username> lxplus7.cern.ch
and then once you're on lxplus
scp -rp np04-srv-014:<path to dune-artdaq or dune-raw-data repo you modified> .
kinit <your fermilab username>@FNAL.GOV
cd <modified repo you scp'd>
git remote rm origin
# The command below could also be ssh://p-dune-raw-data@cdcvs.fnal.gov/cvs/projects/dune-raw-data
git remote add origin ssh://p-dune-artdaq@cdcvs.fnal.gov/cvs/projects/dune-artdaq 
git push origin <branch you want to push>
once you've done this, send an email to John Freeman and Kurt Biery (jcfree@fnalNOSPAMPLEASE.gov and biery@fnalNOSPAMPLEASE.gov) announcing you'd like your changes merged into the develop branch, and CC Karol and Giovanna, revkarol@gmailNOSPAMPLEASE.com and Giovanna.Lehmann@cernNOSPAMPLEASE.ch.

-- JohnChristianFreeman - 2019-12-05

Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2019-12-05 - JohnChristianFreeman
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CENF All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback