-- ParasNaik - 2016-04-12

git instructions for the RICH mirror alignment

Because we are no longer using SVN.

Table of Contents

Nota Bene

NEVER edit the code directly at the pit (e.g. in the AlignmentRelease area). When using multiple projects with git, things can get extremely confused and it will be very difficult to find and merge any changes into to code!

Please make your own branch of any code you change. Then simply lb-checkout the code from your branch into the AlignmentRelease area when you want to install it.

git

It is time to git! Please read everything and please follow the workflow Anatoly describes. If you have any questions email the alignment-development mailing list. If we cannot answer your question, email Jordi, and if all else fails then Marco Clemencic / lhcb-core-soft@cernNOSPAMPLEASE.ch

The basic idea behind git is that "branches" are used to develop new features of the code, each branch forks from the master branch (the "default" branch) of a git repository, and each new branch is isolated from the other branches, until one chooses to merge them. Typically we use these other branches for development, and then merge them back to the master branch when the new feature is tested and ready. At LHCb we have to ask for a merge request in CERN's GitLab to merge one of our branches into the master branch of a Project. Then the Project manager can decide whether to allow the merge to occur.

Standard documentation and other git stuff

Transition of Panoptes from svn to git.

The Panoptes project has been moved to gitlab. https://gitlab.cern.ch/lhcb/Panoptes
The lhcb-rich-software egroup is linked to the gitlab group such that all members have "developer" access.

There is a major change in the workflow, since we cannot use tagged packages in git anymore.

The main consequences are the following:

  • CmakeLists and cmt/requirements still need to have a package version number, these should be updated when necessary.
  • However, packages don't have version tags anymore. From now on, a Panoptes project tag affects all its packages. What used to be "version v2r26 of Rich/RichOnlineMonitors" is now the "Rich/RichOnlineMonitors that comes with Panoptes v5r6."
  • The tag collector becomes obsolete from the point of view of Panoptes.
  • Since we still need to release Panoptes versions, all packages should at least compile when they are added to the master branch of the git repository. For this reason, it is important to have meaningful development branches for stuff that is not yet stable, which will be merged once the new features are fully implemented and tested.

The recommended workflow to add features to packages is the following:

  • Implement the features in a branch.
  • Test them.
  • Push the branch to an upstream repository (e.g. a fork of the project in gitlab).
  • Submit a merge request, so the new features are included in the master branch.

Please get used to the git workflow ASAP.

The JIRA rich project.

Jordi has created a JIRA (project and) component group for (the RICH group and) the mirror alignment, which you can find here. Jordi is not an expert in JIRA, so Jordi cannot guarantee that the configuration that he has done is ideal. Any suggestions are welcome.

The JIRA group does NOT impose any change in the workflow. We could keep managing issues by email, or external service, or keep track of your tasks by tweeting them... However, people who have used JIRA have a good experience in their opinion, so we have decided to use it.

In JIRA there are these things called "*components*", which correspond to different unrelated topics (or mostly unrelated) within a project. Jordi has created four software components: monitoring (for Panoptes issues), alignment (for mirror alignment), piquet (for piquet software tools) and reconstruction (for reconstruction software). Please make sure any new JIRA issues for the alignment are created with component alignment.

(In principle, JIRA can be useful for hardware projects as well, so feel free to create as many components as you want for hardware and for other software topics.)

Rich JIRA issues have references like RICH-XXXX, where XXXX is a number. If you mention a JIRA issue in a git commit, JIRA will automatically create a comment in its corresponding issue page, so please do this (example below).

Members of lhcb-rich-software have "developers" access to the rich JIRA project. (Let Jordi know if you prefer that Jordi changes it to lhcb-rich.)

Anatoly's example git workflow for Panoptes [or Rec]

Anatoly was kind enough to provide us with his git workflow so far. He is still learning as are we, but this will certainly help you get started. Paras is keeping this updated as new information arrives.

To build

Change usernames and email addresses below as appropriate

[ Change projects throughout below, if instead of Panoptes you are working with e.g. Rec ]

Create a new shell

Check your git version

  • git version (sadly there are version-dependent commands below)

Only once and forever do these three things:

These should be done every time you want to start coding (NOTE the choice of User_release_area is completely up to you, actually, but should be in your AFS area):

  • mkdir /afs/cern.ch/user/a/asolomin/gituser (where asolomin = your username)
  • setenv User_release_area /afs/cern.ch/user/a/asolomin/gituser (where asolomin = your username)
  • LbLogin (for some reason you have to run this after setting the user release area)
  • cd $User_release_area

Then this you do only once, when you start from scratch (though peform it separately when starting from each new Panoptes release, here I assume the newest is v7r1, you should check in gitlab):

  • lb-dev --nightly lhcb-head Panoptes/v7r1

These you do as long as you are working with the project and version in the previous line (NOTE the "Dev"):

  • cd $User_release_area/PanoptesDev_v7r1
  • git lb-use Panoptes

To checkout a package (e.g before you start work on a new package for the first time)

  • git fetch --all
  • git lb-checkout Panoptes/master Rich/RichMirrAlign (or perhaps a different package)

Now you can edit the code. You would obviously not necessarily edit the same files, and of course you should use your editor of choice (e.g. nano, emacs, vi, etc...) or nano as in this example:

  • =nano Rich/RichMirrAlign/CMakeLists.txt =
  • =nano Rich/RichMirrAlign/cmt/requirements =
  • nano Rich/RichMirrAlign/doc/release_notes.html
    • Always update the release notes!, only change the other two if the package "version" needs to be updated

Check that your code compiles:

  • make -j 8 (use 8 cores)

Get a list of files that changed:

  • git status or

Check all of your changes (except whitespace changes)

  • git diff --ignore-blank-lines --ignore-space-change (only works in git version 1.8.4 or higher)
  • git diff --ignore-space-change (always works)

For everything that changed and any new files use git add (e.g. for the previous example):

  • git add Rich/RichMirrAlign/src/RichMirrAlign.cpp
  • git add Rich/RichMirrAlign/src/RichMirrAlignFcn.cpp
  • git add Rich/RichMirrAlign/src/RichMirrAlignFcn.h
  • git add Rich/RichMirrAlign/doc/release_notes.html (Please always update the release notes!!!)
  • git add Rich/RichMirrAlign/CMakeLists.txt
  • git add Rich/RichMirrAlign/cmt/requirements

Then make sure to update the commit message:

  • git commit -m 'RICH-9999 regularizationMode introduced.'
RICH-9999 is the associated JIRA task, which automatically links the commit (or later, merge request) to the JIRA task

If after your commit, you edit a file, say Rich/RichMirrAlign/doc/release_notes.html you must git add it again:

  • git add Rich/RichMirrAlign/doc/release_notes.html

If after your commit, whether you changed any files or not, you wish to edit or expand your commit description before a push use:

  • git commit --amend

As new commits are only stored in your local repository, there's no cost to committing often.
You should try to make a new commit every time you've made modifications that can be considered a single unit of changes.

You will now give your branch a name. You could choose anything (but pick something unique!). However let's try to stick to the following prescription: username-YYYYMMDD-title (e.g. asolomin-20170123-RICHMirrorAdjust).
So for example, to push your changes to a new branch called asolomin-20170123-RICHMirrorAdjust:

  • git lb-push Panoptes asolomin-20170123-RICHMirrorAdjust

Then, submit a merge request for asolomin-20170123-RICHMirrorAdjust from the Panoptes gitlab page You can also use the hyperlink provided when you commit, though if it's a sensitive change you may want to look at your commit in gitlab first. NOTE: Your pushed branch (that contains your commit) MUST COMPILE locally to get accepted (though it may not compile against the nightlies).

Important: if you noticed a mistake, fix it, and immediately want to commit a corrected version to a new branch, you should

  • git lb-push Panoptes asolomin-20170123-RICHMirrorAdjust2 (i.e. any different name)
Then, submit a merge request for asolomin-20170123-RICHMirrorAdjust2, instead of asolomin-20170123-RICHMirrorAdjust

Now say you already made a merge request, then you could commit a fix and push it to the same branch for which you made the merge request. Before you start work on the fix however, make sure to change the first word of the title of your merge request to "WIP: " (work-in-progress). This instructs the Project manager to wait until you remove "WIP: " to merge the branch.

If you no longer want the merge request to be accepted (e.g. it was done by mistake) then close the merge request. If the associated branch is no longer in use you may delete it from the Panoptes gitlab page.

To continue working on the same branch after a git lb-push (presuming no other changes were made to your branch [e.g. in GitLab]):

  • git fetch --all # (you must do this in order to get access to remote changes in Panoptes/master or any other branch)
  • git lb-checkout Panoptes/asolomin-20170123-RICHMirrorAdjust Rich/RichMirrAlign
However if you push again to the same branch then it will become part of any existing merge request as well. You can always keep pushing changes to new branches, this is usually preferred unless someone needs you to fix your merge request.

After the merge requests are applied and only after, when you are ready to start work again you will want to update your local packages from the master, incase there were any changes in the master in the meantime

  • cd $User_release_area/PanoptesDev_v7r1
  • git fetch --all # (you must do this in order to get access to remote changes in Panoptes/master or any other branch)
  • git lb-checkout Panoptes/master Rich/RichMirrAlign

To run

new shell ( e.g. ssh -Y asolomin@lxplusNOSPAMPLEASE.cern.ch )

  • setenv User_release_area /afs/cern.ch/user/a/asolomin/gituser
  • LbLogin
  • cd $User_release_area/PanoptesDev_v7r1
  • make install
  • cd /afs/cern.ch/user/a/asolomin/public/rich_align_test (or wherever your test is)
  • lb-run PanoptesDev v7r1 RichMirrAlign.exe Rich1MirrAlign_Mp6Wi8.0Fm5Mm0Sm0_online_Collision15_i0.conf

Slight differences to the above for the Hlt project

Substitute the following above, where appropriate

  • lb-dev Moore/v25r3 (or latest version, check in gitlab)
  • cd $User_release_area/MooreDev_v25r3
  • git lb-use Hlt
  • git fetch --all
  • git lb-checkout Hlt/2016-patches Hlt/HltSettings (perhaps you will be asked by HLT Operations to use a different branch, or need to edit a different package)
  • nano Hlt/HltSettings/python/HltSettings/Physics_pp_Draft2016.py (example)
  • nano Hlt/HltSettings/doc/release.notes
  • make -j 8 (check that your code compiles!)
  • git add Hlt/HltSettings/python/HltSettings/Physics_pp_Draft2016.py
  • git add Hlt/HltSettings/doc/release.notes
  • git commit -m 'RICH-9999 Rich Mirror Line prescales reduced by a factor of 3'
  • git commit --amend

You will now give your branch a name. At this stage you may choose anything (but pick something like {your name}-{something describing your change} that helps you keep track of things). Say you pick pnaik-20160630-RICHMirrorAdjust.
This will push your changes to the pnaik-20160630-RICHMirrorAdjust branch:

  • git lb-push Hlt pnaik-20160630-RICHMirrorAdjust
Then, submit a merge request for pnaik-20160630-RICHMirrorAdjust from the Hlt gitlab page. Create a merge request with the 2016-patches branch. NOTE: Your pushed branch (that contains your commit) MUST COMPILE to get accepted.

Important: if you noticed a mistake and immediately want to commit a corrected version, you should

  • git lb-push Hlt pnaik-20160630-RICHMirrorAdjust2 (i.e. any different name)
Then, submit a merge request for pnaik-20160630-RICHMirrorAdjust2, instead of pnaik-20160630-RICHMirrorAdjust

If you no longer want the merge request to be accepted (e.g. it was done by mistake) then close the merge request. If the associated branch is no longer in use you may delete it from the Hlt gitlab page.

Notes from Paras about our global workflow

Typically before we submit a merge request, we want to "svn update" or whatever the equivalent of getting changes that others made in the repository over to our branch, and then "rebased". However, as of June 2017 I am still not exactly sure how to do it correctly.

Fortunately with GitLab, we can leave (simple) rebasing to the Project managers. Just prepare your change then commit. You can start your merge request in gitlab right away, but if you don't want it to be actually merged please put "WIP: " in front of the title of the merge request. Then you can commit more to that branch if you want. But as soon as it is ready please remove "WIP: " and ask Jordi or Chris to merge the branch. Then after the branch is merged update locally to what is in master:

  • git fetch --all # (you must do this in order to get access to remote changes in Panoptes/master or any other branch)
  • git lb-checkout Panoptes/master Rich/RichMirrAlign (or instead of Rich/RichMirrAlign, whichever package you are going to work on)

Further details on git commands (thanks to Jordi)

1. You should not associate the master branch with the SVN trunk. In SVN, tagged packages were expected to work, and committers could add unstable stuff to the trunk. With the new workflow that we believe is reasonable for Panoptes, the master branch is expected to always be stable, as opposed to the SVN trunk. Unstable contributions to git should take place in branches. So, to summarize, in svn, new contributions go from an unstable trunk to stable branches and from there to release. In git, new contributions go from unstable branches to stable master and from there to release.

2. Simple git workflow for Panoptes (you should of course pick the newest tagged Panoptes version):

lb-dev Panoptes v7r1

This creates a local git repository in PanoptesDev_v7r1. This is called a "satellite project". Then,

cd PanoptesDev_v7r1
git lb-use Panoptes

This adds the gitlab remote to the local repository of the satellite project, and fetches that remote (makes a local copy of the remote repository, although you still don't see it in your working tree).

The output of this last command will give you information like this:

* [new branch] master -> Panoptes/master
* [new tag] v1r0 -> Panoptes/v1r0
[...]
* [new tag] v6r0 -> Panoptes/v5r5
* [new tag] v7r1 -> Panoptes/v7r1

You will see that there are several references that you can use. Some of them are branches (right now only master) and some of them are tags. You will refer to them as Panoptes/[reference], as in the last column of the previous output.

Lets say you want to develop on top of the Rich/RichMirrorAlignmentOnline package that you can find on the master branch. You would do

git fetch --all
git lb-checkout Panoptes/master Rich/RichMirrorAlignmentOnline

This is the equivalent of checking out the package from the head. Then you would add new features to the mirror alignment software, and maybe you will commit unfinished work that you want to share with others. In this case, you would of course make the edits and then git add them (git add must be done immediately before a git commit / git lb-push; if any file changes git add must be done again, and use git commit --amend if you want to edit the git commit -m note):

... edit filename ...

git add filename
git commit -m "edited filename"

and then lb-push to a new branch:

git lb-push Panoptes newfeature

Here, newfeature is the name of your development branch, and you will be able to see it in gitlab. It should be short, but descriptive. If you then would like the commits in newfeature to be merged into master you must perform a Gitlab merge request (there should be a weblink given to you when you lb-push).

If someone else has lb-pushed a new feature branch called that-branch to Panoptes, you can bring it to your local copy with

git fetch --all
git lb-checkout Panoptes/that-branch Rich/RichMirrorAlignmentOnline

You can also tag specific commits on the branch, so you can easily refer to them with others. Please don't use v-tags: the Panoptes project should be the only thing that is v-tagged. If v-tags help you understand better where you are, please prefix them with some short context, e.g. align-v1r0. Basically the only place a package v-tag should show up is within the CMakeLists and cmt/requirements of each package

3. The lb-use and lb-checkout commands also solve the case where you want to use packages as plugins from other projects. For example, you can very consistently do:

git lb-use Rec
git fetch --all
git lb-checkout Rec/v20r1 Rich/RichAlignment

So this is how you would use the Phys/Rec package in Panoptes.

4. New Panoptes releases are created when there's interest in them. It may be for several reasons: to have new features released, to use a new release of Online, to compile with a new CMTCONFIG,... Jordi never opposes creating a new release if it is requested.

5. Currently Jordi Garra Tico and Chris Jones are the only people that can approve merge requests.

List of questions to be asked and answered (by Jordi, etc...) at an appropriate time

Is there a way, say after you commit “mirralign23”, to do something similar to “svn update” to get your local branch to be in sync with the master branch (say, if your changes are accepted, then someone else changes it), or would you have to check out the head version each time?

  • Q: You cannot "git lb-push" to a branch already existing in the remote repository, unless you checked out from that specific branch?
    • A (?) : I think, it is without "unless", meaning that such a branch is always unique, but keep this Q too.

Frequently used commands

see above for starters, but here are a few more

git diff

  • shows all changes between what you checked out and your last commit.

git status

  • lets you know which files have changed since your last git lb-push

git commit -a -m "Tagged Rich/RichMirrorAlignmentOnline as v68r43. Updated Release Notes."

  • is a combination of git add for all files that changed (it does not cover new files to be added to the repository though!) and git commit -m "Hello"

git log

  • tracks everything you have been doing

git log -U3 (3 can be changed to any number)

  • git log with diffs from 3 (or another number) lines before to 3 (or another number) lines after any changes

Troubleshooting

Q: I pushed a commit to a branch for which I already have a merge request. However my new commits don't show up in that merge request!

A: Push another commit

Legacy repository information

In the cloned gitlab repository, https://gitlab.cern.ch/LHCb-SVN-mirrors/Panoptes, all contributions to svn up to release v5r5 have been getpacked and committed, but no other history is kept about them. After release v5r5, all the history has been kept.

For reference, Jordi has kept a complete copy of the SVN repository at https://gitlab.cern.ch/lhcb-rich/panoptes In this longer version, all the svn history is kept and, additionally, all tagged releases have been getpacked and attached at their corresponding point in history. This is not intended to be used in practice, just for reference. Jordi will update this repository just once in a while but not very frequently.

Jordi has created the gitlab group lhcb-rich, intended for rich related repositories. https://gitlab.cern.ch/groups/lhcb-rich

Edit | Attach | Watch | Print version | History: r35 | r29 < r28 < r27 < r26 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r27 - 2017-07-03 - ParasNaik
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCb All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback