Difference: BL4SGitGuide (1 vs. 2)

Revision 22017-03-28 - BranislavRistic

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
<!-- 
 -->
<!-- 
-->

An introduction to Git and Gitlab for BL4S code

The development flow for BL4S

We have adapted what is called the GitHub flow. This means in particular:

  • The master branch is always functional, it compiles and runs at all times.
  • All code in the master branch is reviewed, hence direct pushing to the branch is prohibited.
  • Changes to the code shall be introduced by feature branches following the naming scheme RESPONSIBLE-INTENTION using only - and _ as special characters (e.g. bane-setup_scripts).
  • Branches are merged into master by means of Gitlab Merge Requests.
  • Ideas/Suggestions/Bugs should be documented in Issues in the Gitlab issue tracker.

As nothing can be directly pushed to the master branch, every change, from a small bugfix to a complete rewrite of core components, has to be introduced via feature branches.

  1. Optimally every change starts with at least one entry in the Gitlab Issue Tracker where the changes and intentions can be described and discussed. If you wish to directly involve others in the discussion, mention them in the description by the usual @NICK notation.
  2. Start a feature branch and push it to the server as soon as possible. By this you show your fellow collaborateurs that you are tackling a problem. You can already open a merge request if you like and mark it as work in progress by starting the name with WIP:
    1. Code, code, code and push it to the server. This branch is yours and you can do to it what you like: rebasing, ammending commits,... If you work together with someone else, pleas sync with them.
    2. If the master branch has advanced in the meantime, consider rebasing on the new master.
  3. Finally open a merge request and assign it to someone you'd like to review your code.
  4. Write a meaningful commit message (Button: Modify commit message).
  5. Upon receiving comments, implement and push them to your branch. The changes will automatically be added to the merge request.
  6. Wait for approval by the assignee.
gitlab_flow.png

A brief introduction to Git

Git is a sophisticated version controlling system and you'll find many excellent tutorials in the web (e.g. git-scm,Atlassian or if you like it interactive GitHub) The following section is describing the basic git tools and is by no means complete (or fully correct). It targets at contributing code for BL4S and some of the exeptions to usual practice could make you quite unpopular in other places.

In general, Git's motto is "everything is local". This means, you always have a complete copy of the repository with the full history of all branches. You are in full control of this copy and its contents and can play around like you wish; and you should play to get familiar with the system. As long as you do not use commands to sync with the Gitlab server (push) it will not affect our central repository. If your local copy is messed up, just clone it again from the central repo and start over.

The usual git flow is:

  1. Get a copy of a remote repository (git clone)
  2. Create a feature branch and push it early (git checkout -b NAME; git push -u origin NAME)
  3. Make changes to the local repository
    1. Edit, add and delete local files.
    2. Stage/Bundle your changes for a commit (staging) (git add [-p])
    3. Create a commit (git commit) <-- This is all local.
    4. Goto 3.1 until you are satisfied
  4. Merge parent branches into the feature branch or rebase on parent (prefered)
  5. Sync from/to remote repository (git pull/git push)

Glossary

Repository
A folder containing a .git subdirectory which actually contains the git structure. Repositories can be accessed via usual file tools (ls, cd, cat, cp..., for rm, mv consider using the git equivalents) or served protocols like ssh or https.

Location of files and changes in the local repository
  • The working tree: Currently checked out files that you can handle just like with any other file.
  • The index/staging area: Changes to the working tree, made known to git but not committed yet. Kind of a blue print for the next commit.
  • Commit: A collection of changes to the repository (e.g. added/deleted files, modification (patches) to files, the name/email address of the committer, date, a reference to the parent commit(s)) and a SHA-1 sum of all these information.

Remote (repository)
There can be multiple remote repos attached to a copy. When cloning a repo, a remote called origin, pointing to the location of the original repo is automatically added to the local copy.

Branch
A stream of commits on which end a so called HEAD commit marks the current tip of the branch. Commits can be part of multiple branches and branches can start at arbitrary commits (branch of a branch of a branch... is possible).
Git keeps a history of all local and remote branches, that is branches from remotes (e.g. origin/master). A local branch example can track a remote branch origin/example which means, git pull/push will automatically try to keep these two branches in sync.
If a branch is checked out that has a remote counterpart with the same name the local branch will automatically track the remote one.

Stash
A temporary storage of changes. Useful if one wants to store away some work in progress which is not yet ready for a commit

Git commands

git-operations.png

Cloning a remote repository

git clone URL [DIR] (e.g. git clone https://gitlab.cern.ch/BL4S/TDAQ.git)

This creates a local copy in the empty or non-existent directory DIR ot in a subfolder named like the repository. You'll get the whole remote repository with the full history, all contained in a folder. This is your copy, you can do whatever you like with it. (e.g. making a copy of the folder will create a fully functional replica of the local repository.)

Checking the status of the files (git status)

git status

Untracked: Files which can be included to git.
Changes to be committed: Modifications to the repository (modification/insertion of/deletion of files), which will be part of the next commit.
Changes not staged for commit: The opposite of the last point. (see git add)

Preparing changes to be committed (git add/reset)

git add
adds changes to the currently checked out status of the repository to the staging area or index. Refrain to use this command without any parameters. Instead use it on individual files and folders.
  • Add new files git add files/dirs. Please be careful not to add temporary or data files. Check with git status.
  • "Add" changed files git add [-p] files/dirs: Copies changes to the files (patch) to the index. Using the -p/--patch flag is strongly recommended: Git will present you with your changes chunk by chunk and ask if you want to stage them.
  • Delete files: Use git rm instead of the plain rm. If the file is already deleted, you can inform git by git rm or add PATH.
Reset the index/remove changes: git reset [files...] This will just remove the changes from the index and leave the modification to your working tree intact. Reset all, the index and the working tree: =git reset --hard=
WARNING: All your uncommitted changes to tracked files will be lost.

Creating a commit (git commit)

git commit [-m "message"]

If you omit the -m "message" (mind the "") an editor will open for the commit message.
The first line of the message should summarize the changes in no more than 52 characters. It can be followed by a blank line and then a more in-depth description of the changes. Please brake lines at 72 characters.

Editing a commit:
If you haven't pushed your changes to a remote repository yet (or are working on your very private branch), you can edit your last commit. Just work like preparing a normal commit and run git commit --amend. This will add your corrections to the last commit.

Publishing changes to the repository (git push)

git push [-u] REPO BRANCH
(the repository you cloned from is called origin; branch must not be master)

The -u flag sets the local branch to follow the remote one. This is useful, so you don't need to specify to which branch to sync everytime you issue a push/pull.

This command will fail if the branch on the remote repository changed in the meantime. You will need to merge or rebase (see below) your current branch to the remote one.

Getting changes form a remote repository (git fetch/merge/pull)

git fetch [REPO]: Fetches changes from a remote repository but leaves local branches unaffected.

git merge REMOTE_BRANCH (e.g. origin/example). Merge the changes from the remote branch into the currently checked out (local) one. The remote branch is in the state of the last git fetch/pull, it will not be automatically fetched from the server. Git will try to fast-forward the branch and if not possible, create a merge commit.

git pull REPO: Actually just the combination of git fetch REPO and git merge REPO/BRANCH.

Rebasing a branch (git rebase)

Rebasing means changing the starting point of a branch (you might get a headache here if you come from svn). This can be used as an alternative to merging and produces a more linear history (see figure). It actually starts at a given commit/branch and applies the commits that were added to the current branch.

comparison_merge_rebase.png

It usually works like invoking git merge: git rebase BRANCH/COMMIT
This will let the current branch start at the specified BRANCH or COMMIT.

Just like merge, rebase cannot solve all conflicts by itself. If in doubt, it will stop and ask you to solve the conflict and add the changes to the index. In contrast to merge you then run git rebase --continue and it will proceed with the rebasing. If anything goes wrong, you can run git rebase --abort and your repo will be restored to the state just before the rebase.

This command is extremely powerful as it allows you to completely rewrite the history of a branch (deleting/reordering/editing commits,...). This also means, that the rebased branch is actually completely different from the original one in terms of version controlling. Use it with extreme caution on already pushed branches. The only exception to this rule are branches in merge requests on Gitlab (see below TODO: ANCHOR).
Tipp: Have a look at the rebase interactive mode.

Ignoring temporary files: .gitignore

A .gitignore file defines patterns of file names which will be ignored in the current and all sub directories. Every directory can have its own .gitignore which will add to the rule set of the parent directory.
Warning: These rules do not apply to already tracked files.

Example of working with a git repository


<!--/twistyPlugin twikiMakeVisibleInline-->
Start by cloning the repository from gitlab. (git clone)
BL4S $> git clone ssh://git@gitlab.cern.ch:7999/BL4S/TDAQ.git myTDAQ
Cloning into 'TDAQ'...
Warning: Permanently added the ECDSA host key for IP address '[188.184.66.88]:7999' to the list of known hosts.
remote: Counting objects: 1702, done.
remote: Compressing objects: 100% (507/507), done.
remote: Total 1702 (delta 309), reused 0 (delta 0)
Receiving objects: 100% (1702/1702), 2.67 MiB | 0 bytes/s, done.
Resolving deltas: 100% (931/931), done.

BL4S $> cd myTDAQ/

myTDAQ $> ls
build.sh  Configs  daq_config  event_dump.sh  RCDTDAQ  RCDTDAQ_settings.sh.example  README.md  setup_RCDTDAQ.sh  start_Display.sh  start_Histogramming.sh  start_Igui.sh  start_ohp.sh  toolFunctions.sh
Start a new branch
myTDAQ $> git branch
* master
myTDAQ $> git co -b myNewBranch
Switched to a new branch 'myNewBranch'
myTDAQ $> git branch
  master
* myNewBranch

Adding/(re)moving/editing files

myTDAQ $> vi myNewFile.sh
myTDAQ $> cat myNewFile.sh 
#!/bin/bash

echo "I'm here!"

myTDAQ $> vi README.md #Some lines added, some deleted
Files should be (re)moved via git rm or git mv. In the latter case, git keeps the correct reference to the file. If a file was removed with the shell "rm" command. Just run "git rm" or "git add" (intuitive, isn't it?) on the now non existing path to make the change known.
myTDAQ $> git rm event_dump.sh 
rm 'event_dump.sh'
Inspect current status of repository
myTDAQ $> git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        deleted:    event_dump.sh

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   README.md

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        myNewFile.sh
List changes between the index and the working tree.
myTDAQ $> git diff
diff --git a/README.md b/README.md
index c049bed..05a68ea 100644
--- a/README.md
+++ b/README.md
@@ -4,6 +4,7 @@ This repository contains the BL4S flavour of the RCDTDAQ readout software:
 * TDAQ plugins for the CAEN VME modules and MicroMegas readout
 * Monitoring programmes
 * Configuration files
+* ...and now is part of the git tutorial
...
Add changes to the staging area. Start with the modified/deleted files.
myTDAQ $> git add -p
diff --git a/README.md b/README.md
index c049bed..05a68ea 100644
--- a/README.md
+++ b/README.md
@@ -4,6 +4,7 @@ This repository contains the BL4S flavour of the RCDTDAQ readout software:
 * TDAQ plugins for the CAEN VME modules and MicroMegas readout
 * Monitoring programmes
 * Configuration files
+* ...and now is part of the git tutorial
 
 
 ### IMPORTANT: Please read the [README](https://gitlab.cern.ch/BL4S/Documents) file from the BL4S/Documents repository first </h1>
Stage this hunk [y,n,q,a,d,/,j,J,g,e,?]? 
The -p lists your changes chunk by chunk, so you can review your changes. ? will give you a description of the accepted commands. This will not add new files. For that use git add with a path.
myTDAQ $> git add myNewFile.sh
myTDAQ $> git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   README.md
        deleted:    event_dump.sh
        new file:   myNewFile.sh
Create a new commit.
myTDAQ $> git commit -m"replace event_dump and document
> 
> * Remove event_dump.sh and replace by myNewFile.sh
> * Add myNewFile to README.md and rephrase repo structure"
[myNewBranch 31ef6fd] replace event_dump and document
 3 files changed, 7 insertions(+), 12 deletions(-)
 delete mode 100755 event_dump.sh
 create mode 100644 myNewFile.sh
The commit message shall start with a short line and be descriptive. The first line can be followed by a blank one and more details. Be specific.

Now push the new branch to the server and set it to track the remote branch.

myTDAQ $> git push -u origin myNewBranch 
Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 554 bytes | 0 bytes/s, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: 
remote: To create a merge request for myNewBranch, visit:
remote:   https://gitlab.cern.ch/bristic/TDAQ/merge_requests/new?merge_request%5Bsource_branch%5D=myNewBranch
remote: 
To ssh://gitlab.cern.ch:7999/bristic/TDAQ.git
 * [new branch]      myNewBranch -> myNewBranch
Branch myNewBranch set up to track remote branch myNewBranch from origin.
Gitlab is so kind as to give us a link for a merge request. This is already a good time to start a WIP merge request (see below. TODO: ANCHOR)

From here you can go to editing/adding/removing/moving files and commit and push your changes to the server.

Example on rebasing: myNewBranch started at the master branch and I want to merge it back to master, however master advanced in the meantime. I can now merge the master into my branch, so master can fast-forward to that commit or I can rebase my branch on the new HEAD of the master, by which I obtain a linear history. I decide for the latter:

myTDAQ $> git fetch
remote: Counting objects: 3, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From ssh://gitlab.cern.ch:7999/bristic/TDAQ
   f90faa4..827f5ec  master     -> origin/master
   
myTDAQ $> git branch
  master
* myNewBranch

myTDAQ $> git log --oneline --graph --decorate
* 8ef5c87 (HEAD, myNewBranch, origin/myNewBranch) remove toolFunctions
* 31ef6fd replace event_dump and document
*   f90faa4 (master) Merge branch 'bane-setup_scripts' into 'master'

myTDAQ $> git rebase origin/master
First, rewinding head to replay your work on top of it...
Applying: replace event_dump and document
Applying: remove toolFunctions
myTDAQ $>

myTDAQ $> git log --oneline --graph --decorate
* 4e87c6e (HEAD -> myNewBranch) remove toolFunctions
* c12db8f replace event_dump and document
* 827f5ec (origin/master, origin/HEAD) Update build.sh
*   f90faa4 (master) Merge branch 'bane-setup_scripts' into 'master'
Now myNewBranch starts at the current master HEAD. Please note that the checksums of the two commits have changed and origin/myNewBranch is missing in the list of branches for the first commit, meaning that despite introducing the same changes and having the same comment as before, these are now different commits in terms of git. This leads to an error when the branch is pushed.

myTDAQ $> git push
To ssh://gitlab.cern.ch:7999/bristic/TDAQ.git
 ! [rejected]        myNewBranch -> myNewBranch (non-fast-forward)
error: failed to push some refs to 'ssh://git@gitlab.cern.ch:7999/bristic/TDAQ.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
It is now completely out of sync with the remote branch, but you can overwrite the remote branch with the local one.
Needless to say, be VERY careful when using this command as you can delete whole branches. Use it only on your branches and explicitly state of the pushed branch.
myTDAQ $> git push -f origin myNewBranch
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 745 bytes | 0 bytes/s, done.
Total 6 (delta 3), reused 0 (delta 0)
remote: 
remote: To create a merge request for myNewBranch, visit:
remote:   https://gitlab.cern.ch/bristic/TDAQ/merge_requests/new?merge_request%5Bsource_branch%5D=myNewBranch
remote: 
To ssh://gitlab.cern.ch:7999/bristic/TDAQ.git
 + 31ef6fd...4e87c6e myNewBranch -> myNewBranch (forced update)
myTDAQ $> 
<!--/twistyPlugin-->


A brief introduction to Gitlab

Gitlab is a powerful collaborative coding platform based hosted on CERN servers. It is structured in projects, which hold one git repository and can be member of a group. It adds to the git internal tools for collaboration, a web-based layer with README displays, an issue tracker, merge requests, review processes, continuous integration,... We will mainly use the issue tracker and merge requests which will be briefly described here.

More complete info can be found in the Gitlab Basics and the Gitlab Academy.

Issue tracker

The issue tracker should be the central point to collect reasons for changing the code, from small bugs, over big bugs and new features to suggestions about rewriting everything. It is kind of a ToDo and discussion list for the code. Please use it as such. To open an issue:
  • Click on Issues in the project view of the web interface.
  • Add a nice title and a detailed description. You can (and should) use Markdown for the description.
  • Mention people you want to discuss directly with (e.g. @bStroustrup) in the description text. That way they'll get notified per mail.
  • Add appropriate labels and if possible assign the issue to somebody.
After submitting, your issue will get a number by which you can refer to it in any type of text (even commit messages) in the web interface or git repository. Syntax #NUMBER (e.g. #2). Gitlab will automatically link the reference to the Issue.

Merge requests

Merge requests are a fairly new idea for contributing code and constant review. In the BL4S TDAQ repo for instance, it is not allowed to push commits directly to the master. Every contribution is tracked in a separate feature branch and once the maintainer of the branch wants to merge the code into the master branch, a merge request is open and assigned to another person. This person reviews the code and initiates the merging online. By this, all code is reviewed and tested by at least two persons.

A merge request can be started by clicking on Merge requests and chosing which branch to merge into which, or by visiting the link that git shows when pushing a branch to the server for the first time.

  • Add a nice title and a detailed description. If you want to start a request, but are not finished yet with the branch, begin the title with WIP:. Gitlab will then prohibit the request to be merged until you remove the token.
  • Choose the labels and if it's ready for merging, assign it to the person you want to review your code. This can be changed later.
  • Check the source/target branches and the commits listed below
  • Submit the request
  • Next to the accept button click on modify commit message and add a list of changes that are introduced by this merge request. This is our Changelog, so be detailed if needed. Especially state changes to the data or file format and execution arguments/flags.
  • Wait for comments. In the mean time you can push further commits to your branch or even rebase it if necessary. New commits will be automatically added to the merge request.
  • NOTE: You will notice, that you can accept the merge request by yourself which should obviously not be done. If you urgently need that piece of code, consider just checking out your branch and using that code locally while waiting.

When you have to review a merge request, you can checkout the branch locally and go through the code or get a diff of all files in the request by clicking on Changes in the web interface. The latter has the advantage, that you can directly comment on code by clicking on the bubble next to the line number.

Also here, the merge request has an identifying number. Refer to the request in any text (including commit messages) by !NUMBER (eg. 4).

Troubleshooting/Q&A

  • The repository cannot be checked out by HTTPS (permission denied)
    You are running an old version of git (e.g. on lxplus, the SBC or other SLC6 machine). This version is not smart enough to ask for authentication itself. Try to explicitly add your user name to the url: https://YOUR_NICE_NAME@gitlab.cern.ch/....git
  • I have a branch that is under review, but I want to continue working with that code and commit without adding it to the merge request. What shall I do?
    Git allows for branches to start anywhere. Just create a new branch on top of your already existing one and continue to work on that one. If you want to keep both checked out and ready, you can clone the repository to another folder and have one branch constantly checked out. After merging, you can rebase your branch on the new master.

-- BranislavRistic - 2017-03-27

META FILEATTACHMENT attachment="comparison_merge_rebase.png" attr="" comment="Source: https://git-scm.com/book/en/v1/Git-Branching-Rebasing:" date="1490628593" name="comparison_merge_rebase.png" path="comparison_merge_rebase.png" size="45781" user="bristic" version="1"
META FILEATTACHMENT attachment="gitlab_flow.png" attr="" comment="Source: https://about.gitlab.com/2014/09/29/gitlab-flow/" date="1490628727" name="gitlab_flow.png" path="gitlab_flow.png" size="20600" user="bristic" version="1"
META FILEATTACHMENT attachment="git-operations.png" attr="" comment="Source: http://blog.podrezo.com/git-introduction-for-cvssvntfs-users/" date="1490628776" name="git-operations.png" path="git-operations.png" size="174672" user="bristic" version="1"
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback