TWiki
>
LHCb Web
>
LHCbComputing
>
Git4LHCb
(2021-01-19,
MarcoClemencic
)
(raw view)
E
dit
A
ttach
P
DF
---+ *OBSOLETE* This page has been superseded by: https://lhcb-core-doc.web.cern.ch/lhcb-core-doc/Development.html#use-of-git-in-lhcb %TOC% ---+ Introduction The rationale and the basics of the use of Git in LHCb have been presented and discussed in a few places: * [[https://indico.cern.ch/event/442258/session/0/contribution/37/attachments/1216493/1776676/git_migration.pdf][62nd Analysis and Software Week (2016-01-25)]] * [[https://indico.cern.ch/event/474336/attachments/1208192/1761361/git4lhcb.pdf][Core Software Meeting (2016-01-06)]] * [[https://cds.cern.ch/record/2119399][LHCb-INT-2016-001]] This page is meant to summarize the main commands to work with Git in LHCb. Some complementary information can be found in the [[https://lhcb.github.io/second-analysis-steps/02-lb-git.html][LHCb Starterkit]]. This page can also be used to document [[#TipsNTricks][tips and tricks]], and [[#TroubleshootingInstructions][toubleshooting instructions]]. ---++ About Git Git is a distributed version control system widely used. See http://git-scm.com/ for more details. There are a lot of resources available on the web, for example: * [[http://www.ibm.com/developerworks/linux/library/l-git-subversion-1/][Git for Subversion users, Part 1: Getting started]] * [[http://www.ibm.com/developerworks/linux/library/l-git-subversion-2/][Git for Subversion users, Part 2: Taking control]] * [[http://git-scm.com/book/en/][Git Book]] * cheatsheets (PDF) * [[http://www.cheat-sheets.org/saved-copy/git-cheat-sheet.pdf][www.cheat-sheets.org]] * [[https://training.github.com/kit/downloads/github-git-cheat-sheet.pdf][GitHub]] * [[https://ja.atlassian.com/dms/wac/images/landing/git/atlassian_git_cheatsheet.pdf][Atlassian]] * [[https://www.google.com/search?q=git+tutorial][Git tutorial]] (Google search) CERN opted for [[https://about.gitlab.com/][GitLab]] as a Git hosting platform, at [[https://gitlab.cern.ch]], where you can find documentation about [[https://gitlab.cern.ch/help/gitlab-basics/README.md][GitLab basics]]. To simplify Git use in LHCb, a few [[#LHCbGit][custom Git sub-commands]] have been developed. ---+ Prerequisites There's a few operations that must be done once (and only once) before start working with Git at CERN. 1. [[https://gitlab.cern.ch/help/gitlab-basics/start-using-git.md][Git command line setup]], n.b. for LHCb users an account has likely been created for you on [[https://gitlab.cern.ch]]. Go to [[https://gitlab.cern.ch]] and see if your CERN single sign-on properly logs you on, then click on "Profile Settings" on the left toolbar and check the account details. 2. Set Git default _push_ policy * git < 1.7.11 (SLC6, lxplus):%BR% =git config --global push.default current= * git >= 1.7.11 (!CentOS7, Ubuntu, Mac, ...):%BR% =git config --global push.default simple= 3. (only needed for "ssh authentication") [[https://gitlab.cern.ch/help/gitlab-basics/create-your-ssh-keys.md][Add SSH keys to GitLab]], usually !GitLab asks you to do so if it was not done yet (you can upload the key under "Profile Settings" on [[https://gitlab.cern.ch]]). 4. (only needed for "ssh authentication") =git config --global lb-use.protocol ssh= 5. (only needed for "https authentication") [[https://docs.gitlab.com/ee/integration/kerberos.html#http-basic-access-denied-when-cloning][fix https authentication in newer git versions]] [[https://gitlab.com/gitlab-org/gitlab-ee/issues/2457][see here for more details]] * for work on lxplus7 (for the time being):%BR% =git config --global http.emptyAuth true= Note that step 3 and 4 are not mandatory, because you can use Kerberos based authentication. The ssh protocol has some advantages over https (used for Kerberos) [[http://stackoverflow.com/questions/7489813/github-push-error-rpc-failed-result-22-http-code-413][e.g. "large" transfers may fail over https]]. On the other hand, the ssh port is blocked in some networks (many hotels, some institutes) while the https port is usually open. Point 2 is to avoid mistakes and to get a behavior similar to that of Git 2.0 on old versions of Git (current version at 2016-04-03 is 2.8.1). #LHCbGit ---+ Using Git for LHCb development #SatelliteProjects ---++ Satellite projects Working with _satellite projects_ (AKA _local projects_) is very useful for quick (a few commits) or limited (a few packages) developments. First we need to create the satellite project %CODE{"bash"}% lb-dev Project/vXrY cd ProjectDev_vXrY %ENDCODE% then declare which project we want to get packages from %CODE{"bash"}% git lb-use Project %ENDCODE% and get the package/subdirectory code, from the _master_ (or another) branch %CODE{"bash"}% git lb-checkout Project/master Some/Package %ENDCODE% or from a given project version %CODE{"bash"}% git lb-checkout Project/vXrY Some/Package %ENDCODE% At this point, we can work on the changes we need to make, for example %CODE{"bash"}% vim Some/Package/src/MyStuff.cpp make make test git add Some/Package/src/MyStuff.cpp git commit -m 'fixing feature abc (JIRATICKET-123)' vim Some/Package/src/MyStuff.cpp make make test git commit -a -m 'improved fix to JIRATICKET-123' %ENDCODE% Once we are happy with our changes, we can push them to a branch (JIRATICKET-123 in this example) in the remote repository for the project %CODE{"bash"}% git lb-push Project JIRATICKET-123 %ENDCODE% and create a merge request, by going to the project page on https://gitlab.cern.ch/ and clicking on %ICON{plus}% then "New merge request". If the changes you're proposing are not ready for merging, you can add 'WIP:' at the beginning of the merge request's title, and can then discuss ideas in the MR. ---+++ SetupProject workaround If for some reason (which you don't have time to debug right now) you cannot use lb-dev, you can also use SetupProject %CODE{"bash"}% SetupProject --build-env PROJECT vXrY git init %ENDCODE% and then continue as above %CODE{"bash"}% git lb-use Project %ENDCODE% etc. ---+++ Mixing branches In git, one does not just commit the current content of a file, but a change history. That means whenever you commit work, git needs to know which older version of a branch your change should be applied to. When checking out files from multiple branches with git lb-checkout, this will lead to confusion (i.e. error messages about merge conflicts when doing git lb-push). In this case a combination of vanilla git and git-lb commands might be useful. Example: You want to commit changes on top of the 2018-patches branch of a project, but pick the master version of a few files for that. You want to use the nightly that was compiled last Monday. The steps then look like: %CODE{"bash"}% lb-dev --nightly=lhcb-2018-patches Mon DaVinci/2018-patches cd ./DaVinciDev_2018-patches git lb-use Phys # checkout Phys/LoKiPhys with git lb-checkout git lb-checkout Phys/2018-patches Phys/LoKiPhys # checkout three files from the master branch with vanilla git git checkout Phys/master -- Phys/LoKiPhys/python/LoKiPhys/functions.py git checkout Phys/master -- Phys/LoKiPhys/LoKi/BeamLineFunctions.h git checkout Phys/master -- Phys/LoKiPhys/src/BeamLineFunctions.cpp # edit Phys/LoKiPhys/python/LoKiPhys/functions.py to resolve small bug fix git add Phys/LoKiPhys/python/LoKiPhys/functions.py git commit git lb-push Phys My_awesome_feature %ENDCODE% The git lb-push command then should not fail in merge conflicts and is suited for a merge request into the 2018-patches branch (i.e. the branch used in the git lb-checkout command). #WholeProjects ---++ Whole projects The tools to work with satellite projects have limitations. In particular they fail on binary files and do not support branch juggling. For this reason and because working with plain git is better, it's generally suggested to work on whole projects with a workflow like: %CODE{"bash"}% git clone --recurse-submodules ssh://git@gitlab.cern.ch:7999/lhcb/Project.git cd Project lb-project-init # use '-b' only if that branch does not exist in the remote repository git checkout -b JIRATICKET-123 make make test # edit edit edit... git add Some/Package/src/MyStuff.cpp git commit -m 'fixing feature abc (JIRATICKET-123)' vim Some/Package/src/MyStuff.cpp make make test git commit -a -m 'improved fix to JIRATICKET-123' git push -u origin JIRATICKET-123 %ENDCODE% #DataPackages ---++ Data Packages Since December, 12 2017 data packages are in managed in Gitlab (see [[https://groups.cern.ch/group/lhcb-core-soft/Lists/Archive/Flat.aspx?RootFolder=%2Fgroup%2Flhcb%2Dcore%2Dsoft%2FLists%2FArchive%2FMigration%20of%20data%20packages%20to%20Git&FolderCTID=0x012002002DAD3147837DAE4A9BC6BABCB65AB725][announcement on lhcb-core-soft]]). To work on a data package is equivalent, in some sense, to working with full projects. You can get a local clone of a data package with something like =git clone https://gitlab.cern.ch/lhcb-datapkg/Hat/Name.git Hat/Name=, or, as a handy shortcut, =git lb-clone-pkg Hat/Name=. So, for example: %CODE{"bash"}% git lb-clone-pkg Gen/DecFiles cd Gen/DecFiles git checkout -b ${USER}/my-changes # edit edit edit git add dkfiles/YourDecFile.dec git add doc/release.notes git commit -m "Added dkfiles for XXX" git push -u origin ${USER}/my-changes # create merge request %ENDCODE% If you have a full stack checked out, you can do the following: %CODE{"bash"}% # from your stack directory mkdir DBASE cd DBASE git lb-clone-pkg PRConfig cd .. make Brunel/purge make %ENDCODE% The purge is annoying but necessary. Do not purge the whole stack, just the folder with the application (Brunel, Moore, DaVinci). Then you avoid a costly recompilation. ---+ Examples To better understand how to apply the recipes, here we have a few specific examples. ---++ Committing Hlt2 lines Start from the latest Moore version with lb-dev. %CODE{"bash"}% lb-dev Moore/v<latest> cd MooreDev_v<latest> git lb-use Hlt git lb-checkout Hlt/master Hlt/Hlt2Lines %ENDCODE% Make your changes. Add a line describing your change in doc/release.notes %CODE{"bash"}% git diff git status %ENDCODE% to review changes and see which files need to be added to your commit %CODE{"bash"}% git add <file1> <file2>... git commit -a -m "Describe your changes here, first line short summary, second line can be more comprehensive" git lb-push Hlt <your name>/<something describing your change> %ENDCODE% Go to gitlab.cern.ch, search for the project LHCb/Hlt ([[https://gitlab.cern.ch/lhcb/Hlt][https://gitlab.cern.ch/lhcb/Hlt]]). Create a merge request with the master branch. #UpgradeTrackingExample ---++ Upgrade Tracking *Use case*: work on a few packages from different projects in a specific nightly build slot. Note: The branch _upgradeTracking_ was merged to _master_ and deleted. Therefore, from now on it is recommended to use the _master_ branch. Development: %CODE{"bash"}% #Create environment lb-dev --nightly lhcb-head Brunel/HEAD #for a specific nightly add Mon,Tue,Wed,Thu,Fri,Sat,Sun as i.e.: lb-dev --nightly lhcb-head Mon Brunel/HEAD cd BrunelDev_HEAD # Add packages from Rec that you want to modify. e.g.: git lb-use Rec git lb-checkout Rec/master Tr/TrackFitter git lb-checkout Rec/master Tf/TrackSys #contains properties of the tracking sequence git lb-checkout Rec/master Pr/PrAlgorithms #contains tracking algorithm ( Seeding/Forward/Matching/Downstream ) git lb-checkout Rec/master Pr/PrMCTools #contains truth matching tool and tools to study detector properties git lb-checkout Rec/master Pr/PrKernel #contains base classes and interfaces used by PrAlgorightms and PrMCTools # Add Brunel for testing: git lb-use Brunel git lb-checkout Brunel/master Rec/Brunel # add/change files #Compile make #locally commit the code git commit -a -m "changed something" #Test the changes #run all qm tests make test #or, for specific tests: make test ARGS="-R brunel-upgrade-baseline" #push to server git lb-push Rec upgrade-newbranchname %ENDCODE% Merge requests * go to https://gitlab.cern.ch/lhcb/Rec/merge_requests/new (from https://gitlab.cern.ch/lhcb/Rec click on %ICON{plus}%, then on "New merge request"). * select _my-new-feature_ as source branch and _master_ as destination branch, and click on "Compare branches and continue" * fill in the form with details and submit * repeat for all projects (Brunel) Update local packages *after* the merge requests are applied %CODE{"bash"}% cd BrunelDev_master git fetch --all git lb-checkout Rec/master Tr/TrackFitter git lb-checkout Brunel/master Rec/Brunel %ENDCODE% #UpgradeTrackingExample2 ---++ Upgrade Tracking Example 2 *Use case*: work on significant changes to the tracking code, building packages from scratch Preparation (can be setup as a shell script for easy rebuilds): %CODE{"bash"}% source /cvmfs/lhcb.cern.ch/lib/LbEnv-stable # Only needed for non-lxplus systems #lb-set-platform ${CMTCONFIG/opt/dbg} # Uncomment for debug builds export CMAKE_PREFIX_PATH="$HOME/upgradeTracking:/cvmfs/lhcbdev.cern.ch/nightlies/lhcb-upgradeTracking/Today:${CMAKE_PREFIX_PATH}" export CCACHE_DIR=/<somewhere>/.ccache export CMAKEFLAGS=-DCMAKE_USE_CCACHE=ON unset VERBOSE %ENDCODE% Note that for best performance you should pick =/<somewhere>= local to the machine you run on. Also, you should increase the cache size a bit, as the default is a little small. For instance the following will set the size to 10G. %CODE{"bash"}% ccache --max-size=10G %ENDCODE% Building (using ssh for gitlab authentication): %CODE{"bash"}% mkdir ~/upgradeTracking cd ~/upgradeTracking git clone --recurse-submodules ssh://git@gitlab.cern.ch:7999/lhcb/LHCb.git LHCB/LHCB_upgradeTracking cd LHCB/LHCB_upgradeTracking git checkout -b upgradeTracking origin/upgradeTracking cd ~/upgradeTracking git clone --recurse-submodules ssh://git@gitlab.cern.ch:7999/lhcb/Lbcom.git LBCOM/LBCOM_upgradeTracking cd LBCOM/LBCOM_upgradeTracking git checkout -b upgradeTracking origin/upgradeTracking cd ~/upgradeTracking git clone --recurse-submodules ssh://git@gitlab.cern.ch:7999/lhcb/Rec.git REC/REC_upgradeTracking cd REC/REC_upgradeTracking git checkout -b upgradeTracking origin/raaij-upgradeTracking cd ~/upgradeTracking git clone --recurse-submodules ssh://git@gitlab.cern.ch:7999/lhcb/Brunel.git BRUNEL/BRUNEL_upgradeTracking cd BRUNEL/BRUNEL_upgradeTracking git checkout -b upgradeTracking origin/upgradeTracking %ENDCODE% Edit each of the CMake lists in the last three packages to use the =upgradeTracking= version instead of the versions listed for the above packages. In each directory, in order, run the following commands to build: %CODE{"bash"}% lb-project-init make configure make install %ENDCODE% Note: if you want or need to use prerelease versions of heptools you can edit the toolchain.cmake file created by the =lb-project-init= command with the additional lines at the top: %CODE{"bash"}% set(CMAKE_PREFIX_PATH /cvmfs/lhcb.cern.ch/lib/lhcb /cvmfs/lhcb.cern.ch/lib/lcg/releases /cvmfs/sft.cern.ch/lcg/releases ${CMAKE_PREFIX_PATH}) list(REMOVE_DUPLICATES CMAKE_PREFIX_PATH) %ENDCODE% which adds the location of the central heptools location as used by the nightly builds. ---++ Building everything locally example This example is a minor modification of the upgrade tracking example2. In this example the aim is to build everything, from =Gaudi= upwards, locally. The advantage of this approach is it decouples your development from the nightlies, the only dependencies taken from =/cvmfs= are the =LCG= releases. First, set up your environment in a very similar way :- %CODE{"bash"}% # Only needed for non-lxplus systems source /cvmfs/lhcb.cern.ch/lib/LbEnv-stable # If required, set CMTCONFIG to the platform you wish to build lb-set-platform <platform> # Define the sub-directory you wish to use for the project checkouts export User_release_area="/path/to/somewhere" lb-set-workspace ${User_release_area} export CMAKE_PREFIX_PATH=${User_release_area}:${CMAKE_PREFIX_PATH} # Use ccache export CCACHE_DIR=/<somewhere>/.ccache export CMAKEFLAGS="-DCMAKE_USE_CCACHE=ON" %ENDCODE% Then, starting with =Gaudi=, checkout and build each project in turn %CODE{"bash"}% cd $User_release_area git clone --recurse-submodules ssh://git@gitlab.cern.ch:7999/gaudi/Gaudi.git cd Gaudi git checkout <branch> lb-project-init make configure make install %ENDCODE% %CODE{"bash"}% cd $User_release_area git clone --recurse-submodules ssh://git@gitlab.cern.ch:7999/lhcb/LHCb.git cd LHCb git checkout <branch> lb-project-init make configure make install %ENDCODE% Repeat for whatever project you require, up to the top level application (=Brunel=, =DaVinci= etc.). Note in each step running =make configure= explicitly before =make install= is technically not required, but its useful to do the first time as it allows you to check the configuration before continuing. In particular check that each project is correctly using those it requires from your build area. Do not worry if, the first time you build a project it takes a long time. The likes of =LHCb= and =Rec= indeed take a short while to build at first. In subsequent builds though, you will find only the file you have touched, or those that use them, will need to be rebuilt each time. In addition, if you ever need to trigger a complete rebuild, using =ccache= will significantly improve the time those rebuilds take. #TipsNTricks ---+ Tips & Tricks ---++ Replacement for "svn update" in local projects In an =lb-dev= generated project, with packages checkout out with =git lb-checkout=, it is not possible to run =git merge= to get the local copy of the packages synchronized with remote changes while keeping the local ones. It is possible, though, to emulate the behaviour of =svn update= via a =git rebase= with the following procedure: %TWISTY{ mode="div" showlink="Show example of local project set up " hidelink="Hide example " showimgright="%ICONURLPATH{toggleopen-small}%" hideimgright="%ICONURLPATH{toggleclose-small}%" }% %CODE{"bash"}% lb-dev LHCb/v40r1 cd LHCbDev_v40r1 git lb-use -q LHCb # get an old version as an example git lb-checkout LHCb/v40r0~ GaudiObjDesc # ... some development ... edit edit edit commit commit commit %ENDCODE% %ENDTWISTY% %CODE{"bash"}% # update the package ( imported=$(git config -f .git-lb-checkout lb-checkout.LHCb.GaudiObjDesc.imported) git checkout $(git config -f .git-lb-checkout lb-checkout.LHCb.GaudiObjDesc.base) git lb-checkout $imported GaudiObjDesc ) git lb-checkout LHCb/master GaudiObjDesc git rebase HEAD master # resolve conflicts (if needed) following instructions from git-rebase %ENDCODE% ---++ Remove from a local project a package checked out with "git lb-checkout" If in your local project you have a copy of a package you do not need anymore, you can safely remove it from the local repository and remove the associated metadata with: %CODE{"bash"}% cd ~/cmtuser/MyProjectDev_vXrY git rm -r Hat/MyPackage git config -f .git-lb-checkout --remove-section lb-checkout.TheProject.Hat/MyPackage git add .git-lb-checkout git commit -m 'removed Hat/MyPackage' %ENDCODE% where you replace =TheProject= and =Hat/MyPackage= with the correct values (you can use =git config -l -f .git-lb-checkout= to be sure). ---++ Commit a change to both =master= and =run2-patches= branches =master= and =run2-patches= branches are distinct branches for run3 and run1/run2 development respectively. Both branches are in the process of being cleaned up to contain only code relevant for the supported run period, but some of the code is shared. When making a fix or adding a new feature, think whether it is relevant to both branches - if so you should commit it to both. While it is always possible to make distinct commits to the two branches, it is usually cleaner to make the commit to one branch and then cherry-pick it to the other, as this gives a more legible Git history and can avoid merge conflicts in future. Just make the merge request to one of the two branches and make a note in the MR description for the release manager to port to the other branch. Depending on the type of change it may be better to start from one branch rather than the other: for bug fixes, it is usually better to apply first to =run2-patches=; for new features, apply first to =master= *N.B. if your branch originated from anything other than the target branch, you must first "rebase" the branch, see instructions below* ---++ Commit a change to a legacy =xxx-patches= branch Legacy branches are intended for maintenance of software versions used in official processings (e.g. a =Reco= or =Stripping= version, but also the trigger for a given year). As such, their behaviour should not be modified, so any changes should be considered carefully: it is OK to add new features, but generally not OK to modify algorithms in ways that change their performance, even if it's a bug fix. When propagating a new feature to a (set of) legacy branch(es), it is best to start from the most recent branch (generally, =run2-patches=) and back port. This is best done by a release manager, just make a note in the MR description of the MR that introduces the change, which legacy branches it should be back-ported to. ---++ Rebase a feature branch to a different origin branch Sometimes you may have been working on a new feature or bug fix in a branch that you created starting from e.g. =master=, but you wish to make the merge request against another branch, e.g. =run2-patches=. Before you push your branch you need to rebase it. In the following example, we assume that you are working on a feature branch called =myFeatureBranch= in =Phys= project and wish to rebase it to =run2-patches= branch. Proceeed as follows: %CODE{"bash"}% cd $TMPDIR git clone --recurse-submodules ssh://git@gitlab.cern.ch:7999/lhcb/Phys.git cd Phys git checkout myFeatureBranch git rebase -i origin/run2-patches # an editor opens. You should keep only the lines with your commits and save git log # make sure the history looks right git push --force # much better to use --force-with-lease, but you might not have it in your git version # see e.g. https://developer.atlassian.com/blog/2015/04/force-with-lease/ %ENDCODE% ---++ Port an existing commit to another branch Sometimes you wish to commit a change to more than one branch. If you cannot rely on the automatic merging described in previous headings, use the =git cherry-pick= command: once your commit(s) is in a given branch, just checkout the other branch and issue %CODE{"bash"}% git cherry-pick sha1 [...] %ENDCODE% where sha1 is the name of your commit, or a space separated list of commits (in the order they should be applied). cherry-picking may trigger conflicts if the commit does not apply cleanly. In such a case, one has to resolve the conflicts, readd the fixed files to the index and say %CODE{"bash"}% git cherry-pick --continue %ENDCODE% as explained by git on its output Last remark : if you find somewhere that you could achieve something similar using git merge --ours or --theirs, this is a very bad idea ! These commands are explicitly revoking commits, so losing work. they actually mean "Keep our/their work, and revoke their/our work". This reverting will even be merged into other branch in the future. ---++ Move packages between projects *Warning*: please use this recipe with care! See [[https://its.cern.ch/jira/browse/LHCBPS-1696?focusedCommentId=1359240&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1359240][this JIRA comment]] for the details. ---++ Add a package to a project %CODE{"bash"}% git clone --recurse-submodules https://:@gitlab.cern.ch:8443/lhcb/TheProject.git cd TheProject git checkout -b ${USER}/MyNewPackage # copy the files git add Hat/MyNewPackage git commit -m 'add package Calibration/Pi0Calibration' git push -u origin ${USER}/MyNewPackage %ENDCODE% where you replace =TheProject= and =Hat/MyNewPackage= with the appropriate values ---++ Port changes from an SVN checkout to Git If the project you work on moved to Git, and you didn't have the chance to "svn commit" your changes before the write access to SVN was closed, this is how you can take your changes from your SVN checkout and port them to a Git clone. Let's imagine we were working on the package _Hat/MyPackage_ from the project _MyProject_, so probably you have a directory called =~/cmtuser/MyProjectDev_vXrY/Hat/MyPackage=. %CODE{"bash"}% # ensure we are up to date cd ~/cmtuser/MyProjectDev_vXrY svn update cd .. # rename the old checkout to leave place for a new one mv MyProjectDev_vXrY MyProjectDev_vXrY.svn # initialize the local project to work with Git lb-dev MyProject/vXrY cd MyProjectDev_vXrY git lb-use MyProject git lb-checkout MyProject/master Hat/MyPackage # import the changes ( cd ../MyProjectDev_vXrY.svn/Hat/MyPackage && svn diff ) | ( cd Hat/MyPackage && patch -p0 ) # commit the changes git add Hat/MyPackage git commit -m 'my changes' %ENDCODE% At this point you can continue your development as described [[#SatelliteProjects][above]]. *WARNING*: Please, ensure that all your changes have been ported before removing the old SVN checkout. ---++ Formatting code for LHCb As discussed at the [[https://indico.cern.ch/event/684077/contributions/3150443/attachments/1720765/2779105/Copyrights_and_coding_conventions_in_the_LHCb_code.pdf#1][11th LHCb Computing Workshop]] and detailed in [[https://indico.cern.ch/event/775350/contributions/3222646/attachments/1755747/2848463/Automatic_Formatting_of_LHCb_Source_Code.pdf][a Core Software Meeting]], contributions to LHCb code have to be formatted according to common LHCb style. To avoid issues with different interpretations of the style, the rules are applied using the automatic tools =clang-format= (for C++) and =YAPF= (for Python). A helper command (=lb-format=) is available to simplify the interaction with the low level tools. ---+++ Formatting individual files The simplest way to format a C++ or Python file is to call =lb-format= on them: %CODE{"bash"}% cd <Project> lb-format MyPkg/src/MySource.cpp MyPkg/python/MyPkg/SomePython.py %ENDCODE% =lb-format= ignores unsupported file types (printing a warning), so it's safe to call it with something like: %CODE{"bash"}% cd <Project> git ls-files | xargs -r lb-format find MyPkg -type f | xargs -r lb-format %ENDCODE% ---+++ Formatting only files modified wrt to a reference branch When working on large projects, running a no-op formatting on all files takes a lot of time, so it's useful to run it only on the files you are working on. Assuming you are working on a branch that you want to merge into =origin/master=, you can do something like: %CODE{"bash"}% cd <Project> lb-format --format-patch - origin/master | git am %ENDCODE% for files already committed, or %CODE{"bash"}% cd <Project> git diff --name-only --diff-filter=MA | xargs -r lb-format %ENDCODE% for files not committed yet. #TroubleshootingInstructions ---+ Troubleshooting ---++ Merge conflict in gitlab You have pushed your commits to gitlab into a branch, you try to merge with the master or some other branch, and gitlab tells you there is a merge conflict. The currently easiest way to see where the merge conflict is: %CODE{"bash"}% git clone --recurse-submodules ssh://git@gitlab.cern.ch:7999/lhcb/<Project>.git cd <Project> git checkout <your branch> git merge <target branch> <solve conflict> (Remove every >>>>> or <<<<<< ) git commit git push origin <your branch> %ENDCODE% Go back to gitlab and check if you can merge now. ---+++ Conflict after the global re-format of the project This is a pretty rare case... I'll fill the blank ASAP. ---++ Warning about push.default not being set When calling =git push= from a CentOS7 machine, you may get this warning: | <verbatim> warning: push.default is unset; its implicit value is changing in Git 2.0 from 'matching' to 'simple'. To squelch this message and maintain the current behavior after the default changes, use: git config --global push.default matching To squelch this message and adopt the new behavior now, use: git config --global push.default simple See 'git help config' and search for 'push.default' for further information. (the 'simple' mode was introduced in Git 1.7.11. Use the similar mode 'current' instead of 'simple' if you sometimes use older versions of Git) </verbatim> | To fix it you need to call =git config --global push.default xyz=, as described in [[#Prerequisites]]. ---++ Warning about not being able to push to the repository There are three ways to access the git repostory, for example for the Rec package they are: | *Access* | *Git URL* | | Kerberos | =https://:@gitlab.cern.ch:8443/lhcb/Rec.git= | | HTTPS | =https://gitlab.cern.ch/lhcb/Rec.git= | | ssh | =ssh://git@gitlab.cern.ch:7999/lhcb/Rec.git= | The trailing =.git= is not always necessary but push errors can occur due to the redirection from the url w/o =.git= to the complete one. Ensure the =.git= is present with =git remote -v=. A remote repository url can in anyway be changed with %CODE{"bash"}% git remote set-url Rec <new url> %ENDCODE% The command =git lb-use project= sets the Kerberos method, so anyone with access to their CERN AFS area should be able to push changes to gitlab.cern.ch. If you are working outside CERN the ssh access may be a better option, you must first register your ssh key with CERN (see https://cern.service-now.com/service-portal/article.do?n=KB0003136 for details). To either set or reset the remote site use the command %CODE{"bash"}% git remote -v %ENDCODE% to view the current remote repositories. Then %CODE{"bash"}% git remote set-url Rec ssh://git@gitlab.cern.ch:7999/lhcb/Rec.git %ENDCODE% to adjust a repository or %CODE{"bash"}% git remote add -f Brunel ssh://git@gitlab.cern.ch:7999/lhcb/Brunel.git %ENDCODE% to add a new repository. The official repository links can be found on the gitlab.cern.ch home page for each package. You can globally switch between ssh and Kerberos authentication with the following entry in your =~/.gitconfig= file %CODE{"bash"}% [url "ssh://git@gitlab.cern.ch:7999"] insteadOf = https://:@gitlab.cern.ch:8443 %ENDCODE% a downside is that with Kerberos, for public repositories, authentication is skipped for pull actions. With ssh one needs to unlock the ssh keyring for pushing and pulling. An alternative is to finetune %CODE{"bash"}% [url "ssh://git@gitlab.cern.ch:7999"] pushInsteadOf = https://:@gitlab.cern.ch:8443 %ENDCODE% which globally replaces all gitlab.cern.ch access from Kerberos to ssh for all push actions but leaves pull actions unchanged. ---++ Dealing with submodules when changing branches (incl. checkout, pull, merge, cherry-pick) We are using git submodules. The above =git clone --recurse-submodules= mostly takes care of that, except for changing branches with =git checkout <branch>= for branches that use different versions of a submodule. In this case, updating the submodule manually can be done with %CODE{"bash"}% git submodule update --recursive %ENDCODE% =git submodule update --recursive= can also be run when a project was cloned without the =--recurse-submodules= option. ---+ Known Issues * =git lb-push= does not handle correctly binary files * after a =git lb-push Project my-branch= you have to call =git lb-checkout Project/my-branch My/Package= to continue working on the same branch -- Main.MarcoClemencic - 2016-04-25, Main.MarcoCattaneo - 2017-07-10
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r78
<
r77
<
r76
<
r75
<
r74
|
B
acklinks
|
V
iew topic
|
WYSIWYG
|
M
ore topic actions
Topic revision: r78 - 2021-01-19
-
MarcoClemencic
Log In
LHCb
LHCb Web
LHCb Web Home
Changes
Index
Search
LHCb webs
LHCbComputing
LHCb FAQs
LHCbOnline
LHCbPhysics
LHCbVELO
LHCbST
LHCbOT
LHCbPlume
LHCbRICH
LHCbMuon
LHCbTrigger
LHCbDetectorAlignment
LHCbTechnicalCoordination
LHCbUpgrade
Public webs
Public webs
ABATBEA
ACPP
ADCgroup
AEGIS
AfricaMap
AgileInfrastructure
ALICE
AliceEbyE
AliceSPD
AliceSSD
AliceTOF
AliFemto
ALPHA
Altair
ArdaGrid
ASACUSA
AthenaFCalTBAna
Atlas
AtlasLBNL
AXIALPET
CAE
CALICE
CDS
CENF
CERNSearch
CLIC
Cloud
CloudServices
CMS
Controls
CTA
CvmFS
DB
DefaultWeb
DESgroup
DPHEP
DM-LHC
DSSGroup
EGEE
EgeePtf
ELFms
EMI
ETICS
FIOgroup
FlukaTeam
Frontier
Gaudi
GeneratorServices
GuidesInfo
HardwareLabs
HCC
HEPIX
ILCBDSColl
ILCTPC
IMWG
Inspire
IPv6
IT
ItCommTeam
ITCoord
ITdeptTechForum
ITDRP
ITGT
ITSDC
LAr
LCG
LCGAAWorkbook
Leade
LHCAccess
LHCAtHome
LHCb
LHCgas
LHCONE
LHCOPN
LinuxSupport
Main
Medipix
Messaging
MPGD
NA49
NA61
NA62
NTOF
Openlab
PDBService
Persistency
PESgroup
Plugins
PSAccess
PSBUpgrade
R2Eproject
RCTF
RD42
RFCond12
RFLowLevel
ROXIE
Sandbox
SocialActivities
SPI
SRMDev
SSM
Student
SuperComputing
Support
SwfCatalogue
TMVA
TOTEM
TWiki
UNOSAT
Virtualization
VOBox
WITCH
XTCA
Welcome Guest
Login
or
Register
Cern Search
TWiki Search
Google Search
LHCb
All webs
Copyright &© 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use
Discourse
or
Send feedback