Information for PanDA Pilot developers

This wiki will explain how to get started with pilot development.

How to fork the pilot GitHub repository and work on local files

All pilot code should be kept in GitHub. A new developer can start by forking the PanDA Pilot repository, i.e. go to the Pilot 1 or Pilot 2 repository web pages and click on the 'fork' link on the top right and follow the instructions (you need to have a GitHub account).

Download the repository into your local development area, cd into it and checkout the main-dev (Pilot 1) or next (Pilot 2) branch:

git clone
cd pilot
git checkout main-dev


git clone
cd pilot2
git checkout next

How to add upstream

(Replace "pilot" with "pilot2" for Pilot 2)

git remote add upstream
git remote set-url --push upstream git@xxxxx:PanDAWMS/pilot.git

#git remote -v
origin  ssh:// (fetch)
origin  ssh:// (push)
upstream (fetch)
upstream        git@xxxxx:PanDAWMS/pilot.git (push)

How to update pilot

Edit the files you want to work on, commit every change (frequently), i.e.

git commit -a -m 'some comment..'

and when you are done with all changes, test the code before creating the pull request (instructions will be written at some point). Please make detailed comments for documentation so other people can understand what the update is about (in the pull request).

You will need to synchronize with upstream (i.e. the main repo) frequently, so you need to checkout upstream then rebase on it or merge with it.

You must not push to upstream. You should only push to your own forked repo then you can create pull request.

git push [-u] origin branch_name

How to create a GitHub pull request

After you have committed and tested your changes in your private GitHub fork, go to the main GitHub page in your private fork and click on 'New pull request'. Select the 'compare across forks' option (base fork should be PandaWMS/pilot[2] and head fork should point to your repository), write some comments and then click on 'Create pull request'.

This will send a message to the repository manager (currently Paul) who will review and approve the changes. In Pilot 2, the manager will issue the test suite (which runs pep8, flake8 and unit tests) before any changes will be approved.


How to sync my private fork with the original repository

Ok so you have been developing in your own fork for a while, but in the meantime the original repository will probably have changed a bit. Sync your private fork with the original repository by following these instructions. For pilot development, replace master with main-dev (Pilot 1) or next (Pilot 2) in the instructions.

Note: when you want to continue making local commits to your own fork, you need to set the following:

git push --set-upstream origin main-dev

which will push the current branch and set the remote as upstream (or you will get an error message if you just do a 'git push').

Cut-and-paste example (Pilot 1)

Starting with a fresh checkout of code from the private repository belonging to USERNAME.

git clone
cd pilot
git checkout main-dev
git remote add upstream
git fetch upstream
git merge upstream/main-dev

Cut-and-paste example (Pilot 2)

git clone
cd pilot2
git checkout next
git remote add upstream
git fetch upstream
git merge upstream/next

How to verify code before a pull request (Pilot 2)

For Pilot 2 we enforce PEP8 and Flake8 coding standards, A pull request will trigger an automated test which also runs unit tests. It can be practical to do these tests locally before making any pull request as it can take several iterations to get things right. For running these test suites, it is also practical to use virtualenv (available on lxplus - but some lxplus nodes have problems, in case of strange errors, just switch to another node) since you might not have root privileges to install additional software.

Cut-and-paste example of how to run test suites

cd [somewhere]
mkdir test
virtualenv test
cd test
source bin/activate
pip install 'unittest2'
pip install "flake8${FLAKE8_VERSION}"
pip install 'pep8-naming'
pip install 'flake8-blind-except'

cp -r [someplace]/pilot2 .
cd pilot2
flake8 --config .flake8 pilot/
pycodestyle --ignore E501 pilot/
unit2 -v

How to test pilot code

A few examples can be found in the RunPilot section.

How to send test jobs

(To be written)

-- PaulNilsson - 2016-09-07

Edit | Attach | Watch | Print version | History: r13 < r12 < r11 < r10 < r9 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r10 - 2018-02-20 - PaulNilsson
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback