BigPanDA Monitoring Developers

Introduction

Production environment configuration

The Code

Code style

  • Please follow PEP 8 Style Guide for Python Code

Code repository

GIT tips for BigPanDA monitoring development.

Preparing of the repository for work.

  • Go to workspace directory
    cd WORKSPACE
  • Clone repository from the central server. REPOSITORY_URL could be find at the corresponding GITHub page
    git clone REPOSITORY_URL # Example git@github.com:PanDAWMS/panda-bigmon-core.git or https://github.com/PanDAWMS/panda-bigmon-core.git 
  • Go to cloned repository root
    cd LOCAL_REPO
  • Check status and branch name
    git status
  • Prepare development environment, if for that some files were changed, but these changed must not go to commits, there is possibility to ignore local changes by adding them to
    .git/info/exclude
    file in repository root.
  • Status of repository must be clean, if so repository is ready for development
    git status

Simple commit activity.

Simple changes, like modification or adding of one file, may be done directly at the GITHub pages. Or by these steps without creation of separate branch.
  • Fetch last changes from the central repository
    git fetch origin
  • Checkout to target for commit branch
    git checkout TARGET
  • Check status and branch name
    git status
  • Make changes
  • Check status and review done changes
    git status
    git diff
  • Preparing changes for commit
    git add CHANGED_FILES # use . for all files
  • Commit changes to the local branch
    git commit -m 'MESSAGE'
  • Review history and recheck result tree
    git log
  • Push applied changes to the central repository
    git push origin TARGET

Feature development

If needed for development time is more then one working day or amount of changes is significant, it is better to create separate branch for development.
  • Fetch last changes from the central repository
    git fetch origin
  • Checkout to base branch
    git checkout BASE
  • Create separate branch and checkout there
    git branch FEATURE_BRANCH
    git checkout FEATURE_BRANCH
    or in one command
    git checkout -b FEATURE_BRANCH
  • Check status and branch name
    git status
  • Make and commit changes.
  • Then the development is finished, review history and recheck result tree
    git log
  • Push branch with new feature to the central repository
    git push origin FEATURE_BRANCH
  • Create pull request for merging or do manual merge
  • If feature branch is not needed anymore remove it at local and central repositories
    git branch -D FEATURE_BRANCH
    git push origin :FEATURE_BRANCH

Manual merge activity guideline.

Manual_merge_activity.png

BigPanDA monitoring. The model of development, integration and production.

Current state.

* Current state is not fully determined. Different repositories use differ models. But there is direct connection between personal branch and RPMs installed on production machines. * The current state alike:
Variant_2.png

Proposed Model:

Variant #1. Leak admin control, strong developer responsibility.

  • master is used for RPM build.
    • tags are created from the master, therefore you should under no circumstances push broken or intermediate-version of your code to the master.
  • Code development is performed in the local copy of repository in all branches and the results are pushed into central repository at GITHub.
  • The local copy of repository is under full user control and responsibility.
  • Any change to project's settings or urls or manage or wsgi files has to be broadcasted to every developer.
  • Each developer can create his/her own feature branch into the central repository.
    • Name of the branch follows naming convention:
      <username>_<feature name>[_v<version>]
      • is the developer's CERN login name, e.g. "jschovan",
      • is the code name for the feature developed in the branch, e.g. "jedimon",
      • v is optional version string for the branch, e.g. "v1".
    • in order that you always have the local repository, please do not create many branches in the central repository.
    • Each developer is responsible for cleanup of his/her branch area.
  • Each developer is responsible for merge of his/her code into the master branch.
  • Each developer is responsible for providing tests (unit, smoke, regression, integration, etc.) so that others can test that their new merge does not affect others before the merge is committed into repository.
    • When no tests are provided, at least a short description of functionality to be checked manually should be available.
    • Once tests are made available, tests need to pass before building a RPM, and after RPM is deployed to integration machine (before it is deployed to the production machine).
  • Installation of the RPM to the PanDA RPM repository rests with Jaroslava Schovancova and Sergey Belov.
  • Deployment to the production machine (bigpanda.cern.ch) rests with Jaroslava Schovancova and Sergey Belov.
  • The model of variant #1:
    Variant_2.png

Variant #2. Leak developer responsibility, strong admin control.

  • master is used for RPM build.
    • tags are created from the master
    • developers have no possibility to push code to the master, but they may create GITHub pull requests for meging devel to master or their own branch to devel (if they have issues).
    • only administrators have possibility to push code to the master, and they processing developers pull requests, with disscussion if needed.
  • devel is development branch. It is used as container for collecting changes, testing, and for next propagaion to the master
    • all developers have possibility to push code to the devel
  • Code development is performed in the local copy of repository in all branches and the results are pushed into central repository at GITHub.
  • The local copy of repository is under full user control and responsibility.
  • Any change to project's settings or urls or manage or wsgi files has to be broadcasted to every developer.
  • Each developer can create his/her own feature branch into the central repository.
    • Name of the branch follows naming convention:
      <username>_<feature name>[_v<version>]
      • is the developer's CERN login name, e.g. "jschovan",
      • is the code name for the feature developed in the branch, e.g. "jedimon",
      • v is optional version string for the branch, e.g. "v1".
    • in order that you always have the local repository, please do not create many branches in the central repository.
    • Each developer is responsible for cleanup of his/her branch area.
  • Each developer is responsible for merge of his/her code into the devel branch, if he/she do that manualy.
  • Each developer is responsible for providing tests (unit, smoke, regression, integration, etc.) so that others can test that their new merge does not affect others before the merge is committed into repository.
    • When no tests are provided, at least a short description of functionality to be checked manually should be available.
    • Once tests are made available, tests need to pass before building a RPM, and after RPM is deployed to integration machine (before it is deployed to the production machine).
  • Installation of the RPM to the PanDA RPM repository rests with Jaroslava Schovancova and Sergey Belov.
  • Deployment to the production machine (bigpanda.cern.ch) rests with Jaroslava Schovancova and Sergey Belov.
  • The model of variant #2:
    Variant_2.png

BigPanDAmon at a glance

Modules structure

  • BigPanDAmon application consists of 2 modules: bigpandamon-core and the VO/project-specific module, e.g. bigpandamon-lsst or bigpandamon-atlas.

bigpandamon-core structure

  • bigpandamon-core consists of several submodules providing core functionality for the bigpandamon application:
    • modules focused on presentation of a particular Model:
      • pandajob: presentation of Models derived from the PandaJob model, e.g. Jobsdefined4, Jobsactive4, Jobswaiting4, Jobsarchived4
      • resource: presentation of Models of the Schedconfig family
      • htcondor: presentation of HTCondor jobs
      • task: intended for presentation of tasks in the ProdSys2 context
    • modules providing functionality of a particular monitoring page element
      • common: content common to all applications: the core.common.settings module, static, templates, urls placeholder with imported urls of other core submodules
      • api: placeholder for REST API functionality for the BigPanDAmon package
      • graphic: intended as a placeholder for the plotting functionality
      • table: placeholder for tabular data presentation using jQuery plugin dataTables
  • bigpandamon-core should contain monitoring functionality from which may benefit multiple VOs/projects.
  • VO/project-specific modules import submodules of the core as standalone Django applications. Each of the submodules may have its own models, views, urls, templates, static files, tests, and other files.
  • show structure hide structure
    core
    +-- api
    |   +-- htcondorapi
    |   |   +-- __init__.py
    |   |   +-- serializers.py
    |   |   +-- status.py
    |   |   +-- tests.py
    |   |   +-- urls.py
    |   |   +-- utils.py
    |   |   +-- views.py
    |   +-- __init__.py
    |   +-- urls.py
    +-- common
    |   +-- config-templates
    |   |   +-- data
    |   |   |   +-- bigpandamon_settings
    |   |   |       +-- settings_bigpandamon.py-template
    |   |   +-- etc
    |   |       +-- httpd
    |   |           +-- conf.d
    |   |               +-- bigpandamon-apache.conf-template
    |   +-- __init__.py
    |   +-- media
    |   +-- models.py
    |   +-- settings
    |   |   +-- base.py
    |   |   +-- config.py
    |   |   +-- __init__.py
    |   |   +-- local.py-template
    |   +-- static
    |   |   +-- css
    |   |   |   +-- bigpandamon.css
    |   |   |   +-- bigpandamon-v1.css
    |   |   |   +-- foundation.css
    |   |   |   +-- jquery.dataTables_themeroller.css
    |   |   |   +-- jquery-ui-1.10.3.custom.min.css
    |   |   |   +-- jquery-ui-timepicker-addon.css
    |   |   |   +-- normalize.css
    |   |   +-- images
    |   |   |   +-- animated-overlay.gif
    |   |   |   +-- back_disabled.png
    |   |   |   +-- back_enabled_hover.png
    |   |   |   +-- back_enabled.png
    |   |   |   +-- details_close.png
    |   |   |   +-- details_open.png
    |   |   |   +-- favicon.ico
    |   |   |   +-- forward_disabled.png
    |   |   |   +-- forward_enabled_hover.png
    |   |   |   +-- forward_enabled.png
    |   |   |   +-- htcondor
    |   |   |   |   +-- ko.png
    |   |   |   |   +-- ok.png
    |   |   |   +-- sort_asc_disabled.png
    |   |   |   +-- sort_asc.png
    |   |   |   +-- sort_both.png
    |   |   |   +-- sort_desc_disabled.png
    |   |   |   +-- sort_desc.png
    |   |   |   +-- Sorting icons.psd
    |   |   |   +-- ui-bg_flat_0_aaaaaa_40x100.png
    |   |   |   +-- ui-bg_flat_55_fbec88_40x100.png
    |   |   |   +-- ui-bg_glass_75_d0e5f5_1x400.png
    |   |   |   +-- ui-bg_glass_85_dfeffc_1x400.png
    |   |   |   +-- ui-bg_glass_95_fef1ec_1x400.png
    |   |   |   +-- ui-bg_gloss-wave_55_5c9ccc_500x100.png
    |   |   |   +-- ui-bg_inset-hard_100_f5f8f9_1x100.png
    |   |   |   +-- ui-bg_inset-hard_100_fcfdfd_1x100.png
    |   |   |   +-- ui-icons_217bc0_256x240.png
    |   |   |   +-- ui-icons_2e83ff_256x240.png
    |   |   |   +-- ui-icons_469bdd_256x240.png
    |   |   |   +-- ui-icons_6da8d5_256x240.png
    |   |   |   +-- ui-icons_cd0a0a_256x240.png
    |   |   |   +-- ui-icons_d8e7f3_256x240.png
    |   |   |   +-- ui-icons_f9bd01_256x240.png
    |   |   +-- js
    |   |       +-- bigpandamon-v1.js
    |   |       +-- custom.js
    |   |       +-- custom.modernizr.js
    |   |       +-- foundation.js
    |   |       +-- jquery
    |   |       |   +-- jquery-2.0.2.min.js
    |   |       +-- jquery-1.9.1.min.js
    |   |       +-- jquery.cookie.js
    |   |       +-- jquery.dataTables.js
    |   |       +-- jquery.dataTables.min.js
    |   |       +-- jquery-ui-1.10.3.custom.min.js
    |   |       +-- jquery-ui-timepicker-addon.js
    |   |       +-- testhtcondor.js
    |   |       +-- vendor
    |   |       |   +-- custom.modernizr.js
    |   |       |   +-- jquery.js
    |   |       |   +-- zepto.js
    |   |       +-- zepto.js
    |   +-- templates
    |   |   +-- 404.html
    |   |   +-- 500.html
    |   |   +-- _base_bigpandamon.html
    |   |   +-- _base_grid.html
    |   |   +-- _base_prodsys2.html
    |   |   +-- common
    |   |   |   +-- _index_grid.html
    |   |   |   +-- testing.html
    |   |   +-- htcondor
    |   |   +-- templatetags
    |   |   |   +-- empty_div.html
    |   |   |   +-- joblist_dict.html
    |   |   |   +-- top_menu_item.html
    |   |   +-- topology
    |   +-- templatetags
    |   |   +-- core_extras.py
    |   |   +-- __init__.py
    |   +-- tests.py
    |   +-- urls.py
    |   +-- utils.py
    |   +-- views.py
    +-- doc
    |   +-- template_blocks.txt
    

bigpandamon-core settings

  • To be continued...
  • settings location: core.common.settings
  • core.common.settings consist of 3 files:
    • base.py: general project configuration, should not change this one
    • config.py: machine-specific environment config with sensible defaults
    • local.py: settings to override machine-specific environment config (e.g. DB configuration). Do not commit this file to SVN!
  • core.common.settings should contain self-consistent values of the Django settings variables. These variables can be overridden in the VO/project-specific package settings. You should not need to touch core.common.settings in order to develop VO/project specific monitoring.

VO/project-specific package structure

  • Intentions with the VO/project-specific package
    • Contains project-specific settings (e.g. DB connection, location of static files, templates which extend core submodules' templates).
    • May contain functionality to present Models which inherit from core's Models.
      • A new functionality from which more VOs/projects would benefit should be carefully merged into the core.

bigpandamon-lsst structure

  • To be continued...
  • show structure hide structure
    lsst
    +-- config-templates
    | +-- data
    | | +-- bigpandamon_settings
    | |     +-- settings_bigpandamon_lsst.py-template
    | +-- etc
    |     +-- httpd
    |         +-- conf.d
    |             +-- wsgi-bigpandamon-lsst.conf-template
    +-- __init__.py
    +-- manage.py
    +-- media
    +-- settings
    | +-- base.py
    | +-- config.py
    | +-- __init__.py
    | +-- local.py-template
    +-- static
    | +-- css
    | +-- img
    | +-- js
    +-- templates
    +-- urls.py
    +-- wsgi.py
    

VO/project-specific package settings

  • To be continued...
  • example of VO/project-specific settings location: lsst.settings
  • <project>.settings override core.common.settings
    • Please note the import order in the <project>.settings modul when updating the <project>.settings:
      ### settings.base
      from core.common.settings.base import *
      from <project>.settings.base import *
      ### settings.config
      from core.common.settings.config import *
      from <project>.settings.config import *
      ### settings.local
      from core.common.settings.local import *
      from <project>.settings.local import *
      

Template tags

  • To be continued...

ATLAS/ProdSys2 development

References

Django framework






Major updates:

-- JaroslavaSchovancova - 16 Jan 2014
-- StavroGayazov - 29 Aug 2014





Responsible: JaroslavaSchovancova

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng Manual_merge_activity.png r3 r2 r1 manage 209.6 K 2014-08-28 - 16:42 StavroGayazov  
PNGpng Now.png r1 manage 220.8 K 2014-08-28 - 17:01 StavroGayazov  
PNGpng Variant_1.png r3 r2 r1 manage 226.0 K 2014-08-28 - 16:44 StavroGayazov  
PNGpng Variant_2.png r3 r2 r1 manage 257.6 K 2014-08-28 - 16:42 StavroGayazov  
Edit | Attach | Watch | Print version | History: r10 < r9 < r8 < r7 < r6 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r10 - 2014-08-29 - StavroGayazov
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback