Manual build of RPMs for Big PanDA

Introduction

Manual build of ATLAS BigPanDAmon RPMs

  • Let us assume that we need to build both bigpandamon-atlas and bigpandamon-core RPMs.

Manual build of bigpandamon-atlas

Automatic bigpandamon-atlas RPM build

Automatic bigpandamon-atlas RPM build - trunk

Automatic bigpandamon-atlas RPM build - branches/devel

OBSOLETE - Manual RPM build for bigpandamon-atlas

Update RPM configs in the devel branch

  • Check the latest version available, do not forget to increment version string.
  • cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas
    # cd panda-bigmon/trunk/bigpandamon-atlas
  • edit file branches/devel/bigpandamon-atlas/setup.cfg
    • update "requires" if necessary
      # svn diff setup.cfg
      Index: setup.cfg
      ===================================================================
      --- setup.cfg   (revision 18781)
      +++ setup.cfg   (working copy)
      @@ -4,5 +4,6 @@
       provides = bigpandamon-atlas
       release = 1
       packager = Panda Team <hn-atlas-panda-pathena@cern.ch>
      -requires = python bigpandamon-core
      +requires = python 
      +           bigpandamon-core >= 0.0.4
      
  • edit file branches/devel/bigpandamon-atlas/setup.py
    • update/check whatever is marked with #FIXME, i.e. release_version, expected_extensions, packages, data_files
  • commit changes to RPM configs
    # svn ci -m "bigpandamon-atlas/branches/devel: prepared RPM setup."
    

Merge devel branch to trunk

  • cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas
    # cd panda-bigmon/trunk/bigpandamon-atlas
    
  • update local copy of trunk of bigpandamon-atlas
    # svn up
    
  • merge branches/devel/bigpandamon-atlas branch to trunk/bigpandamon-atlas
    • postpone conflicts for setup.py and setup.cfg
      # svn merge --reintegrate svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/branches/devel/bigpandamon-atlas
      --- Merging differences between repository URLs into '.':
      U    atlas/prodtask/views.py
      U    atlas/prodtask/request_views.py
      U    atlas/prodtask/urls.py
      U    atlas/prodtask/forms.py
      U    atlas/prodtask/templates/prodtask/_index.html
      U    atlas/prodtask/templates/prodtask/_reqdatatable.html
      C    setup.cfg
      C    setup.py
      
    • resolve conflicts for setup.py and setup.cfg
      # svn resolve --accept theirs-full setup.cfg
      Resolved conflicted state of 'setup.cfg'
      # svn resolve --accept theirs-full setup.py
      Resolved conflicted state of 'setup.py'
      
  • now go through changes before committing the merge, and adapt what is necessary, resolve collisions etc.
  • commit the merge
    # svn ci -m "Merged devel/bigpandamon-atlas to trunk/bigpandamon-atlas, RC for bigpandamon-atlas-0.0.4"
    

Merge troubleshooting

  • If someone accidentally committed into trunk instead of branches/devel, you may get error message such as
    # svn merge --reintegrate svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/branches/devel/bigpandamon-atlas
    svn: Reintegrate can only be used if revisions 18773 through 19007 were previously merged from svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas to the reintegrate source, but this is not the case:
      panda-bigmon/branches/devel/bigpandamon-atlas/atlas
        Missing ranges: /panda-bigmon/trunk/bigpandamon-atlas/atlas:18773-18862
    
The resolution of this issue is described in http://stackoverflow.com/a/2648969:
    • go to trunk/bigpandamon-atlas and perform
      svn propdel svn:mergeinfo -R
      svn revert .
      svn ci -m "Removed mergeinfo"
      
    • go to branches/devel/bigpandamon-atlas and perform
      svn propdel svn:mergeinfo -R
      svn revert .
      svn ci -m "Removed mergeinfo"
      
    • go back to trunk/bigpandamon-atlas and restart the merge.

Create tag of bigpandamon-atlas-0.0.4

  • cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas
    # cd panda-bigmon/trunk/bigpandamon-atlas
    
  • create a new tag
    svn copy \
          svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas \
          svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/tags/bigpandamon-atlas/bigpandamon-atlas-0.0.4 \
          -m "bigpandamon-atlas-0.0.4"
    

Build RPM

  • login to aipanda004 as jschovan/sbelov
    # [lxvoadm]# ssh jschovan@aipanda004
    
  • cd to a workdir
    # MYWORKDIR="/data/jschovan/new-rpm/MYWORKDIR"
    # mkdir -p $MYWORKDIR
    # cd $MYWORKDIR
    
  • prepare .rpmmacros
    # echo "%_unpackaged_files_terminate_build 0" >> ~/.rpmmacros 
    
  • checkout the tagged code
    # svn co svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/tags/bigpandamon-atlas/bigpandamon-atlas-0.0.4
    
  • cd to the tagged directory
    # cd $MYWORKDIR/bigpandamon-atlas-0.0.4
    # ls $MYWORKDIR/bigpandamon-atlas-0.0.4
    atlas  INSTALL.txt  MANIFEST.in  README.txt  setup.cfg  setup.py
    
  • build RPM of bigpandamon-atlas-0.0.4
    # QA_SKIP_BUILD_ROOT=1 python setup.py bdist_rpm 
    ... lots of output, and then ...
    moving build/bdist.linux-x86_64/rpm/SRPMS/bigpandamon-atlas-0.0.4-1.src.rpm -> dist
    moving build/bdist.linux-x86_64/rpm/RPMS/noarch/bigpandamon-atlas-0.0.4-1.noarch.rpm -> dist
    
    # ls $MYWORKDIR/bigpandamon-atlas-0.0.4
    atlas  build  dist  INSTALL.txt  MANIFEST  MANIFEST.in  README.txt  setup.cfg  setup.py
    
    # ls $MYWORKDIR/bigpandamon-atlas-0.0.4/dist -1
    bigpandamon-atlas-0.0.4-1.noarch.rpm  
    bigpandamon-atlas-0.0.4-1.src.rpm  
    bigpandamon-atlas-0.0.4.tar.gz
    
  • check files in the noarch RPM
    • make sure that none of the python modules is missing (make sure every python module has init.py committed in SVN)
    • make sure that none of the templates files is missing
    • make sure that the static files are not missing
      # rpm -qpl dist/bigpandamon-atlas-0.0.4-1.noarch.rpm | less
      
  • remove .rpmmacros
    # rm ~/.rpmmacros
    

Copy RPMs to the repository

  • sudo su on aipanda004
  • cd to the dist directory
    # MYWORKDIR="/data/jschovan/new-rpm/MYWORKDIR"
    # cd $MYWORKDIR/bigpandamon-atlas-0.0.4/dist
    
  • cp src.rpm to /data/build/SRPMS
    # cp bigpandamon-atlas-0.0.4-1.src.rpm /data/build/SRPMS/
    
  • cp noarch.rpm to /data/build/x86_64
    # cp bigpandamon-atlas-0.0.4-1.noarch.rpm /data/build/x86_64
    

Manual build of bigpandamon-core for ATLAS

Update RPM configs in the devel branch

  • Check the latest version available, do not forget to increment version string.
  • cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-core
    # cd panda-bigmon/trunk/bigpandamon-core
  • edit file branches/devel/bigpandamon-core/setup.py
    • update/check whatever is marked with #FIXME, i.e. release_version, expected_extensions, packages, data_files
  • commit changes to RPM configs
    # svn ci -m "bigpandamon-core/branches/devel: prepared RPM setup."
    

Merge devel branch to trunk

  • cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-core
    # cd panda-bigmon/trunk/bigpandamon-core
    
  • update local copy of trunk of bigpandamon-core
    # svn up
    
  • merge branches/devel/bigpandamon-core branch to trunk/bigpandamon-core
    • postpone conflicts for setup.py and setup.cfg
      # svn merge --reintegrate svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/branches/devel/bigpandamon-core
      --- Merging differences between repository URLs into '.':
      ... updates ...
      C    setup.py
      
    • resolve conflicts for setup.py (and setup.cfg)
      # svn resolve --accept theirs-full setup.py
      Resolved conflicted state of 'setup.py'
      
  • now go through changes before committing the merge, and adapt what is necessary, resolve collisions etc.
  • commit the merge
    # svn ci -m "Merged devel/bigpandamon-core to trunk/bigpandamon-core, RC for bigpandamon-core-0.0.4"
    

Create tag of bigpandamon-core-0.0.4

  • cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-core
    # cd panda-bigmon/trunk/bigpandamon-core
    
  • create a new tag
    svn copy \
          svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-core \
          svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/tags/bigpandamon-core/bigpandamon-core-0.0.4 \
          -m "bigpandamon-core-0.0.4"
    

Build RPM

Automatic bigpandamon-core RPM build

Automatic bigpandamon-core RPM build - trunk
Automatic bigpandamon-core RPM build - branches/devel

OBSOLETE - Manual RPM build for bigpandamon-core

  • login to aipanda004 as jschovan/sbelov
    # [lxvoadm]# ssh jschovan@aipanda004
    
  • cd to a workdir
    # MYWORKDIR="/data/jschovan/new-rpm/MYWORKDIR"
    # mkdir -p $MYWORKDIR
    # cd $MYWORKDIR
    
  • prepare .rpmmacros
    # echo "%_unpackaged_files_terminate_build 0" >> ~/.rpmmacros 
    
  • checkout the tagged code
    # svn co svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/tags/bigpandamon-core/bigpandamon-core-0.0.4
    
  • cd to the tagged directory
    # cd $MYWORKDIR/bigpandamon-core-0.0.4
    # ls $MYWORKDIR/bigpandamon-core-0.0.4
    core  INSTALL.txt  MANIFEST.in  README.txt  setup.cfg  setup.py
    
  • build RPM of bigpandamon-core-0.0.4
    # QA_SKIP_BUILD_ROOT=1 python setup.py bdist_rpm 
    ... lots of output, and then ...
    moving build/bdist.linux-x86_64/rpm/SRPMS/bigpandamon-core-0.0.4-1.src.rpm -> dist
    moving build/bdist.linux-x86_64/rpm/RPMS/noarch/bigpandamon-core-0.0.4-1.noarch.rpm -> dist
    
    # ls $MYWORKDIR/bigpandamon-core-0.0.4
    core  build  dist  INSTALL.txt  MANIFEST  MANIFEST.in  README.txt  setup.cfg  setup.py
    
    # ls $MYWORKDIR/bigpandamon-core-0.0.4/dist -1
    bigpandamon-core-0.0.4-1.noarch.rpm  
    bigpandamon-core-0.0.4-1.src.rpm  
    bigpandamon-core-0.0.4.tar.gz
    
  • check files in the noarch RPM
    • make sure that none of the python modules is missing (make sure every python module has init.py committed in SVN)
    • make sure that none of the templates files is missing
    • make sure that the static files are not missing
      # rpm -qpl dist/bigpandamon-core-0.0.4-1.noarch.rpm | less
      
  • remove .rpmmacros
    # rm ~/.rpmmacros
    

Copy RPMs to the repository

  • sudo su on aipanda004
  • cd to the dist directory
    # MYWORKDIR="/data/jschovan/new-rpm/MYWORKDIR"
    # cd $MYWORKDIR/bigpandamon-core-0.0.4/dist
    
  • cp src.rpm to /data/build/SRPMS
    # cp bigpandamon-core-0.0.4-1.src.rpm /data/build/SRPMS/
    
  • cp noarch.rpm to /data/build/x86_64
    # cp bigpandamon-core-0.0.4-1.noarch.rpm /data/build/x86_64
    

Update RPM repos

  • sudo su on aipanda004 (the pandabuild.cern.ch machine)

  • cd to dir with repobuild scripts
    # cd /data/build-not-public
    
  • run the repobuild scripts
    # ./repo_SRPMS
    # ./repo_x86_64
    

Deployment to the integration/production machines

Deployment to the integration machine

  • sudo su on aipanda043 - integration
  • to be continued...

Deployment to the production machine

  • sudo su on aipanda022 - production
    # yum clean all
    # yum upgrade
    
  • make sure local.py is in place and is correct
    # /data/virtualenv/django1.6.1__python2.6.6__atlas/pythonpath/core/common/settings/local.py
    # /data/virtualenv/django1.6.1__python2.6.6__atlas/pythonpath/atlas/settings/local.py
    
  • activate the virtualenv
    • make sure "pip freeze" contains all the virtualenv dependencies
      # cd /data/virtualenv/django1.6.1__python2.6.6__atlas
      # source bin/activate
      # export PYTHONPATH=/data/virtualenv/django1.6.1__python2.6.6__atlas/pythonpath:$PYTHONPATH
      
  • try out ./manage.py runserver
    # cd /data/virtualenv/django1.6.1__python2.6.6__atlas/pythonpath/atlas
    # python manage.py runserver
    
  • run ./manage.py collectstatic
    # python manage.py collectstatic
    
  • restart the httpd service and change the owner of the logs files to apache:apache
    # service httpd restart
    # chown apache:apache -R /data/atlpan/bigpandamon/var/log/* /data/bigpandamon_virtualhosts/atlas/logs/* /data/bigpandamon_virtualhosts/jedimon/logs/*
    






Major updates:

-- JaroslavaSchovancova - 24 Apr 2014






Responsible: JaroslavaSchovancova

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2014-05-19 - JaroslavaSchovancova
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback