BigPanDARPMbuildManual < PanDA

PanDA Web>AtlasDistributedComputing>PanDA>BigPanDARPMbuildSystem>BigPanDARPMbuildManual (2014-05-19, JaroslavaSchovancova) (raw view)
EditAttachPDF
%CERTIFY% 
%STARTINCLUDE%
---+!! Manual build of RPMs for Big !PanDA

%TOC%

---+ Introduction
   * This page is evolving. Please keep checking. 
   * This page contains information about manual build of RPMs for !BigPanDA. Keep checking [[BigPanDARPMbuildSystem]].
   * Feel free to contact Main.JaroslavaSchovancova and Main.TorreWenaus with questions. 

---+ Manual build of ATLAS BigPanDAmon RPMs
   * Let us assume that we need to build both =bigpandamon-atlas= and =bigpandamon-core= RPMs.

---++ Manual build of bigpandamon-atlas
---+++ Automatic bigpandamon-atlas RPM build
---++++ Automatic bigpandamon-atlas RPM build - trunk
   * Upon every commit to https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/trunk/bigpandamon-atlas an RPM is built.
   * Trunk RPM is available https://pandabuild.cern.ch/x86_64/ as <verbatim>
bigpandamon-atlas-<stable version>-1.noarch.rpm
</verbatim>
      * <verbatim><stable version></verbatim> is the most recent (current, new) stable version of the =bigpandamon-atlas= package
   * Naming convention steering for trunk: in [[https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/trunk/bigpandamon-atlas/setup.cfg][trunk/bigpandamon-atlas/setup.cfg]]
      * =version = VALUE=, where =VALUE= is the new stable version number
      * =release_type = stable=
   * Naming convention steering for devel: in [[https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/branches/devel/bigpandamon-atlas/setup.cfg][devel/bigpandamon-atlas/setup.cfg]]
      * =version = VALUE=, where =VALUE= is the newest stable version number
      * =release_type = dev=
---++++ Automatic bigpandamon-atlas RPM build - branches/devel
   * Upon every commit to https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/branches/devel/bigpandamon-atlas an RPM is built.
   * Devel RPM is available https://pandabuild.cern.ch/unstable/ as <verbatim>
bigpandamon-atlas-<stable version>.dev_r<svn revision of the commit>-1.noarch.rpm
</verbatim>
      * <verbatim><stable version></verbatim> is the most recent stable version of the =bigpandamon-atlas= package
      * <verbatim><svn revision of the commit></verbatim> is the revision number of the latest commit into =branches/devel= of =bigpandamon-atlas=.
   * Naming convention steering: in [[https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/branches/devel/bigpandamon-atlas/setup.cfg][devel/bigpandamon-atlas/setup.cfg]]
      * =version = VALUE=, where =VALUE= is the latest stable version number
      * =release_type = dev=
---+++ OBSOLETE - Manual RPM build for bigpandamon-atlas
---+++ Update RPM configs in the devel branch
   * Check the latest version available, do not forget to increment version string.
      * https://pandabuild.cern.ch/x86_64/
      * https://pandabuild.cern.ch/SRPMS/ 
   * cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas  <verbatim>
# cd panda-bigmon/trunk/bigpandamon-atlas</verbatim>
   * edit file =branches/devel/bigpandamon-atlas/setup.cfg=
      * update "requires" if necessary<verbatim>
# svn diff setup.cfg
Index: setup.cfg
===================================================================
--- setup.cfg   (revision 18781)
+++ setup.cfg   (working copy)
@@ -4,5 +4,6 @@
 provides = bigpandamon-atlas
 release = 1
 packager = Panda Team <hn-atlas-panda-pathena@cern.ch>
-requires = python bigpandamon-core
+requires = python 
+           bigpandamon-core >= 0.0.4
</verbatim>
   * edit file =branches/devel/bigpandamon-atlas/setup.py=
      * update/check whatever is marked with =#FIXME=, i.e. release_version, expected_extensions, packages, data_files
   * commit changes to RPM configs<verbatim>
# svn ci -m "bigpandamon-atlas/branches/devel: prepared RPM setup."
</verbatim>

---+++ Merge devel branch to trunk
   * cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas <verbatim>
# cd panda-bigmon/trunk/bigpandamon-atlas
</verbatim>
   * update local copy of trunk of bigpandamon-atlas <verbatim>
# svn up
</verbatim>
   * merge =branches/devel/bigpandamon-atlas= branch to =trunk/bigpandamon-atlas=
      * postpone conflicts for setup.py and setup.cfg <verbatim>
# svn merge --reintegrate svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/branches/devel/bigpandamon-atlas
--- Merging differences between repository URLs into '.':
U    atlas/prodtask/views.py
U    atlas/prodtask/request_views.py
U    atlas/prodtask/urls.py
U    atlas/prodtask/forms.py
U    atlas/prodtask/templates/prodtask/_index.html
U    atlas/prodtask/templates/prodtask/_reqdatatable.html
C    setup.cfg
C    setup.py
</verbatim>
      * resolve conflicts for setup.py and setup.cfg <verbatim>
# svn resolve --accept theirs-full setup.cfg
Resolved conflicted state of 'setup.cfg'
# svn resolve --accept theirs-full setup.py
Resolved conflicted state of 'setup.py'
</verbatim>
   * now go through changes before committing the merge, and adapt what is necessary, resolve collisions etc. 
      * diff from svnweb can be useful as an additional check: e.g. [[https://svnweb.cern.ch/trac/panda/changeset?new=18778%40panda-bigmon%2Fbranches%2Fdevel&old=18772%40panda-bigmon%2Ftrunk]]
   * commit the merge <verbatim>
# svn ci -m "Merged devel/bigpandamon-atlas to trunk/bigpandamon-atlas, RC for bigpandamon-atlas-0.0.4"
</verbatim>
---++++ Merge troubleshooting
   * If someone accidentally committed into trunk instead of branches/devel, you may get error message such as<verbatim>
# svn merge --reintegrate svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/branches/devel/bigpandamon-atlas
svn: Reintegrate can only be used if revisions 18773 through 19007 were previously merged from svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas to the reintegrate source, but this is not the case:
  panda-bigmon/branches/devel/bigpandamon-atlas/atlas
    Missing ranges: /panda-bigmon/trunk/bigpandamon-atlas/atlas:18773-18862
</verbatim>
The resolution of this issue is described in [[http://stackoverflow.com/a/2648969]]: 
      * go to trunk/bigpandamon-atlas and perform <verbatim>
svn propdel svn:mergeinfo -R
svn revert .
svn ci -m "Removed mergeinfo"
</verbatim>
      * go to branches/devel/bigpandamon-atlas and perform <verbatim>
svn propdel svn:mergeinfo -R
svn revert .
svn ci -m "Removed mergeinfo"
</verbatim>
      * go back to trunk/bigpandamon-atlas and restart the merge.



---+++ Create tag of bigpandamon-atlas-0.0.4
   * cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas <verbatim>
# cd panda-bigmon/trunk/bigpandamon-atlas
</verbatim>
   * create a new tag <verbatim>
svn copy \
      svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-atlas \
      svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/tags/bigpandamon-atlas/bigpandamon-atlas-0.0.4 \
      -m "bigpandamon-atlas-0.0.4"
</verbatim>

---+++ Build RPM
   * login to aipanda004 as jschovan/sbelov <verbatim>
# [lxvoadm]# ssh jschovan@aipanda004
</verbatim>
   * cd to a workdir <verbatim>
# MYWORKDIR="/data/jschovan/new-rpm/MYWORKDIR"
# mkdir -p $MYWORKDIR
# cd $MYWORKDIR
</verbatim>
   * prepare .rpmmacros <verbatim>
# echo "%_unpackaged_files_terminate_build 0" >> ~/.rpmmacros 
</verbatim>
   * checkout the tagged code <verbatim>
# svn co svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/tags/bigpandamon-atlas/bigpandamon-atlas-0.0.4
</verbatim>
   * cd to the tagged directory <verbatim>
# cd $MYWORKDIR/bigpandamon-atlas-0.0.4
# ls $MYWORKDIR/bigpandamon-atlas-0.0.4
atlas  INSTALL.txt  MANIFEST.in  README.txt  setup.cfg  setup.py
</verbatim>
   * build RPM of bigpandamon-atlas-0.0.4 <verbatim>
# QA_SKIP_BUILD_ROOT=1 python setup.py bdist_rpm 
... lots of output, and then ...
moving build/bdist.linux-x86_64/rpm/SRPMS/bigpandamon-atlas-0.0.4-1.src.rpm -> dist
moving build/bdist.linux-x86_64/rpm/RPMS/noarch/bigpandamon-atlas-0.0.4-1.noarch.rpm -> dist

# ls $MYWORKDIR/bigpandamon-atlas-0.0.4
atlas  build  dist  INSTALL.txt  MANIFEST  MANIFEST.in  README.txt  setup.cfg  setup.py

# ls $MYWORKDIR/bigpandamon-atlas-0.0.4/dist -1
bigpandamon-atlas-0.0.4-1.noarch.rpm  
bigpandamon-atlas-0.0.4-1.src.rpm  
bigpandamon-atlas-0.0.4.tar.gz
</verbatim>
   * check files in the noarch RPM
      * make sure that none of the python modules is missing (make sure every python module has __init__.py committed in SVN)
      * make sure that none of the templates files is missing
      * make sure that the static files are not missing <verbatim>
# rpm -qpl dist/bigpandamon-atlas-0.0.4-1.noarch.rpm | less
</verbatim>
   * remove .rpmmacros <verbatim>
# rm ~/.rpmmacros
</verbatim>

---+++ Copy RPMs to the repository
   * sudo su on aipanda004
   * cd to the =dist= directory <verbatim>
# MYWORKDIR="/data/jschovan/new-rpm/MYWORKDIR"
# cd $MYWORKDIR/bigpandamon-atlas-0.0.4/dist
</verbatim>
   * cp src.rpm to /data/build/SRPMS<verbatim>
# cp bigpandamon-atlas-0.0.4-1.src.rpm /data/build/SRPMS/
</verbatim>
   * cp noarch.rpm to /data/build/x86_64 <verbatim>
# cp bigpandamon-atlas-0.0.4-1.noarch.rpm /data/build/x86_64
</verbatim>


---++ Manual build of bigpandamon-core for ATLAS

---+++ Update RPM configs in the devel branch
   * Check the latest version available, do not forget to increment version string.
      * https://pandabuild.cern.ch/x86_64/
      * https://pandabuild.cern.ch/SRPMS/ 
   * cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-core  <verbatim>
# cd panda-bigmon/trunk/bigpandamon-core</verbatim>
   * edit file =branches/devel/bigpandamon-core/setup.py=
      * update/check whatever is marked with =#FIXME=, i.e. release_version, expected_extensions, packages, data_files
   * commit changes to RPM configs<verbatim>
# svn ci -m "bigpandamon-core/branches/devel: prepared RPM setup."
</verbatim>

---+++ Merge devel branch to trunk
   * cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-core <verbatim>
# cd panda-bigmon/trunk/bigpandamon-core
</verbatim>
   * update local copy of trunk of bigpandamon-core <verbatim>
# svn up
</verbatim>
   * merge =branches/devel/bigpandamon-core= branch to =trunk/bigpandamon-core=
      * postpone conflicts for setup.py and setup.cfg <verbatim>
# svn merge --reintegrate svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/branches/devel/bigpandamon-core
--- Merging differences between repository URLs into '.':
... updates ...
C    setup.py
</verbatim>
      * resolve conflicts for setup.py (and setup.cfg) <verbatim>
# svn resolve --accept theirs-full setup.py
Resolved conflicted state of 'setup.py'
</verbatim>
   * now go through changes before committing the merge, and adapt what is necessary, resolve collisions etc. 
      * diff from svnweb can be useful as an additional check: e.g. [[https://svnweb.cern.ch/trac/panda/changeset?new=18778%40panda-bigmon%2Fbranches%2Fdevel&old=18772%40panda-bigmon%2Ftrunk]]
   * commit the merge <verbatim>
# svn ci -m "Merged devel/bigpandamon-core to trunk/bigpandamon-core, RC for bigpandamon-core-0.0.4"
</verbatim>



---+++ Create tag of bigpandamon-core-0.0.4
   * cd to the local copy of svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-core <verbatim>
# cd panda-bigmon/trunk/bigpandamon-core
</verbatim>
   * create a new tag <verbatim>
svn copy \
      svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-core \
      svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/tags/bigpandamon-core/bigpandamon-core-0.0.4 \
      -m "bigpandamon-core-0.0.4"
</verbatim>

---+++ Build RPM
---++++ Automatic bigpandamon-core RPM build
---+++++ Automatic bigpandamon-core RPM build - trunk
   * Upon every commit to https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/trunk/bigpandamon-core an RPM is built.
   * Trunk RPM is available https://pandabuild.cern.ch/x86_64/ as <verbatim>
bigpandamon-core-<stable version>-1.noarch.rpm
</verbatim>
      * <verbatim><stable version></verbatim> is the most recent (current, new) stable version of the =bigpandamon-core= package
   * Naming convention steering for trunk: in [[https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/trunk/bigpandamon-core/setup.cfg][trunk/bigpandamon-core/setup.cfg]]
      * =version = VALUE=, where =VALUE= is the new stable version number
      * =release_type = stable=
   * Naming convention steering for devel: in [[https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/branches/devel/bigpandamon-core/setup.cfg][devel/bigpandamon-core/setup.cfg]]
      * =version = VALUE=, where =VALUE= is the newest stable version number
      * =release_type = dev=
---+++++ Automatic bigpandamon-core RPM build - branches/devel
   * Upon every commit to https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/branches/devel/bigpandamon-core an RPM is built.
   * Devel RPM is available https://pandabuild.cern.ch/unstable/ as <verbatim>
bigpandamon-core-<stable version>.dev_r<svn revision of the commit>-1.noarch.rpm
</verbatim>
      * <verbatim><stable version></verbatim> is the most recent stable version of the =bigpandamon-core= package
      * <verbatim><svn revision of the commit></verbatim> is the revision number of the latest commit into =branches/devel= of =bigpandamon-core=.
   * Naming convention steering: in [[https://svnweb.cern.ch/trac/panda/browser/panda-bigmon/branches/devel/bigpandamon-core/setup.cfg][devel/bigpandamon-core/setup.cfg]]
      * =version = VALUE=, where =VALUE= is the latest stable version number
      * =release_type = dev=
---++++ OBSOLETE - Manual RPM build for bigpandamon-core
   * login to aipanda004 as jschovan/sbelov <verbatim>
# [lxvoadm]# ssh jschovan@aipanda004
</verbatim>
   * cd to a workdir <verbatim>
# MYWORKDIR="/data/jschovan/new-rpm/MYWORKDIR"
# mkdir -p $MYWORKDIR
# cd $MYWORKDIR
</verbatim>
   * prepare .rpmmacros <verbatim>
# echo "%_unpackaged_files_terminate_build 0" >> ~/.rpmmacros 
</verbatim>
   * checkout the tagged code <verbatim>
# svn co svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/tags/bigpandamon-core/bigpandamon-core-0.0.4
</verbatim>
   * cd to the tagged directory <verbatim>
# cd $MYWORKDIR/bigpandamon-core-0.0.4
# ls $MYWORKDIR/bigpandamon-core-0.0.4
core  INSTALL.txt  MANIFEST.in  README.txt  setup.cfg  setup.py
</verbatim>
   * build RPM of bigpandamon-core-0.0.4 <verbatim>
# QA_SKIP_BUILD_ROOT=1 python setup.py bdist_rpm 
... lots of output, and then ...
moving build/bdist.linux-x86_64/rpm/SRPMS/bigpandamon-core-0.0.4-1.src.rpm -> dist
moving build/bdist.linux-x86_64/rpm/RPMS/noarch/bigpandamon-core-0.0.4-1.noarch.rpm -> dist

# ls $MYWORKDIR/bigpandamon-core-0.0.4
core  build  dist  INSTALL.txt  MANIFEST  MANIFEST.in  README.txt  setup.cfg  setup.py

# ls $MYWORKDIR/bigpandamon-core-0.0.4/dist -1
bigpandamon-core-0.0.4-1.noarch.rpm  
bigpandamon-core-0.0.4-1.src.rpm  
bigpandamon-core-0.0.4.tar.gz
</verbatim>
   * check files in the noarch RPM
      * make sure that none of the python modules is missing (make sure every python module has __init__.py committed in SVN)
      * make sure that none of the templates files is missing
      * make sure that the static files are not missing <verbatim>
# rpm -qpl dist/bigpandamon-core-0.0.4-1.noarch.rpm | less
</verbatim>
   * remove .rpmmacros <verbatim>
# rm ~/.rpmmacros
</verbatim>

---+++ Copy RPMs to the repository
   * sudo su on aipanda004
   * cd to the =dist= directory <verbatim>
# MYWORKDIR="/data/jschovan/new-rpm/MYWORKDIR"
# cd $MYWORKDIR/bigpandamon-core-0.0.4/dist
</verbatim>
   * cp src.rpm to /data/build/SRPMS<verbatim>
# cp bigpandamon-core-0.0.4-1.src.rpm /data/build/SRPMS/
</verbatim>
   * cp noarch.rpm to /data/build/x86_64 <verbatim>
# cp bigpandamon-core-0.0.4-1.noarch.rpm /data/build/x86_64
</verbatim>


---++ Update RPM repos
   * sudo su on aipanda004 (the pandabuild.cern.ch machine)

   * cd to dir with repobuild scripts <verbatim>
# cd /data/build-not-public
</verbatim>
   * run the repobuild scripts <verbatim>
# ./repo_SRPMS
# ./repo_x86_64
</verbatim>


---++ Deployment to the integration/production machines
---+++ Deployment to the integration machine
   * sudo su on aipanda043 - integration
   * to be continued...

---+++ Deployment to the production machine
   * sudo su on aipanda022 - production <verbatim>
# yum clean all
# yum upgrade
</verbatim>
   * make sure =local.py= is in place and is correct <verbatim>
# /data/virtualenv/django1.6.1__python2.6.6__atlas/pythonpath/core/common/settings/local.py
# /data/virtualenv/django1.6.1__python2.6.6__atlas/pythonpath/atlas/settings/local.py
</verbatim>
   * activate the =virtualenv=
      * make sure "pip freeze" contains all the =virtualenv= dependencies <verbatim>
# cd /data/virtualenv/django1.6.1__python2.6.6__atlas
# source bin/activate
# export PYTHONPATH=/data/virtualenv/django1.6.1__python2.6.6__atlas/pythonpath:$PYTHONPATH
</verbatim>
   * try out =./manage.py runserver= <verbatim>
# cd /data/virtualenv/django1.6.1__python2.6.6__atlas/pythonpath/atlas
# python manage.py runserver
</verbatim>
   * run =./manage.py collectstatic= <verbatim>
# python manage.py collectstatic
</verbatim>
   * restart the httpd service and change the owner of the logs files to apache:apache <verbatim>
# service httpd restart
# chown apache:apache -R /data/atlpan/bigpandamon/var/log/* /data/bigpandamon_virtualhosts/atlas/logs/* /data/bigpandamon_virtualhosts/jedimon/logs/*
</verbatim>



%BR%
%BR%
%BR%
%BR%
%BR%


*Major updates*:%BR% <!--Sign below for major edits--> 
%BR%-- Main.JaroslavaSchovancova - 24 Apr 2014
%BR%
%BR%
%BR%
%BR%
%BR%

%STOPINCLUDE%

<!--Please add the name of someone who is responsible for this page so that they can be contacted if the contents need updating! The creator's name will be added by default using the REVINFO function, but this can be replaced if appropriate.--> 
%RESPONSIBLE% Main.JaroslavaSchovancova 
%BR%
Topic revision: r2 - 2014-05-19 - JaroslavaSchovancova
PanDA
Public webs
- Cern Search
- TWiki Search
- Google Search
PanDA All webs
Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback