BigPanDA Monitoring Environment (Production)

Introduction

pandawms.org machine configuration

  • AWS EC2 machine running image with Scientific Linux release 6.4 (Carbon).
  • BigPanDA monitoring application is a Django application running in virtualenv, presented via apache/mod_wsgi.

Deployment model

  • Environment:
    • Python 2.6.6 (as of 2014-01-16)
    • Using virtualenv 1.11 (as of 2014-01-16)
    • Upgrade of a virtualenv project comes with with pip requirements upgrade
  • BigPanDA monitoring application
    • As of 2014-01-16 upgrade of the BigPanDA monitoring app is done with rsync of directories from SVN.
    • The plan is to deploy BigPanDA monitoring app with a set of RPM packages.
  • Database backend(s)

Update of the production machines

Update of bigpanda.cern.ch

  • The production machines are listed on external TWiki: AtlasComputing.DistributedComputingMachines#BigPanda_Monitor
  • Update can be performed only by a limited group of sudoers: jschovan,wenaus,sbelov,plove.
  • Update is performed by running a script from lxvoadm.cern.ch machine:
    # ssh jschovan@lxvoadm.cern.ch
    # /afs/cern.ch/user/a/atlpan/public/panda/mon/cs-ops/prodBigMonRPMUpdate.sh
    
    

Environment configuration

virtualenv configuration

  • A quick README file is in SVN.

Install virtualenv

  • Install python installer pip
    # yum install python-pip
  • Install virtualenv
    # pip install virtualenv

Create a virtualenv project

  • Only once create a placeholder directory where virtualenv project will be stored
    # mkdir -p /data/virtualenv
  • Create a new virtualenv project, e.g. django1.6.1__python2.6.6
    # MyNewProject=django1.6.1__python2.6.6
    # cd /data/virtualenv
    # virtualenv ${MyNewProject}
  • The new virtualenv project has been created into directory /data/virtualenv/${MyNewProject}/
    # ls /data/virtualenv/${MyNewProject}
       bin  include  lib  local

Activate/deactivate a virtualenv project

  • Activate ${MyNewProject}
    # source /data/virtualenv/${MyNewProject}/bin/activate
    • Upon successful activation the virtualenv project name is shown in front of your usual $PS1 text in terminal prompt.
  • Deactivate ${MyNewProject}
    # deactivate

Environment settings for a virtualenv project

  • Dump current settings
    # pip freeze > requirements.txt
  • Load saved settings (override current settings!)
    # pip install -r requirements.txt

Install packages inside a virtualenv project

  • gcc
    • system-wide environment
      # yum install build-essential python-devel gcc
  • MySQL interface for python
    • The system-wide environment
      # yum install mysql-devel
    • virtualenv project environment
      # pip install MySQL-python
  • pycurl package for python
    • system-wide environment
      # yum install libcurl-devel
    • virtualenv project environment
      # pip install pycurl
  • Django
    # pip install Django
  • Django dataTables view
    # pip install django-datatables-view
  • Django REST framework
    # pip install djangorestframework
  • python-dateutil
    # pip install python-dateutil
  • pytz
    # pip install pytz

BigPanDA monitoring application configuration: LSST

Configure virtualenv

  • BigPanDA monitoring virtualenv requirements.txt
  • Create virtualenv directory for your project
    # MyNewProject="django1.6.1__python2.6.6__lsst"
    # cd /data/virtualenv
    # virtualenv ${MyNewProject}
    # cd /data/virtualenv/${MyNewProject}
    # pip install -r requirements.txt
    
  • Create the application directory ${APPDIR}, directory into which you will symlink your monitoring site code:
    # APPDIR="/data/virtualenv/${MyNewProject}/pythonpath/"
    # mkdir -p ${APPDIR}
    • R to apache user, RW to panmonusr1 user
  • Create the code directory, ${CODEDIR}, directory into which you will install/checkout bigpandamon-core and bigpandamon-lsst code:
    # CODEDIR="/data/bigpandamon_virtualhosts/lsst"
    # mkdir -p ${CODEDIR}
    
    • R to apache user, RW to panmonusr1 user

Configure bigpandamon-core

  • Check out/install bigpandamon-core:
    # cd ${CODEDIR}
    # svn co svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-core/core core
    
  • TODO (as of 2014-02-12): prepare bigpandamon-core RPM
  • Rename local.py-template to local.py, and change appropriately.
    # cd ${CODEDIR}/core/common/settings/
    # mv local.py-template local.py
    

Configure VO/project-specific monitoring application

  • Check out/install bigpandamon-lsst:
    # cd ${CODEDIR}
    # svn co svn+ssh://svn.cern.ch/reps/panda/panda-bigmon/trunk/bigpandamon-lsst/lsst lsst
    
  • Logs directory ${LOGDIR}
    # LOGDIR=${CODEDIR}/logs
    • RW to apache, panmonusr1 users
    • lsst.settings.config.LOG_ROOT must point to ${LOGDIR}
  • Content of ${APPDIR}
    # ls -l ${APPDIR}
    core -> ${CODEDIR}/core
    lsst -> ${CODEDIR}/lsst
    
  • Content of ${CODEDIR}
    # ls -l ${CODEDIR}
    core  logs  lsst
    
  • Django settings ${DJANGO_SETTINGS_MODULE}
    # DJANGO_SETTINGS_MODULE=lsst.settings
  • BigPanDAmon settings file to load in lsst.wsgi: ${WSGI_PRE_SETTINGS}
    # WSGI_PRE_SETTINGS=/data/bigpandamon_settings/settings_bigpandamon_lsst.py
    • Add directory with ${WSGI_PRE_SETTINGS} to pythonpath of mod_wsgi (WSGIPythonPath)
    • Load WSGI_PATH from ${WSGI_PRE_SETTINGS} in lsst.settings.config
    • Example of wsgi.py for LSST.
    • Example of settings_bigpandamon_lsst.py.
  • Rename local.py-template to local.py, and change appropriately.
    # cd ${CODEDIR}/lsst/settings/
    # mv local.py-template local.py
    
  • Every settings configuration is done in ${CODEDIR}/lsst/settings. More information in BigPanDAmonitoringDevelopers#VO_project_specific_package_sett.
  • Run collectstatic:
    # cd /data/virtualenv/${MyNewProject}
    # source bin/activate
    # cd ${APPDIR}/lsst
    # ./manage.py collectstatic
    # deactivate
    

httpd configuration

  • mod_wsgi configuration
  • SSL configuration - use SSL certificate per host

References

Further reading about Django, virtualenv and mod_wsgi






Major updates:

-- JaroslavaSchovancova - 13 Feb 2014






Responsible: JaroslavaSchovancova

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r3 - 2014-07-29 - JaroslavaSchovancova
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback