Development of EMI Verification Dashboard

Introduction

The EMI Verification dashboard web page is available a thttp://emi-dashboard.cern.ch/ and at http://emi-verification.cern.ch/ It's written in Django 1.1 with all the information stored into a mysql database.

The computer used for hosting the web pages is lxbrf2713.cern.ch and are located under /srv/emi_dashboard.

You can find the release scripts and the web application source code in https://svn.cern.ch/reps/grelscripts/emi_dashboard/ or in websvn http://svnweb.cern.ch/world/wsvn/grelscripts/emi_dashboard/

For backups, have a look at the backup.sh script, and you can find them at /afs/cern.ch/project/gd/gliteweb-backup/emi_dashboard (permission is needed to access it)

Installation

  • An SL5 machine it's recomended
  • Enable the EPEL repo (needed for mod_wsgi) modifying /etc/yum.repos.d/epel.repo
  • Install apache2 and mysql
    • Install mysql: yum install mysql-server
    • Install apache2: yum install httpd mod_ssl mod_wsgi
  • Install subversion: yum install subversion
  • Download the source code from https://svn.cern.ch/reps/grelscripts/
  • Link grelscripts/emi_dashboard/ from /srv and make sure apache can read it.
    • ln -s grelscripts/emi_dashboard/ && chown -R apache:apache emi_dashboard
    • chown -R apache:apache grelscripts/emi_dashboard
    • mkdir /var/www/forbiden && chown -R apache:apache /var/www/forbiden
  • Copy the apache configuration and configure it
    • cp -b emi_dashboard/httpd.conf /etc/httpd/conf/
    • Change the domain name to the new one (if that's the case) in /etc/httpd/conf/httpd.conf in the VirtualHost section.
    • Review the configuration and remove things that are not needed (e.g. gLite web pages, if that's the case; For development only, remember to remove the ServerAlias).
    • Uncoment the last line in /etc/httpd/conf.d/wsgi.conf to enable the module
    • Comment all the lines in the Virtualhost section (including it) at the end of the file /etc/httpd/conf.d/ssl.conf
  • Install django 1.1
  • Install python mysql module (version 1.2.1_p2)
  • Install South database migration tool:
  • Start apache and mysql
    • /etc/init.d/mysqld start
    • /etc/init.d/httpd start
  • Migrate the database
    • Execute mysql.
    • Create the database: "create database emi_dashboard char set utf8;"
    • Give permissions to a user in the database: "grant all on emi_dashboard.* to emi@localhost identified by 'thepassword';"
    • Exit form mysql: exit
    • Load a backup: mysql emi_dashboard < /afs/cern.ch/project/gd/gliteweb-backup/emi_dashboard/backup_file_to_restore
    • Change username and password in /srv/emi_dashboard/settings.py
    • Update the schema: /srv/emi_dashboard/manage.py migrate
  • Change the secret key in /srv/emi_dashboard/settings.py ( you can use http://mylesbraithwaite.com/journal/2007/10/secret-key/ )
  • Get a savannah cookie (using your savannah user / passwd): ./get_savannah_cookie.py
  • Restart the web server: /etc/init.d/httpd restart
  • Setup a CRON job to update the tasks: crontab -e
  • Configure autostart of afs, apache and mysql so that it will start again after a reboot (it's what you want):
    • chkconfig --levels 345 afs on
    • chkconfig --levels 345 mysqld on
    • chkconfig --levels 345 httpd on

Database Migration

If you change the models in a way that requires a change to the database, you will need to create a new migration. You can do it with the command:

./manage.py schemamigration dashboard --auto

Then, you have to apply the migration:

./manage.py migrate dashboard

Don't forget to commit the new migration file created under doashboard/migrations, and to apply the migration in production if everything worked fine during the tests.

How to make backups

You can create a backup running backup.sh from the emi_dashboard directory. In order to copy the backups to AFS (in addition to local) you will need to have write access, so please make sure that you have the needed tokens/tickets.

You should backup every few days (manually, as you need special rights), and more often in periods when it's changing quickly. For example, backup every day before the updates.

How to restore a backup

With enough rights, you can run: mysql emi_dashboard < /afs/cern.ch/project/gd/gliteweb-backup/emi_dashboard/any_backup_I_want

This should recreate the database completely. If you are restoring an old backup, that it's not in sync with the latest code, you should apply the migrations, as described before.

Changes in the dashboard

Add a check

In order to add a check, you need to create a new class in dashboard/models.py that inherits from one of the Check subclases (GenericCheck, EticsCheck, TestCheck DocumentationCheck) Documentation checks are more complex and will not be covered here as are not extensively used.

You will have to implement the getHelpMsg method, that it's used to return a string explaining the check. The preCheck method, that it's used to fill the Certification, and Test report results of the check. And the doCheck that will perform the SA2 check. There are many examples in the other checks, so probably it's just a matter of copy and modify a similar one. It's also important to provide a cert_unique_id value that corresponds with the report hash where the fields are described.

Once the check is implemented, you will need to migrate the database as described in the previous section. You might also need to update a task, or all the tasks, as described afterwards.

Add a savannah field

To parse a new field from savannah, you need to add the corresponding field to the Task, ReleaseTask or ComponentTask in dashboard/models.py, depending of the type of task. You will also need to implement the parse of the field in the parse or specificParse methods of Task, ReleaseTask or ComponentTask, and assing the value to the new field in the class you just created. There are many simple examples using BeautifulSoup for other fields that can be used as a base.

Finally, you will need to migrate the database as explained before. You might need to implement a check that uses this information too, as otherwise it's useless.

Add a new version of the certification / test report

If the certification or test report release a new version, you will need to update the hashes that contains a description of the reports for each version. You need to update them even if you don't want to use the new version, because otherwise the dashboard will crash on this versions.

The hashes are in the class AbstractReportField, on dashboard/models.py and are named certification_hash and test_hash. You can just copy the previous version, change the version number, and add the modifications. Respect the python dictionary syntax, otherwise it wont work.

You might want to update some tasks to get the checks updated again the new added version. More information available in the following sections.

Add a new CSV report

To add a new CSV report, you need to modify many different things.

First, the HTML and Javascript.

You need to modify templates/dashboard/_tasks_display.html to add a link for your new CSV. Decide a name (see the current ones) and use it for the id and href. Make sure you have something similar to the other links.

Now the js. Under static/js/scripts.js there are CSV comment (line 86 at this time) you define what happens when you click on the link you just added. You will have to duplicate one of this sections, and change the id on the 3 places to your new one.

Then we need to modify the file urls.py and add a line for the new csv. Duplicate one of the previous ones and change the names. It's important to use the same name all the time, otherwise it's difficult to understand.

Now we have to create the new view we just pointed to in the url. Edit dashboard/views.py, more or less at the end, you have more csv classes already. Duplicate one of them, and change the name. The logic will be similar, but the content string is what you have to change to provide the new data, mainly on the for loop. You might need some help form the Django Queryset documentation (remember to use the 1.1 version) to retrieve the data you need.

With that in place, restart the server, and the new CSV should work.

Update a task

If you need to update just a few tasks, there is an easy (and safer that updating all the tasks) way to do it using the django shell.

Run: python manage.py shell > from dashboard.models import *
> t = Task.getUpdatedTask("26124", force = True)
> t.save()

Changing the task number for the one needed.

In this shell you can execute any code you could put in models.py or views.py, so you have full access to do whatever you need in an easy way. But please, consider you are touching the database, and you can break things, so test in a test instance before, and make a backup, just in case.

Update all the tasks

To update all the tasks, you can access a url with some parameters. On the test environment this url is https://veri-dash.cern.ch/update_all_tasks/1/2/1.

The details to update all the tasks in production are:

Go to URL : http://emi-dashboard.cern.ch/update_all_tasks/[ updateAll ]/[ dayBefore ]/[ force ] Where the parameters mean:

  • updateAll = 0 or 1, boolean describing if you want to update all the task or not
  • dayBefore = (integer), will update the tasks that was updated in the last dayBefore
  • force = 0 or 1, boolean describing if you want to force a reparse of the tasks
All arguments are required, so even if updateAll is set to True, please put something in dayBefore (it will be ignored)

For example: https://emi-dashboard.cern.ch/update_all_tasks/1/2/1

Take into account that this can take a lot of time, and that you are recalculating all the tasks, so you might break something. Please make a backup before, test this on a test instance before doing it in production, and avoid it if possible.

-- PabloGuerrero - 14-Jun-2012

Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2012-07-16 - unknown
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    EMI All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback