PanDA Packaging Corner

Packaging.png

Mandate

The purpose of the Packaging working group, is to simplify the adoption of PanDA for other experiments. Our mandate defines 4 main activities:
  • Refactoring code: work on making the PanDA code generic, where different experiments can plugin their customized libraries (e.g. for data handling) and also be compatible to different database backends (currently MySQL and Oracle). We have to avoid the creation of branches and forks that we will have to support separately and that we will never be able to merge again.
  • Deployment/validation: installation and configuration of the components should be simple and automated as much as possible. Steps for validation of the installation should be available.
  • CI/Testing: We should target continuous integration and testing, in order to validate our code changes.
  • Documentation: We need to spread the knowledge across teams and provide the necessary documentation to promote independent work.

Current stories

PanDA@EC2 (Owner: Fernando, Sergey Padolski)

We host a PanDA instance on Amazon EC2, that can be used to new experiments to begin their integration without the initial hurdle of setting up their own infrastructure. This instance is currently used by ALICE (Titan tests and Alien integration) and LSST. Open issues:
  • The server installation is frozen to a rather old version and should be upgraded
  • New PanDa monitor has been reported to have Oracle specific queries that are incompatible towards MySQL
  • Setup a parallel PanDA monitor instance (same machine, different port) to test changes for MySQL compatibility

PanDA@NRC KI (Owner: Ruslan)

NRC KI has a full PanDA installation with some additional layers (user interface and DM plugins) to abstract the usage of PanDA to the local biology community.

PanDA for AMS - migration to MariaDB (Owner: KuoHao, Fernando)

Current issues that need a code fix:
  • Line 64 in /usr/lib/python2.6/sitepackages/pandaserver/taskbuffer/WrappedCursor.py is not compatible with MariaDB:
self.execute("SET @@SESSION.DATETIME_FORMAT='%%Y/%%m/%%d %%H:%%i:%%s")
ERROR 1238 (HY000): Variable 'datetime_format' is a read only variable
  • WrappedCursor.py also seems to need a fix against next(). To be confirmed:
     def next(self):
        if self.backend == 'mysql:
            return self.cur.fetchone()
        else:
            return self.cur.next()
  • panda-cacheschedconfig is not compatible to MySQL

Monitor refactoring (Owner: Misha, Sergey Padolski)

Merging of Core, ATLAS and LSST monitoring packages
  • Repackaging has been done. Now there are only 2 packages (CORE for bigpandamon and ATLAS for prodsys2). ATLAS and CORE have been merged. ATLAS inherits from CORE.
  • Pending is to have only one django application, e.g. same authentication mechanism for both packages

Use PanDA for COMPASS (Owner: Artem)

  • Users will be Sergey Gerasimov and Elena
  • Final purpose is to run jobs on lxbatch and store the output in EOS
  • Artem has set up PanDA server
    • commented out code in PanDA server to bypass them
    • old version of monitor, since the newest version has some Oracle specific parts

PanDA server refactoring (Owner: Fernando, Ruslan, Ivan)

Need to define what to refactor for the next milestone. TBD during SW&C week.

PanDA@OSG (Owner: Fernando)

  • OSG(Carl, Brian) wants to host a PanDA instance for other sciences
  • PanDA server deployment was standardized thanks to Carl's changes. Now need to give clear instructions on how to install, configure and validate.

PanDA server configuration (Owner: Artem)

Moved Artem's description to https://twiki.cern.ch/twiki/bin/view/PanDA/PanDAQueueConfiguration

PanDA-AliEN integration (Owner: Andrey Kondratyev)

We set up Sergey on the PanDA@EC2 instance and he was able . Sergey started setting up his own server, but got stuck due to certificate issues.

Stories to be initiated

JEDI and other components for other experiments

We need to start looking at them.

Sexy component documentation

Following Torre's example on github, we should expand our github documentation
Topic revision: r8 - 2015-07-21 - FernandoHaraldBarreiroMegino
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback