Most important new features for Ganga

Introduction

For the efficient discussion about new features it is useful first to recall the main goal of the Ganga Project and organize new features in their respective domains.

The following are the most important aspects of Ganga:

1. HEP Physicist tool: Ganga should cover all needed functionality by Atlas and LHCb experiments to efficiently peforms their tasks within their corresponding computing models. Some of the experiment-specific features may require special support in the Core Ganga system - and these are in focus here.
2. Generic end-user tool: the great virtue of Ganga is that it is not restricted to HEP experiments and as such may be used by other communites, and maybe beyond standard tasks (e.g.: Educational Purpose in EGEE, Analysis of the Grid behaviour: MSc thesis in Computing Science by Derek Groen). Therefore the secondary goal is to make sure that Ganga is:
   1. experiment-neutral
   2. generic and extensible
   3. appealing to users: easy to use and friendly
3. Grid interface API: Ganga may be used as an abstraction layer for Grid job management for other frameworks (e.g. UNOSAT uses Ganga server side CGI scripts behind a Grid portal, DIANE uses Ganga as a job submission engine) or to build other tools (for example GangaRobot). The goal is to make sure that Ganga is:
   1. accessible via a cleanly defined, programming-oriented API
   2. stable and robust

The plan of the session

• Part1: Experiment-specific features the most important ones and the ones which affect other parts of Ganga - 60 minutes
  • Andrew: LHCb
    1. Merging of output from split jobs (Part 1)
    2. Fine grained control over location of output from jobs (Part 1)
    3. Resubmission of failed subjobs (Part 1)
    4. Access to MC information for DSTs (see feature request #20924) (Part 1)
  • Dietrich: Atlas
• Part2: General Ganga features - 90 minutes
  1. Kuba: Changes in existing API: Kuba_new_old_features
     • select() and other collective operations (also with relation to the jobtree)
     • exceptions for job operations (j.submit(), j.kill())
  2. Alexander: Implementation of the remote workspace (aka LSE)
  3. Alvin: Print-levels for Ganga objects -> GangaAttributeVisibility
  4. Kuba: Timestamps
  5. Karl: Expired tokens handling -> GangaCleanShutdown
  6. Alvin: Type information in Ganga schema -> GangaTypes
  7. Ulrik: Job status transition: failing jobs manually etc. -> GangaJobStateMachine
  8. Alexander: Lazy reading of the repository
  9. Kuba: Automatic generation of config file template
  10. Kuba: Logging levels cleanup
  11. Karl: Waltk-thru the General Issues (below)
• Part3: Other experiment-specific features - 30 minutes
  • Andrew: LHCb
  • Dietrich: Atlas

Other, general issues for Ganga

• The workplan in the GridPP Ganga Logbook foresees a release in May 2007 that maximises Ganga portability, and allows for user mobility. In particular, there are suggestions to try and make Ganga platform indepdendent (there's some interest in having it run under Windows), and to make some Ganga functionality available via a web browser. It would be good to understand what, if anything, we want to do in this area. Kuba: User mobility should be enhanced by having a remote workspace (LSE). Porting to Windows is a pretty independent task.
• In the context of EGEE, Ganga has been used to good effect together with DIANE - see, for example, slide 14 of presentation at GridPP16. Is there some useful DIANE functionality that could be incorporated into Ganga? Kuba: Ganga has been used a submission engine, hidden behind DIANE shell commands. On the other hand there exists Ganga interface for DIANE too - so users can use Ganga as the primary interface. Some preview of this and more explanation is available here: DIANEGangaIntegration.
• Peek functionality - possibility to list/view files in job's work directory. How can this be done for a Grid backend? Kuba: unless there is a tool on the Grid to do it, our own implementation might be too heavy. Dietrich: Hurng-Chun is working integrating the LCG tool for job monitoring into the LCG handler. The problem is that its status is shakey and functionallity is limited. We are also looking at the Wuppertal monitoring using R-GMA. The only complete approach to monitoring is done by Alien, but they open a socket from each job.
• Job-options editor - should further work on this be undertaken in the context of Ganga? Rather than an all-new editor, it might be better to have customisation files for some existing editor(s) (e.g. emacs, vi). Kuba: Probably we should not invest more in the job-options editor in the near future, given also the possible move to python-based option files Dietrich: Atlas is using python-based option files since long time. An intresting feature, that we could profit from is the automatic extraction of important parameters from the python file by using a fake Athena instance. This would allow to collect information on the requested output files. Such an apprpach might be also useful for LHCb.
• Ensure that Grid proxy doesn't expire until a job has completed. Is this something that can/should be handled at Ganga level? Kuba: we could emit a warning if the proxy timeleft would be less than the time required for running a job (with some rounding up). This is implemented in DIRAC handler but not in LCG currently. Dietrich: The LCG handler already supports myproxy for renewal. What is missing is the automatic maintainance of the long term credidential by GANGA. Evidently this could be an interesting extension for the Credidential package.
• Code compilation at remote sites - is this something Ganga could/should support? Kuba: as a first step of running the job -> this is application specific. If it comes to compiling the code as a separate step (i.e.remote application manager), then probably this feature could be difficult for users to understand. Dietrich: I do not share Kubas view in that matter. pathena/PANDA is doing just that and I think the users have no problem in understanding whats going on. The problem is that seperate compilation requires the support of dependencies between jobs and how they can be supported on our backends.
• Provide useful diagnostics when a Grid job fails - a minimum requirement might be that stdout and stderr should be returned to the user Kuba:This could be certainly useful: if the application exit code is non-zero, extra informaiton such as ls, ldd, env, etc. could be added to stderr Dietrich: If there is no output returned we have a problem in any case. Should we support a mechanism to expose application failures and their reason in the GPI. E.g. a standard status/sucess field for all applications. If the application fails this could be reported in a standard file that is read at the end of the job and the failure is visible already in the application object. Without looking into the log file. Kuba: agree, this information can be stored in a separate file with a uniform name for all backends. BTW: in LCG we always have the output because the wrapper returns always 0 (even if the application fails).
• Karl - user request: Provide estimate at submission time of time for job to complete - better to be told in advance that it will take 3 days than to wait for a random time between 10 minutes and 2 weeks!. Dietrich: This is a nice idea. For the LCG handler we can print the estimated response time in the monitoring loop, when the assignment of the job to a CE is known. (We should include a ldap module in the GANGA distribution to simplify the access to the BDII.
• Karl - user request: Minimise Ganga startup time - try and do more on-demand, for example create a Grid proxy only when it is really needed Kuba: This is related to Lazy reading of the repository (Alexander) and Expired token handling (Karl). I've made an update in the agenda.
• Karl - user request: Improve scheme for deleting sub-jobs - start from sub-jobs not in running state

LHCb Feature List

• Merging of output from split jobs (Part 1)
• Fine grained control over location of output from jobs (Part 1)
• Resubmission of failed subjobs (Part 1)
• Access to MC information for DSTs (see feature request #20924) (Part 1)
• Support for different production tools with Gauss (work in progress?)
• Add possibility to chain applications - at least functionality analogous to AtlasAthenaMC for Atlas
• Provide mechanism for working with user-provided data files (e.g. XML detector descriptions, decay files)
• Additions to the Dirac backend. Since Dirac is the only way LHCb users are supposed to use the grid, it should have all the functionallity of the LCG plugin. While this is not necessarily a Ganga issue, we might want ot compile a wishlist for enhancments of the Dirac API (Part 1, may be discussed as part of "Access to MC information for DSTs")
• Support for Bender application
• Use of Panoramix with Interactive backend. Is this something that could/should be supported? Would there be any advantage compared with running Panoramix outside of Ganga?
• Access to CMT functionality for package, for example app.cmt("show uses") should show the packages used by the master package of an LHCb application

Atlas Feature List

• Improve GANGA handling of monitoring loop. For scripts the monitoring loop should be optional. There should also be a lock so that only one instance of GANGA handles the status updates. * Access to result data, merging etc from the GUI. What is possible today, what hould be possible. I guess its more a discussion item
• Support for PANDA Dietrich: I will attach information later the weekend
• Support for Nordugrid backends. Plans are attached
• GANGA on top of VDT UI (US Grid)
• The Athena handler already support for Local and LSF backends for easy debugging. We should extend support to all applications. Do we have a good solution for the proxy problem ?
• Improve job splitting and master/subjobs management.
• Access to AMI and minimalistic dataset browser
• Tag navigator tool (work being done by Mike Kenyon)
• Improving integration with DDM system DietrichSome recent DDM issues

• GangaRobot Dietrich: I will attach information later the weekend
• Remote file repository, support for large sandboxes

• JakubMoscicki - 20 Oct 2006
• UlrikEgede - 03 Nov 2006