Bookkeeping for Panda analysis jobs

Introduction

pbook is the bookkeeping application for all Panda analysis jobs.

Getting Started

Installation and Setup

pbook is included in the panda-client package. See Installation and Setup for how to.

Changes from old (<1.4.7) pbook

  • Removal of local database and overhead for data synchronization
  • Cleanup of obsolete functions including sync() and restoreDB
  • Retrieval of task data from panda monitor to leverage the server-side cache mechanism
  • Queries of tasks with multiple parameters in show()

Usage

show tasks

Show task summary table:

>>> show()
Showing only max 1000 tasks in last 14 days. One can set days=N to see tasks in last N days, and limit=M to see at most M latest tasks
JediTaskID    ReqID     Status      Fin%     TaskName
________________________________________________________________
  19889136    56785    defined    0%     user.hoppen.Zjets.364106.e5271_s3126_r10201_p3875_mc16d_271119_v1_noSys_t2/
  19889134    56783     ready    0%     user.hoppen.Zjets.364104.e5271_s3126_r10201_p3875_mc16d_271119_v1_noSys_t2/
  19889056    56690    running    0%     user.hoppen.ZZ.361603.e4475_s3126_r10201_p3875_mc16d_271119_v1_noSys_t2/
  19888165    56587    running  33%    user.hoppen.lowMassDY.364215.e5421_s3126_r9364_p3875_mc16a_271119_v1_noSys_t2/
  19888160    56586     throttled    0%     user.hoppen.lowMassDY.364214.e5421_s3126_r9364_p3875_mc16a_271119_v1_noSys_t2/
  19888157    56570      done    100%    user.hoppen.lowMassDY.364211.e5421_s3126_r9364_p3875_mc16a_271119_v1_noSys_t2/
  19888155    56567   scouting    0%     user.hoppen.lowMassDY.364210.e5421_s3126_r9364_p3875_mc16a_271119_v1_noSys_t2/
...

>>> show(days=60, limit=2000)
...

By default, only tasks submitted within last 14 days and at most 1000 tasks are shown.

One can specify "days" and "limit" keyword arguments to show more (or less) tasks. The second example command above will show at most 2000 tasks within recent 60 days.


Show one task with JediTaskID or ReqID:

>>> show(19889136)

JediTaskID    ReqID     Status      Fin%     TaskName
________________________________________________________________
  19889136    56785    defined    0%     user.hoppen.Zjets.364106.e5271_s3126_r10201_p3875_mc16d_271119_v1_noSys_t2/


>>> show(56785)

JediTaskID    ReqID     Status      Fin%     TaskName
________________________________________________________________
  19889136    56785    defined    0%     user.hoppen.Zjets.364106.e5271_s3126_r10201_p3875_mc16d_271119_v1_noSys_t2/


Show multiple tasks with a list of JediTaskIDs or ReqIDs:

>>> show([56783, 56785])

JediTaskID    ReqID     Status      Fin%     TaskName
________________________________________________________________
  19889136    56785    defined    0%     user.hoppen.Zjets.364106.e5271_s3126_r10201_p3875_mc16d_271119_v1_noSys_t2/
  19889134    56783     ready    0%     user.hoppen.Zjets.364104.e5271_s3126_r10201_p3875_mc16d_271119_v1_noSys_t2/

Note: NOT working with a list that mixes both JediTaskID and ReqID !!


Show in long detailed format:

>>> showl()

JediTaskID     Status     CreationDate         ModificationTime     (finished|  failed|   total NInputFiles)
ReqID        Finished_%     TaskName
                         Webpage
______________________________________________________________________________
  19887329      throttled     2019-11-27 09:30:11   2019-11-27 09:54:05      (      0|      0|     45)
  55670        0%     user.hoppen.singleTop.410648.e6615_s3126_r9364_p3895_mc16a_261119_v1_noSys/
                         https://bigpanda.cern.ch/tasknew/19887329/
______________________________________________________________________________
  19887324      throttled     2019-11-27 09:28:28   2019-11-27 09:54:05      (      0|      0|     21)
  55664            0%     user.hoppen.ZZ.361604.e4475_s3126_r9364_p3875_mc16a_261119_v1_noSys/
                         https://bigpanda.cern.ch/tasknew/19887324/
______________________________________________________________________________
...


Show tasks matching certain filters (similar to bigpnada monitor):


>>> show(username='Kevin Nicholas Barends', limit=7, days=30)

JediTaskID    ReqID     Status      Fin%     TaskName
________________________________________________________________
  19889136    56785    defined    0%     user.hoppen.Zjets.364106.e5271_s3126_r10201_p3875_mc16d_271119_v1_noSys_t2/
  19889134    56783     ready       0%     user.hoppen.Zjets.364104.e5271_s3126_r10201_p3875_mc16d_271119_v1_noSys_t2/
  19889056    56690    running    0%     user.hoppen.ZZ.361603.e4475_s3126_r10201_p3875_mc16d_271119_v1_noSys_t2/
  19888165    56587    running      33%    user.hoppen.lowMassDY.364215.e5421_s3126_r9364_p3875_mc16a_271119_v1_noSys_t2/
  19888160    56586  throttled    0%     user.hoppen.lowMassDY.364214.e5421_s3126_r9364_p3875_mc16a_271119_v1_noSys_t2/
  19888157    56570   done         100%    user.hoppen.lowMassDY.364211.e5421_s3126_r9364_p3875_mc16a_271119_v1_noSys_t2/
  19888155    56567   scouting    0%     user.hoppen.lowMassDY.364210.e5421_s3126_r9364_p3875_mc16a_271119_v1_noSys_t2/


Show tasks in other format:

>>> show(limit=2, format='plain')

________________________________________________________________
_pandaserver       :    https://pandaserver.cern.ch:25443/server/panda
_timestamp         :    1573552390
_sourceurl         :    https://bigpanda.cern.ch/tasks/?username=Kevin+Nicholas+Barends&json=1&datasets=True&limit=2&days=14
_weburl            :    https://bigpanda.cern.ch/tasknew/19665576/
jeditaskid         :    19665576
reqid              :    32682
taskname           :    user.kbarends.411160.PhPy8EG.DAOD_BPHY9.e6943_a875_r10724_p3641.2019-11-07.mc16e_elChan.NTUP/
username           :    Kevin Nicholas Barends
creationdate       :    2019-11-06 12:47:14
modificationtime   :    2019-11-07 03:31:19
superstatus        :    done
status             :    done
pctfinished        :    100%
pctfailed          :    0%
nfiles             :    14
nfilesfinished     :    14
nfilesfailed       :    0
________________________________________________________________
_pandaserver       :    https://pandaserver.cern.ch:25443/server/panda
_timestamp         :    1573552390
_sourceurl         :    https://bigpanda.cern.ch/tasks/?username=Kevin+Nicholas+Barends&json=1&datasets=True&limit=2&days=14
_weburl            :    https://bigpanda.cern.ch/tasknew/19665572/
jeditaskid         :    19665572
reqid              :    32676
taskname           :    user.kbarends.411159.PhPy8EG.DAOD_BPHY9.e6943_a875_r10724_p3641.2019-11-07.mc16e_elChan.NTUP/
username           :    Kevin Nicholas Barends
creationdate       :    2019-11-06 12:46:51
modificationtime   :    2019-11-06 14:38:45
superstatus        :    done
status             :    done
pctfinished        :    100%
pctfailed          :    0%
nfiles             :    14
nfilesfinished     :    14
nfilesfailed       :    0


>>> show(limit=2, format='json')

[
    {
        "architecture": "x86_64-slc6-gcc62-opt@slc6", 
        "basewalltime": 600, 
        "campaign": "", 
        "cloud": "", 
        "corecount": 1, 
        "countrygroup": "", 
        ...
        "workinggroup": "", 
        "workqueue_id": 100
    }, 
    {
        "architecture": "x86_64-slc6-gcc62-opt@slc6", 
        "basewalltime": 600, 
        "campaign": "", 
         ...
        "workqueue_id": 100
    }
]



Contact

Contact Email Address: hn-atlas-dist-analysis-help@cern.ch




Responsible:

Never reviewed

Edit | Attach | Watch | Print version | History: r19 < r18 < r17 < r16 < r15 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r19 - 2019-12-09 - TadashiMaeno
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    PanDA All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback