MariaDimouEnd2016MiddlewareReadiness < Sandbox

---+!! Middleware Readiness WG

%TOC%

---++ Mandate

   * To ensure that operation won't suffer when a site is called to upgrade to version X of Middleware (MW) package Y, i.e. that versions are not only *tested* and *certified* internally by the Product Teams (PT), *validated* by the EGI staged rollout but also *ready* to be used in operation at the sites, integrated into the experiment workflows.
   * The Working Group (WG) concentrates on european WLCG sites, as the OSG sites are covered by an experiment expert at the site who takes care of the software to be deployed. Any OSG-EGI inter-operability issues are handled at the *monthly* meeting, (3rd Friday of each month at 16hrs CET/CEST) from where we can draw advice.

---++ Members

Maria Dimou (chairperson 2013-2016), Andrea Manzi (MW Officer), Lionel Cons (Monitoring expert, developer -until March 2016), Maarten Litmaath (advisor & ALICE), David Cameron (ATLAS), Andrea Sciabà (CMS), Marcelo Vilaca Pinheiro Soares (LHCb), Jeremy Coles (!LCG.GridPP), João Pina (EGI staged rollout manager), Peter Solagna (EGI operations manager), Rob Quick (OSG), [Helge Meinhard, Jerome Belleman] (Tier0 service managers), [Ben Jones, Alberto Rodriguez Peon] (!HEPiX configuration WG), [Patrick Fuhrmann, Andrea Ceccanti, Andy Hanushevsky, Gerardo Ganis, Lukasz Janyst, Andreas Peters, Massimo Sgaravatto] (Product Team Contacts). Better [[https://e-groups.cern.ch/e-groups/Egroup.do?egroupId=10108570][check the members of the e-group]] wlcg-ops-coord-wg-middleware@cern.ch. This static listing here may be out of date.

---++ Coming Events
   * No meeting scheduled due to [[https://twiki.cern.ch/LCG/MWReadinessMeetingNotes20161102#MW_readiness_mandate_review_and][the decision at the 19th meeting of the WG on 2016/11/02]]. Work is going on in  [[https://e-groups.cern.ch/e-groups/Egroup.do?egroupId=10108570][the e-group]] and  [[https://its.cern.ch/jira/secure/Dashboard.jspa?selectPageId=15100][jira]].


---+++ Our JIRA tracker 
Please click [[https://its.cern.ch/jira/issues/?jql=project%20%3D%20MWREADY][HERE!]]. Alternative [[https://its.cern.ch/jira/secure/Dashboard.jspa?selectPageId=15100][Dashboard view]].

---++ Meetings

   * 2016/11/02: 19th meeting [[http://indico.cern.ch/e/MW-Readiness_19][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20161102][Minutes]]. This was the last regular meeting.
   * 2016/07/06: 18th meeting [[http://indico.cern.ch/e/MW-Readiness_18][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160706][Minutes]].
   * 2016/05/18: 17th meeting [[http://indico.cern.ch/e/MW-Readiness_17][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160518][Minutes]].
   * 2016/03/16: 16th meeting [[http://indico.cern.ch/e/MW-Readiness_16][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160316][Minutes]].
   * 2016/01/27: 15th meeting [[http://indico.cern.ch/e/MW-Readiness_15][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160127][Minutes]].
   * 2015/12/02: 14th meeting *VIRTUAL* [[http://indico.cern.ch/e/MW-Readiness_14][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20151202][Minutes]].
   * 2015/10/28: 13th meeting. [[http://indico.cern.ch/e/MW-Readiness_13][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20151028][Minutes]].
   * 2015/09/16: 12th meeting. [[http://indico.cern.ch/e/MW-Readiness_12][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20150916][Minutes]].
   * 2015/06/17: 11th meeting. [[http://indico.cern.ch/e/MW-Readiness_11][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20150617][Minutes]].
   * 2015/05/06: 10th meeting. [[http://indico.cern.ch/e/MW-Readiness_10][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20150506][Minutes]].
   * 2015/03/18: 9th meeting. [[http://indico.cern.ch/e/MW-Readiness_9][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20150318][Minutes]].
   * 2015/01/21: 8th meeting. [[http://indico.cern.ch/e/MW-Readiness_8][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20150121][Minutes]].
   * 2014/11/19: 7th meeting. [[http://indico.cern.ch/e/MW-Readiness_7][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20141119][Minutes]].
   * 2014/10/01: 6th meeting. [[http://indico.cern.ch/e/MW-Readiness_6][Agenda]]. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20141001][Minutes]].
   * 2014/07/02: 5th meeting. [[http://indico.cern.ch/event/MW-Readiness_5][Agenda]]. [[https://indico.cern.ch/event/320011/material/minutes/1.pdf][Minutes]].
   * 2014/05/15: 4th meeting. [[http://indico.cern.ch/event/MW-Readiness_4][Agenda]]. [[http://indico.cern.ch/event/314807/material/minutes/][Minutes]].
   * 2014/03/18: 3rd meeting. [[http://indico.cern.ch/event/MW-Readiness_3][Agenda]]. [[http://indico.cern.ch/event/306380/material/minutes/][Minutes]].
   * 2014/02/06: 2nd meeting. [[https://indico.cern.ch/e/MW-Readiness_2][Agenda]]. [[http://indico.cern.ch/event/298170/material/minutes/][Minutes]].
   * 2013/12/12: 1st meeting of the "Middleware Readiness Working Group". [[http://indico.cern.ch/e/MW-Readiness_1][Agenda]]. [[https://indico.cern.ch/getFile.py/access?resId=1&materialId=minutes&confId=285681][Minutes]].

---++ Product Teams

*NB!!* This product list is not exhaustive! Some products were suggested at the [[http://indico.cern.ch/event/MW-Readiness_3][3rd meeting of this WG]], the !LHCb Computing Workshop of  2014/05/22 and  [[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGOpsMinutes140522#Action_list][the 2014/05/22 WLCG Ops Coord meeting]]).

| *Product* | *Contact* | *Test Repository* | *Production Repository* | *Puppet modules' availability? y/n (*)* | *Used by which experiment* | *Tested at site* | *Comments* |
| [[http://www.dcache.org/][dCache]] | P. Fuhrmann | same as prod repo | [[http://www.dcache.org/downloads/IAgree.shtml][dCache site]](recommended versions appear %GREEN%GREEN!%ENDCOLOR%) + EMI repo + EGI-UMD repo | ready but not yet published | All 4 | !LCG.IN2P3, PIC, NL_T1, NDGF, FNAL, BNL, Triumf, RWTH-Aachen (1 of a set of T2s)  |  Green versions mean internal tests are done by the developers as well as functional tests + 2 or 7 day stress tests are done at FNAL as part of the dCache.org collaboration. The sites test with their own configuration the version they intend to upgrade to for a month (not necessary the green ones). |
| [[http://italiangrid.github.io/storm/][LCG.StoRM]] | A. Ceccanti | same as prod repo | [[http://italiangrid.github.io/storm/download.html][LCG.StoRM site]] + EMI repo + EGI-UMD repo | n - still using yaim | ATLAS, CMS, !LHCb | INFN-T1, some T2s, e.g. QMUL and others (INFN Milano T2) | |
| [[http://eos.cern.ch/][EOS]] | A. Peters , L Mascetti | doesn't exist | [[http://eos.cern.ch/cgi-bin/cgit.cgi/eos/][git repo]], [[http://eos.cern.ch/rpms/eos-beryl/][rpm repo]] | y | All 4 | CERN, FNAL, ASGC, SPBSU.ru (ALICE T2)  | ATLAS tests via !Hammercloud |
| [[http://xrootd.org/][xrootd]] | A. Hanushevsky, G. Ganis, L. Janyst | yum testing | yum stable | y - [[https://forge.puppetlabs.com/lcgdm/xrootd][here]] | All 4 | OSG sites (SLAC, UCSD & Duke) and CERN | xrootd is a basic dependency for EOS and also a dependency for CASTOR. It's also used by DPM and FTS3 to extend their functionality and some sites run it on top of GPFS or HDFS. |
| [[https://svnweb.cern.ch/trac/lcgdm/wiki/Dpm][DPM]] | O. Keeble | EPEL testing | EPEL stable | y - [[https://forge.puppetlabs.com/lcgdm/][here]] | All 4 | DPM Collaboration (ASGC, Edinburgh, Glasgow, Napoli...) | DPM needs an extra repo for stuff that doesn't qualify for EPEL, eg yaim or an Argus client. This is currently EMI and could be the WLCG repo in the future, in which case a testing area would be beneficial |
| [[https://svnweb.cern.ch/trac/lcgdm/wiki/Lfc][LFC]] | O. Keeble | EPEL testing  | EPEL stable | y | ATLAS, !LHCb | CERN | Atlas and LHCb have both stated their plan to retire the LFC. For lfc-oracle (used only at CERN) the CERN koji/mash repo is used instead of EPEL.  |
| [[https://svnweb.cern.ch/trac/fts3][FTS3]]  | O. Keeble | EPEL testing | EPEL stable | y - [[https://forge.puppetlabs.com/CERNOps/fts][here]] | ATLAS, CMS, !LHCb | CERN, !RAL (in prod), PIC, KIT, ASGC, BNL (for test) | Non-WLCG VOs snoplus.snolab.ca, ams02.cern.ch, vo.paus.pic.es, magic, !T2K also use FTS3. All sites and experiments take *for the moment!* the code from the grid-deployment afs directory because the fixes are available there much faster than according to the EPEL mandatory cycle. [[https://indico.cern.ch/getFile.py/access?resId=0&materialId=slides&confId=271133][Recent FTS3 presentation]] |
| [[http://italiangrid.github.io/voms][VOMS client]] | A. Ceccanti | same as prod repo | [[http://italiangrid.github.io/voms/download.html][VOMS site]] + EMI repo + MAVEN for java stuff | y - [[https://github.com/cernops/puppet-voms][here]] & [[https://forge.puppetlabs.com/rocha/voms][here]] & [[https://forge.puppetlabs.com/lcgdm/voms][here]]  | All 4 | INFN-T1 | |
| [[http://research.cs.wisc.edu/htcondor/][HTCondor]] | Tim Theisen (HTCondor), Tim Cartwright (OSG Middleware), Rob Quick (OSG Ops) |  [[http://research.cs.wisc.edu/htcondor/downloads][downloads]] (Development Release) | [[http://research.cs.wisc.edu/htcondor/downloads][downloads]] (Stable Release) | y [[https://github.com/opensciencegrid/puppet-contrib/tree/master/condorHepWisc/modules/condor][in OSG]] | ALICE, ATLAS, [[http://www.uscms.org/uscms_at_work/physics/computing/setup/batch_systems.shtml][USCMS]], [[https://twiki.cern.ch/twiki/bin/view/Main/LHCbEdinburghGroupCondor][LHCb]] | OSG (FNAL & ?) | tested internally by HTCondor team, then in production at Univ. of Wisconsin, then by OSG before inclusion in their release. *CMS position:* The testing that OSG and in particular the glideinWMS developers do, is enough. CMS experts discuss with them which version should be deployed on the pilot factories, etc. From this point of view, HTCondor is seen as an experiment service, because of course its usage as batch system is not tested by CMS or the glideinWMS team. So, the feeling is that the current interaction between CMS and the HTCondor team is good enough and there is no strong motivation to set up any new testing system as we do in for the MW Readiness verification of other services. *ATLAS position* [[https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/MiddlewareReadinessWorkingGroup#HTCondor][HERE]] |
| [[https://twiki.cern.ch/twiki/bin/view/CvmFS][CVMFS]] | J.Blomer, P.Buncic, D.Dykstra, R.Meusel  | [[http://cvmrepo.web.cern.ch/cvmrepo/yum/cvmfs-testing/EL/][here]] disabled by default | [[http://cernvm.cern.ch/portal/downloads][cvmfs site]] | y [[https://forge.puppetlabs.com/CERNOps/cvmfs][here]] | All 4 | !RAL &CERN mostly but also BNL, SFU, TRIUMF, !HPC2N, AGLT2, and others. | Technical [[http://cernvm.cern.ch/portal/sites/cernvm.cern.ch/files/cvmfstech-2.1-5.pdf][paper]]. Release [[http://cernvm.cern.ch/portal/filesystem/release-procedure][procedure]]. D.Dykstra runs the OSG tests. |
| [[http://www.nordugrid.org/arc/ce/][ARC CE]] | Balazs.Konya (hep.lu.se) | EPEL testing (but also _final tags_ in OS-specific repos) & [[http://download.nordugrid.org/repos.html][ARC own repos]] for yum & apt _release candidates_ | [[http://download.nordugrid.org/repos.html][Nordugrid web site]], includes prod and test variants | y [[https://github.com/HEP-Puppet/arc_ce][here]] (supplied by UK T2s, untested at CERN) | All 4 | NDGF-T1, !SiGNET, !RAL-LCG2 | D. Cameron is the WLCG ARC expert. Info provided by O.Smirnova |
| !CREAM CE | M. Verlato | [[http://ci-01.cnaf.infn.it/igi-mw/testing/3/][INFN products' testing area]] | [[http://emisoft.web.cern.ch/emisoft/dist/EMI/3/][EMI]], [[http://repository.egi.eu/sw/production/umd/3/][UMD]] | y [[https://github.com/cernops/puppet-creamce][here]] | All 4 | INFN-Padova & [[https://www.egi.eu/earlyAdopters/table][the EGI early adopters]] | Project leader L. Zangrando |
| [[http://gridinfo.web.cern.ch][BDII]] | M.Alandes | [[http://koji.cern.ch/mash/][listed in this index]], separate dir per OS | [[http://gridinfo.web.cern.ch/sys-admins/bdii-releases][gridinfo web site]] | y [[https://github.com/cernops/puppet-bdii][here]]& [[https://forge.puppetlabs.com/lcgdm/bdii][here]] & (soon) [[https://forge.puppetlabs.com/CERNOps/][here]] | All 4 | CERN & many (who?) | [[http://indico.cern.ch/event/309413/material/slides/1.pdf][2014/06/04 Presentation]]. [[https://svnweb.cern.ch/trac/gridinfo/browser/bdii/trunk/tests][Trac dir with test cases]] found, 3 yrs old though... |
| ARGUS | Valery Tschopp (SWITCH) | none? | [[https://github.com/argus-authz/argus-authz.github.com][in github]] & EMI/UMD(repo used by the sites) | y [[https://github.com/cernops/puppet-argus][here]] | All 4? | none regular | Info provided by Andrea Ceccanti on June 10. The future of ARGUS support is brought to the GDB & MB in Sept 2014. Valery & Andrea C. made [[https://twiki.cern.ch/twiki/bin/view/LCG/ARGUSCollaboration][this twiki about its components]] |
| UI | C. Aiftimiei | [[http://ci-01.cnaf.infn.it/igi-mw/testing/3/][INFN products' testing area]] | [[http://emisoft.web.cern.ch/emisoft/dist/EMI/3/][EMI]], [[http://repository.egi.eu/sw/production/umd/3/][UMD]] | n |  All 4 | INFN-Padova | Cristina does the testing |
| WN | C. Aiftimiei | [[http://ci-01.cnaf.infn.it/igi-mw/testing/3/][INFN products' testing area]] | same as the UI | y [[https://github.com/cernops/puppet-glexecwn][here]] & [[https://github.com/HEP-Puppet/puppet-glexecwn][here]] |  All 4 | INFN-Padova | Cristina does the testing |
| gfal/lcg_utils | O.Keeble | EPEL testing and WLCG Application Area AFS space | EPEL stable | n | Atlas, CMS, LHCb | none | being replaced by gfal2 |
| gfal2 (inc gfal2-util) | O.Keeble | EPEL testing and WLCG Application Area AFS space | EPEL stable | n | Atlas, CMS, LHCb | none | Developers are Alejandro Alvarez Ayllon & Adrien Devresse |

(*) Info taken from https://github.com/cernops with help from Ben Jones.
   * List of [[https://wiki.egi.eu/wiki/FAQ_GGUS-QoS-Levels][Supported Products]] with contacts and !QoS levels, maintained by GGUS.
   * (*) Puppet-related modules in [[https://github.com/cernops][THIS LIST]] are pre-fixed _puppet-_. Input by Ben Dylan Jones (CERN/IT-PES & !HEPiX).

---++ Experiment workflows

| *Experiment VO* | *Contacts for this WG* | *Experiment application* | *Middleware product to test* | *Sites where testing* | *Documentation pages* | *Comments* |
| ATLAS | D. Cameron | Panda pilot/DQ2/Rucio | All in the [[MariaDimouEnd2016MiddlewareReadiness#Product_Teams][above table]] | Mostly those where the Product owners work | [[https://twiki.cern.ch/twiki/bin/viewauth/AtlasComputing/MiddlewareReadinessWorkingGroup][The ATLAS twiki on !MiddlewareReadinessWorkingGroup]] | [[https://twiki.cern.ch/twiki/bin/view/AtlasComputing/CentOS7Readiness#ATLAS_statement_about_migration][ATLAS !CentOS7 Readiness document]] |
| CMS | A. Sciabà (temporarily J. Hernandez) | !PhEDEX/CRAB | All in the [[MariaDimouEnd2016MiddlewareReadiness#Product_Teams][above table]] except LFC | Mostly those where the Product owners work | [[https://twiki.cern.ch/twiki/bin/view/CMSPublic/MiddlewareValidation][Middleware Validation twiki]] | |
| ALICE | M. Litmaath | !AliEn | xrootd | ? | ? | Position after the [[http://indico.cern.ch/conferenceDisplay.py?ovw=True&confId=274974][ALICE T1-T2 workshop]] on March 5-7 2014: _A parallel validation infrastructure for Xrootd is desirable, when other priorities recede._ |
| !LHCb | J. Closier | !LHCbDIRAC | all of the above list | all Sites | [[http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/lhcbdirac/rst/html/Certification/tree.html][doc]] | |

---++ _Volunteer_ Sites
   1 The _Volunteer_ Sites' rewarding method proposal was presented to the 2014/03/18 WLCG MB. [[https://indico.cern.ch/event/296397/contribution/4/material/0/][Document HERE]].
   1 Their tasks are listed in the [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness#Procedure_Guidelines_VOs_Sites][Procedure Guidelines for VOs & Sites]] here below.
   1 The first group of _Volunteer_ sites is assembled below for presentation to the 2014/04/15 WLCG MB. Information is taken from the *Documentation* column of the [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness#Experiment_workflows][Experiment workflows]] table.
   1 More UK sites than the ones listed below are keen/available to be involved when needed. E.g. Brunel University undertakes various MW testing as part of LCG.GridPP operations work.
   1 LHCb has a well-established testing procedure. Nevertheless, site names participating in the MW Readiness Verification effort, from within the LHCb workflows do not yet appear in [[http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/lhcbdirac/rst/html/Certification/tree.html][their documentation.]]
   1 ALICE will eventually set-up a parallel validation infrastructure for Xrootd, after having dealt with important Spring 2014 priorities.
   1 The table of _Volunteer_ Sites was presented at the 2014/04/15 WLCG MB [[http://indico.cern.ch/event/302033/contribution/4/material/slides/][Slides]]. The MB approved of this activity and the list of sites. It also  endorsed [[https://indico.cern.ch/event/296397/contribution/4/material/0/2.pdf][the last version of the document]] on _Volunteer_ Sites' rewarding. There was a suggestion on adding a column about the existence of a parallel infrastructure for readiness verification at the site. This column is now added.

| *_Volunteer_ Site* |  *Experiment VO* | *Middleware product to test* | *Experiment application* | *Set-up at the site for the MW Readiness effort* | * Other Comments* |
| TRIUMF | ATLAS | dCache SE | Panda pilot/DQ2/Rucio | Simon wrote on 2014/05/12: For TRIUMF, the setup is done, dcache 2.6.25, dCache upgraded to 2.6.34 on Sep. 25 2014. Di explained that the infrastructure for MW Readiness verification is still with the same site name, of course it's defined as different resource queue in the ATLAS Panda production system.  | Contacts: Simon Liu, Di Qing. Site used by the PT too |
| NDGF-T1 | ATLAS | dCache SE | Panda pilot/DQ2/Rucio | Gert wrote on 2014/05/14: We made a separate installation, but with a configuration similar to the production system. All services are concentrated on a single physical machine (no VM), except for actual storage. A few tens of TBs of storage has been diverted from the production system, but no tape system is attached. Currently running dCache 2.9, but we will deploy new software versions as they become available - sometimes even before officially released.  | Contact: Gerd Behrmann. Site used by the PT too |
| Edinburgh | ATLAS | DPM SE | Panda pilot/DQ2/Rucio | This is a separate installation, with all services are concentrated on a single physical machine (no VM), including actual storage. Around 10TB of storage is available. The machine has the epel-test repository enabled and auto-updates DPM from that. It currently has DPM 1.8.8. | Contact: Wahid Bhimji. Site used by the PT too, as part of the DPM collaboration |
| QMUL | ATLAS  | !LCG.StoRM SE | Panda pilot/DQ2/Rucio | Chris wrote on 2014/05/19: Test SE running  (se01.esc.qmul.ac.uk) with same backend storage  - accessible by production worker nodes. Used for initial testing. se04 used as a production !GridFTP node  - used for production load testing of !GridFTP. se03 - production SE - final production test. | Contact: Chris Walker. Site used by the PT too |
| INFN-T1 | ATLAS  | !LCG.StoRM SE | Panda pilot/DQ2/Rucio | Salvatore wrote on 2014/05/20: At CNAF we're going to set up a VM where the endpoint will be hosted with production-wise resources and it's going to be declared on gocdb. Such test instance will host FE, BE and gridFTP for LCG.StoRM. | Contact: Salvatore Tupputi. Site used by the PT too |
| OSG | ATLAS  | Xrootd | Panda pilot | !RobQ at EGI CF now. We shall have OSG participation at the next meeting. This is not urgent (see also [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness#Mandate][our Mandate]]) | Contact: Rob Quick. Site used by the PT too |
| CERN_T0 | ATLAS  | FTS3 | DQ2/Rucio | Michail's input: A parallel infrastructure exists that runs all new FTS3 versions for 1 month from within experiment workflows before it is considered 'production.  Endpoints (gSoap):  https://fts-pilot.cern.ch:8443, (REST):   https://fts-pilot.cern.ch:8446,  Monitoring: https://fts-pilot.cern.ch:8449. Steve's input: Two FTS3 instances exist at CERN fts3.cern.ch and fts3-pilot.cern.ch The fts3-pilot runs rolling updates of the FTS service, operating system and configuration. At suitable times a current fts3-pilot version is installed on the production fts3 service.The pilot service is continuously tested by CMS and ATLAS.| Contact: Michail Salichos & Steve Traylen. Michail "is" the PT |
| T1_ES_PIC | CMS | dCache | !LCG.PhEDEx, HC, SAM | info is coming a.s.a.p. Site mgrs now at EGI CF & !HEPiX | Contact: Antonio Perez. Site used by the PT too |
| T2_FR_GRIF_LLR | CMS | !CREAM CE/WN, DPM | !LCG.PhEDEx, HC, SAM | Andrea wrote on 2014/05/13: We installed a Preproduction DPM cluster (which is meant to be up to date with latest MW releases). This is a small resources running on VM's, but enough to make tests that  do not involve load. The cluster has been setup to run !LCG.PhEDEx !LoadTest transfers to/from the production storage of T2_FR_GRIF_IRFU. | Contact: Andrea Sartirana. Site used by the PT too, as part of the DPM collaboration |
| CERN_T0 | CMS | !EOS | !LCG.PhEDEx, HC, SAM | CERN/IT-DSS uses a well-sized preproduction service (EOSPPS) for final release validation before deployment. EOSPPS is monitored and configured in the same way as the five production instances: EOSPUBLIC,EOSALICE,EOSATLAS,EOSCMS,EOSLHCB | Contact: Andreas Peters. He is the main PT member. |
| CERN_T0 | CMS | FTS3 | !LCG.PhEDEx, ASO? | Michail's input: A parallel infrastructure exists that runs all new FTS3 versions for 1 month from within experiment workflows before it is considered 'production.  Endpoints (gSoap):  https://fts-pilot.cern.ch:8443, (REST):   https://fts-pilot.cern.ch:8446,  Monitoring: https://fts-pilot.cern.ch:8449. Steve's input: Two FTS3 instances exist at CERN, fts3.cern.ch and fts3-pilot.cern.ch. The fts3-pilot runs rolling updates of the FTS service, operating system and configuration. At suitable times a current fts3-pilot version is installed on the production fts3 service. The pilot service is continuosly tested by CMS and ATLAS. | Contact: Michail Salichos & Steve Traylen. Set-up identical to  the one for ATLAS higher in this table. |
| T2_IT_Legnaro | CMS | !CREAM CE | HC, glidein factory?, SAM |Massimo wrote on 2014/05/13: A virtual machine is allocated where to install the !CREAM CE MW to be verified. Guidelines are needed for this machine installation (e.g. which MW repo should we consider ?, should the CE appear in the site bdii ?). This !CREAM CE will use the same WNs used by !CREAM CEs used in production and the same batch system (!LSF in our case) installation. Should a separate !LSF queue be used for these activities? | Contact: Massimo Sgaravatto. |
| T2_IN2P3_LAPP Annecy | ATLAS | SRM-less DPM | Verifying whether the ATLAS workflow can work with only gridftp+http access. | This is an additionnal, testbed style, DPM SE (lapp-se99.in2p3.fr) , separated from the production instance. Composition: Virtualised separate headnode, a single disk server (physical machine) and 7 TB available. atlas and ops VOs supported. http, xrootd and griftp with redirection are configured. SRM is disabled.  | Contact: Frederique Chollet |

---++ MW versions in production (T0+T1s) & usage of the [[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGBaselineVersions#Baseline_versions_of_services_an][Baseline versions' table]]

This table was produced following a poll to the sites in *May 2014*. It was presented at the [[http://indico.cern.ch/e/MW-Readiness_4][4th meeting of the WG]].

| *Site Name* | * If, How and Where you publish the MW versions you run in production.* | *How you use the Baseline versions' table given that the "baseline version" number doesn't necessarily reflect all individual updates of the packages in the dependencies.* | *Info provided by* | *Comment* |
| CERN | BDII | As a general reference to deploy services for WLCG, we keep up to date our production services with the versions referenced there, if not higher.  | Maite Barroso | |
| ASGC | We publish version information only by BDII. | We basically follow baseline version for service upgrade after we finish internal tests on testbed. | Jhen-Wei Hwang | |
| BNL | Since we are an OSG site, MW versions are captured in the OSG version we run. Info is published via the OSG BDII, which is fed to the WLCG interop BDII. http://is.grid.iu.edu/cgi-bin/status.cgi Content for the T1 is [[http://is.grid.iu.edu/data/cemon_processed_osg/gridgk01.racf.bnl.gov.processed?which=BNL_ATLAS_1][here]] which includes !GlueHostApplicationSoftwareRunTimeEnvironment: OSG 3.2.7 There is also some info about Globus CE, dCache, and SRM versions in the BDII info for those services, e.g. !GlueCEImplementationName: Globus, !GlueCEImplementationVersion: 4.0.6 etc... | We don't use the table. Minimum platform standards get discussed and decided upon between OSG and WLCG, then OSG includes the necessary packages in the OSG middleware release. | John Hover | |
| FNAL | The versions are published in BDII and the twiki page for the WLCG Ops Coordination meeting. | We don't. We rely on OSG where possible to deliver the baseline. | Burt Holzman | |
| JINR | In the BDII via our site BDII | We do not use this table. We follow updates in multiple repositories: EMI, EPEL, dCache, WLCG, so our installed MW usually have a bit newer version than Baseline. In practice, this scheme has advantages and disadvantages, but we have chosen this method for a long time and do not have big problems.  | Valery Mitsyn | We (SCODs) had forgotten to include the site in the wlcg-tier1-contacts e-group till 2014/05/12, this is why  !MariaD emailed the questions late and separately. |
| !LCG.IN2P3 | At CC-LCG.IN2P3  the information system provided by the middleware is used to publish the version in production | The page is checked frequently and the updates are done most of the time following the recommendations. | Vanessa Hamar | |
| INFN-T1 | BDII | Our installed versions are always equal or greater than the ones listed in the baseline. We upgrade from time to time, particularly to fix security problems. | Andrea Chierici | |
| KIAE | Our site BDIIs | We usually don't use it: we're trying to keep up with various recommendations straight from the vendors (EMI, UMD, dCache) and experience gained by other sites or our testbeds.  And we try to be aligned to the WLCG/EGI/VO requirements for the (minimal) needed versions. | Eygene Ryabinkin | We (SCODs) had forgotten to include the site in the wlcg-tier1-contacts e-group till 2014/05/12, this is why  !MariaD emailed the questions late and separately. |
| KISTI | We, KISTI, are not sure whether the MW versions are being published by BDII as well or not but currently we do not publish the information anywhere (we keep the information up-to-date on the internal twiki) | The link has been useful for us e.g. EMI-2 and EMI-3 migration. We usually do not keep the latest version of middleware unless there is a security issue. | Sang-Un Ahn | |
| KIT | As many others we publish the MW versions in BDIIs only. | Almost all our MW services have baseline versions or higher. We perform updates to the recommended versions following the release cycles, but sometimes also upon requests from user communities. The updates come in production after thorough tests and validation by all supported VOs. For our dCache instance we try to stick to a so-called "Golden Release" in order to avoid often upgrades. If you need we could provide exact versions for all services we are running. | Pavel Weber | |
| NDGF | Automatically in the infosystem. The MW versions not being published in the infosys we don't publish anywhere else. | Not at all. | Mattias Wadenstein | |
| NL_T1 (NIKHEF) | Isn't this information now amply published in the BDII on a per-service basis? We anyway do not publish this information publicly anywhere else. | We follow the EGI UMD release cycles, with the caveat that upgrades to services that are actually working and for which no service change requests are explicitly requested by users (including those outside wLCG) are upgraded based on personnel availability. Also any changes are done only after internal validation and re-testing at Nikhef before they are deployed to 'production'. Deployed services are available to all our user communities, which makes EGI the more obvious place to coordinate that through UMD. I personally don't remember we referred to the wiki above after 2009 or so ...| David Groep | |
| NL_T1 (SARA) | For SURFsara, almost the same applies as for Nikhef. We publish our MW versions in the BDII. | We follow the UMD release cycles, with the exception of dCache, which we get directly from www.dcache.org. We regularly report our dCache version on the twiki pages of the WLCG ops coordination meeting. | Onno Zweers | |
| PIC | BDII | We install versions equal or greater than the ones listed in the baseline (with some excepcional cases, which are widely discussed, where interoperability of some underlying services is affected). We promptly react and upgrade, particularly to fix security problems. | Pepe Flix | |
| !RAL-LCG2 | BDII | We don’t really use this twiki page with any regularity. We carry out more or less rolling updates following the MW release cycles. We do pour own testing (and participate in staged rollout and early adopter activities) before deploying new versions of MW to production. | Ian Collier | |
| Triumf | The only place we publish the MW versions is information system (BDII) | We always try to follow the Baseline versions' table. However, we do some tests before upgrading the middleware, and only upgrade to the version listed in  the Baseline versions' table or newer after our tests passed. | Di Qing | Unclear for !MariaD how the individual updates are reflected in the global version number. |

---++ Procedure Guidelines VOs Sites

---+++ Actions for the VOs

   * Each VO to select: 
      * The *MW products* from the [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness#Product_Teams][Product Teams table]] which are relevant to their workflows and which the VO is willing to verify for readiness. Rate them by order of importance.
      * Their *own applications* to be tried with these MW products.
      * _Volunteer_ *sites* (approved by the WLCG MB) where: 
         * The PTs already test their products' new versions before releasing *AND*
         * There are expert VO contacts.
   * The VO clearly lists their above choices in a table. Here is [[https://twiki.cern.ch/twiki/bin/view/AtlasComputing/MiddlewareReadinessWorkingGroup][the ATLAS]] and [[https://twiki.cern.ch/twiki/bin/view/CMSPublic/MiddlewareValidation][CMS]] set-up.
   * The VO writes and maintains a document with concrete instructions to the _Volunteer_ sites on how to configure their parallel infrastructure for readiness verification to use the designated experiment applications with new MW versions. This document should remain linked from the [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness#Experiment_workflows][Experiment workflows table]]. 

---+++ Actions for the _Volunteer_ Sites

   * Each _Volunteer_ Site to: 
      * Set-up an environment, equipped with sufficient amount of space (&gt;=1TB), where validation jobs routinely run. This can be a, parallel to production, infrastructure, but this is not a strict requirement.
      * Find a way for the Monitoring results to appear separately, not to disturb the site availability figures. This could be by giving a separate site name to the validation infrastructure, e.g. the-usual-sitename-Volunteer, if the VO and site processes allow this ([[http://indico.cern.ch/e/MW-Readiness_4][the 4th WG meeting]] in May 2014 showed this is not possible for ATLAS...
      * Take the new client versions (after PT internal testing) from the test trees of the CVMFS area grid.cern.ch. (the WLCG MW Officer will maintain this area).
      * Use !LCG.HammerCloud (HC) for all automatic testing (clients & services).
      * Points concluded at the WG's [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20141119#Summary][7th meeting of 2014/11/19]]:
         * When a MW package version is proved to work via the workflow of a given experiment and a new version is out for verification, other experiments which started later should go directly to the most recent version at hand.
         * The validation of a new version will (need to) have a deadline in practice, beyond which the affected MW may (need to) get deployed anyway e.g. to fix issues experienced by sites.
         * When a product runs at very few sites, it can be considered, de facto, verified, as no large operation risks to suffer from unexpected bugs in production. 

---+++ WLCG MW Officer's Tasks

   1. For WLCG Operations and Coordination effort:
      1. Maintain and verify the baseline versions for the MW components [[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGBaselineVersions#Baseline_versions_of_services_an][baseline table here]]
      1. Communicate with sites not compliant with baseline versions (if needed launch a TF) [[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGMiddlewareVersionCheck][tracked here]] 
      1. Monitor and liaise with EGI and UMD for matters regarding the WLCG MW and versions (issues recorded [[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGBaselineVersions#Issues_Affecting_the_WLCG_Infras][here]])
      1. Maintenance of the WLCG repo (well covered now, maybe future involvement as a back-up)
      1. Management of the Application Area AFS space  (well covered now, maybe future involvement as a back-up) [[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGGridApplicationsArea][Doc by MW Officer Andrea M]]
   2. For the MW Readiness WG:
      1. Discover new versions ready for entering Readiness verification
      1. Make sure that new “release candidate” versions of clients are available in the agreed CVMFS area grid.cern.ch [[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGCVMFSGridArea][Doc under construction]]
      1. Communicate with Volunteer sites to agree which version is tested at which Volunteer site and by which VO
      1. Monitor results of the Volunteer sites to decide on Readiness of a given release being verified
      1. Decide if versions can enter production at a subset of these sites and if testing under load is necessary
      1. Announce to the WLCG community the Readiness Verification results by maintaining a new extended baseline versions table

*wlcg-middleware-officer@cern.ch should be used to contact the MW Officer!* This allows information to be updated in case of absences.

---++ The WLCG MW Package Reporter 
   * [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160518#Pakiti_documentation_enhancement][Documentation update]] following the March 2016 WG meeting. The rest is historical information which explains the rationale behind the development:
   * First presentation at the WG's 4th meeting on 2014/05/15 [[http://indico.cern.ch/event/314807/material/slides/1.pdf][by Lionel Cons here]].
   * Update presentation at the [[http://indico.cern.ch/e/MW-Readiness_5][WG's 5th meeting]] on 2014/07/02, including detailed documentation by Lionel Cons with links to the installation manual [[LCG.MiddlewarePackageReporter][HERE]].
   * Rationale for the non-use of Pakiti prepared on GDB request on 2015/06/11:
      * The use of Pakiti for the needs of the MW Readiness WG has been considered. It has not been selected for the following reasons:
         * Pakiti v.2 can't be used as it is today. It does not scale (as acknowledged on their web site [[http://pakiti.sourceforge.net/pakiti3.html][here]] "current version of the Pakiti reaches its limits").
         * 90% of Pakiti's code is in the GUI which is security-centric and not suited to our needs.
         * The data collection part is not well developed in Pakiti, but it is the part we need most.
         * Nobody actively works on Pakiti now but, Lionel is in touch with its developers, so we hope to converge in the autumn, if Pakiti evolves to cover our requirements. 

---++ Presentations
   * 2015/09/09: MW Readiness WG presentation at the [[https://indico.cern.ch/event/319751/contribution/2/attachments/1151417/1653073/MW-Readiness-GDB-SEP-2015.pdf][September 2015 GDB]]
   * 2014/12/10: MW Readiness WG presentation at the [[https://indico.cern.ch/event/272780/session/0/contribution/6/material/slides/][December 2014 GDB]]
   * 2014/11/27: MW Readiness WG presentation at the [[https://indico.cern.ch/event/336012/contribution/4/material/slides/][CERN IT/SDC Group meeting]] (Access restricted to the group)
   * 2014/07/07: MW Readiness WG presentation at the [[https://indico.cern.ch/event/305362/other-view?view=standard][Barcelona WLCG Workshop]].
   * 2014/06/11: MW Readiness WG status report at the [[https://indico.cern.ch/event/272622/][June 2014 GDB]]. Slides from the agenda and minutes [[https://twiki.cern.ch/twiki/bin/view/LCG/GDBMeetingNotes20140611#MW_Readiness_WG_M_Dimou][here]].
   * 2014/05/22: MW Readiness WG (what it is, what it does) presentation at the [[https://indico.cern.ch/event/278289/#2014-05-22][LHCb Computing Workshop]]. The agenda restricted, slides are [[https://twiki.cern.ch/twiki/pub/LCG/MiddlewareReadiness/LHCb_Comp_Workshop_MW_Readiness_WG_20140522.pdf][attached to *this* twiki]].
   * 2014/02/11: MW Readiness WG status report at the f2f WLCG Operations Coordination Meeting. [[https://indico.cern.ch/event/272784/contribution/4/material/slides/][Slides]]
   * 2013/11/07: A few words about our intention to learn from the DPM certification procedures to report to the [[http://indico.cern.ch/conferenceDisplay.py?confId=280057][WLCG Operations Planning]] meeting.
   * 2013/10/14-18: CHEP [[https://indico.cern.ch/getFile.py/access?contribId=473&sessionId=9&resId=0&materialId=slides&confId=214784][poster]] about sustainable WLCG software life-cycle
   * 2013/07/10: Markus' presentation at the [[http://indico.cern.ch/materialDisplay.py?contribId=4&sessionId=1&materialId=slides&confId=197806][on potential WLCG SW life cycle]] at the July GDB.

---++ Completed Tasks' overview (archive)

| *Task* | *Deadline* | *Progress* | *Affected VO* | *Affected Siites* | *Comment* | 
| Cream-CE and BDII Readiness verification | Middle of November | The latest Cream-CE and Bdii update have been installed at LNL-T2 site ( Legnaro). The configuration for the Job submission is ongoing ( Andrea S is following this up). The latest Cream-ce ha been also installed at INFN-NAPOLI for ATLAS, Panda jobs have been configured | CMS and ATLAS | LNL-T2 and INFN-NAPOLI | 50%. ATLAS installation ready. Waiting for jobs. CMS to verify v. 1.16.4 at Legnaro. |
| dCache  Readiness verification  | Middle of November | Setting up verification workflow at TRIUMF and NDGF for ATLAS ( 2.6.x and 2.10.x version) and PIC for CMS ( 2.6.x). TRIUMF and NDGF installation ready, PIC is ongoing | CMS and ATLAS | TRIUMF, EDGF, PIC | 75% dCache 2.6.35 and 2.11.0 verified. Still not update from PIC. Now verifying 2.6.38 at TRIUMF and 2.11.4 at NDGF |
| STORM  Readiness verification  | Middle of November | QMUL and INFN-T1. QMUL installation ready, INFN-T1 is ongoing. |ATLAS | QMUL and INFN-T1 | 50% |
| MW package reporter integration with Pakiti|  End of January |   the WLCG package reporter has been rebranded after the discussion with EGI security team as Pakiti v3 and released to EPEL ( under review). It will be used both by EGI security team and MW readiness. | n/a | n/a | Packiti v.3 under EPEL review |
| MW Readiness Collector REST API | End of November |  A first design  and prototype are ready | n/a | n/a | Lionel is working on it. Things will change due to new Collector/Reporter-Pakiti integration design. | 
| MW Readiness "dashboard" design and first prototype | End of november |  This effort can start after the tool's redesign following the Collector/Reporter-Pakiti integration discussions of mid-November 2014. | n/a | n/a | Lionel and Andrea M. with Maarten supervision. Lionel is working on the reported info visualisation part ("dashboard") [[https://its.cern.ch/jira/browse/MWREADY-12?jql=project%20%3D%20MWREADY][JIRA:MW12]]. !AndreaM is testing the design of a DB to hold the info of rpms running at the site. [[https://its.cern.ch/jira/browse/MWREADY-15][JIRA:MW15]] |
| MW clients' deployment in grid.cern.ch | a.s.a.p. | On the long run, all clients in the [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness#Product_Teams][PT table]] | LHCb | CERN | Feedback from LHCb, given their experience would be very useful. The work is done by !AndreaM (MW Officer) |
| Action List from the July 2nd meeting | The next meeting October 1st | 20% | ATLAS, CMS, LHCb | OSG, Edinburgh, GRIF, CERN, ? | [[https://indico.cern.ch/event/320011/material/minutes/1.pdf][Read the Actions]] |
| DPM Readiness verification | End of July | The full 1.8.9 release was postponed so it was decided to use 1.8.8+ memcache plugin 1.6.4 recently released. It was successfully installed at GRIF and Edinburgh. At GRIF we have successfully verified the CMS workflows ( Phedex + HC)  and at Edinburgh the Atlas production panda jobs | ATLAS/CMS | DPM Collaboration sites | Done |
| "WLCG MW Package Reporter" | July 2014 | The tool is now installed at Edinburgh, GRIF and LNL-T2. Following their feedback, fixes were applied to improve the code.  | ATLAS & CMS at first | CERN at first | We look forward to CERN running the Package Reporter. Their feedback will be very useful. The work is done by Lionel (MW Package Reporter developer) | 
| Editing & Planning | [[htttp://indico.cern.ch/e/MW-Readiness_5][2 July 2014 meeting]] | 100% | All 4 | the [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness#Volunteer_Sites][Volunteer Sites]] | !MariaD in [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness#Volunteer_Sites][the WG twiki]]. Done | 
| !HTCondor | [[htttp://indico.cern.ch/e/MW-Readiness_5][2 July 2014 meeting]] | 50% | ATLAS & CMS | OSG | !MariaD, Maarten & !AndreaM to prepare with !RobQ and the experiment contacts in the WG (!DaveC, !AndreaS) the Readiness verification procedure for !HTCondor. The ATLAS and CMS position is ready and attached to the !HTCondor line of the [[MariaDimouEnd2016MiddlewareReadiness#Product_Teams][PT Table]]. |
| CVMFS  Client upgrade| End of July | 100% | ALL | ALL | Andrea Manzi opened 75 tickets to sites, the vast majority of them already upgraded and some are planning before the servers upgrade in August.|
| DPM Readiness verification  | Middle of November | DPM 1.8.9 installed at GRIF and Edinburgh | CMS and ATLAS | GRIF and Edinburgh | completed |


---++ Background material:

   * [[LCG.WLCGBaselineVersions][Baseline versions page]] 
   * 2013/11/05: A UK middleware verification tel. meeting (Maarten, Jeremy and the UK sysadmins). 
      * [[http://indico.cern.ch/conferenceDisplay.py?confId=281944][Agenda]]
      * detailed [[http://indico.cern.ch/getFile.py/access?resId=0&materialId=minutes&confId=281944][Minutes]] summarizing the introductory presentation plus the subsequent discussion
   * TEG Operations conclusions  [[https://twiki.cern.ch/twiki/pub/LCG/WLCGTEGOperations/Ops_TEG_finalreport-05042012.docx][Final document 2013/04/05]],

---++ WLCG ops coord reports:

2016-12-01

%STARTSECTION{"20161201"}%

This is the status of  [[LCG.MWReadinessMeetingNotes20161102#Actions][Actions from the 19th MW Readiness WG meeting of 20161102]].
   * 20161102-05: Christoph to investigate EL7 UI testing by CMS. Keep Andrea S. informed as maintainer of the workflow twiki.
   * 20161102-04: Andrea M. to update the [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewarePackageReporter][pakiti documentation]].
   * 20161102-03: Maria to remove the out-of-date _Tasks overview_ from the [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness][WG twiki]]. *DONE* twiki up-to-date and announced on 20161201.
   * 20161102-02: Stefan to appoint a !LHCb member to join the WG. *DONE* Marcello is appointed.
   * 20161102-01: Andrea S. to update the [[https://twiki.cern.ch/twiki/bin/view/CMSPublic/MiddlewareValidation][CMS workflow twiki]].
   * 20160518-02: !EL7 experiments' intentions *Done* via Ops Coord on Sep 1st - see details on the agenda 

This is the status of [[https://its.cern.ch/jira/secure/Dashboard.jspa?selectPageId=15100][jira ticket updates]] since the last Ops Coord of 20161103:

   * [[https://its.cern.ch/jira/browse/MWREADY-104][MWREADY-104]] DPM 1.9.0 SRM-less verification for ATLAS at LAPP Annecy - on-going
   * [[https://its.cern.ch/jira/browse/MWREADY-139][MWREADY-139]] FTS 3.5.7 for ATLAS & CMS at CERN - completed
   * [[https://its.cern.ch/jira/browse/MWREADY-140][MWREADY-140]] ARC-CE 5.2.0 on EL7 for CMS at Brunel - on-going
   * [[https://its.cern.ch/jira/browse/MWREADY-30][MWREADY-30]] ARGUS for CMS at Brunel - no update for a month - status unknown 

Maria Dimou is changing duties in CERN IT and leaves this WG in the competent hands of the MW Officer Andrea Manzi. Grateful to the excellent work by the Volunteer Sites and the experiment contacts!

%ENDSECTION{"20161201"}%

2016-11-03

%STARTSECTION{"20161103"}%

The 19th MW Readiness WG meeting took place yesterday Nov. 2nd. [[http://indico.cern.ch/e/MW-Readiness_19][Agenda]], [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20161102][MInutes]]. Summary:
   * The pakiti client is in cvmfs now. Details [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20161102#WLCG_MW_Readiness_Software_Statu][here]].
   * !LHCb will participate in the FTS verification effort, a way to avoid, as much as possible, suprises like the checksum problem (GGUS:124136) met on Sept. 28th. They will also participate in the verification of the CE and storage types that they use.
   * CMS will discuss internally participation in the EL7 UI bundle/rpm testing.
   * The experiment plans around EL7 migration will be discussed in this WG. Today's situation is, mostly, with the exception of [[https://indico.cern.ch/event/575028/contributions/2344134/attachments/1360911/2059309/20160524_ADC_weekly_centos7.pdf][an ATLAS update]]  as reported at [[https://indico.cern.ch/event/540420/][the dedicated Ops Coord]] meeting of Sept. 1st.
   * The [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness#Mandate][WG Mandate]] was reviewed and confirmed as still valid. 
   * The date for the next meeting is not yet defined Please email [[https://e-groups.cern.ch/e-groups/Egroup.do?egroupId=10108570][the e-group of the WG]] as soon as a vidyo meeting is desirable and to accelerate exchanges in jira. Our tracker is https://its.cern.ch/jira/projects/MWREADY. The [[https://its.cern.ch/jira/secure/Dashboard.jspa?selectPageId=15100][jira dashboard view]] always shows a snapshot of open tickets.
   * Please observe the actions and communicate progress to the e-group.

%ENDSECTION{"20161103"}%

2016-09-29

%STARTSECTION{"20160929"}%

   * The agenda of the 2/11 meeting http://indico.cern.ch/e/MW-Readiness_19 is taking shape and the twiki is reachable from there. Maria will prepare the table of jira tickets' status closer to the date, so please, record all progress in jira or email the e-group wlcg-ops-coord-wg-middleware at cern.
   * WN and UI rpm for EL7 have been prepared ( with the clients/lib available now on EL7) and pushed to UMD preview repo for testing ( [[https://its.cern.ch/jira/browse/MWREADY-135][MWREADY-135]] and [[https://its.cern.ch/jira/browse/MWREADY-128][MWREADY-128]]). Looking for sites available for the validation
   * We'd like to debate at this meeting the future of the WG. It completes 3 years of life in December. Some products are now verified for Readiness "be default" [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160518#ATLAS_workflow_Readiness_Verific][see examples here]]. Other products and 2/4 experiments never embarked this effort. Participation is declining. It is a good moment to review the continuation/transformation/dissolution of the WG. 
   * This idea was circulated in email on 22/9. Alessandra's feedback is the WG should remain alive even if meetings are not very frequent. Example reason: !CentOS7 will require some coordination and it seems you are the bridge with EGI. The MW Readiness jira tickets are useful, e.g. https://its.cern.ch/jira/browse/MWREADY-128 and https://its.cern.ch/jira/browse/MWREADY-135

%ENDSECTION{"20160929"}%

2016-09-01

%STARTSECTION{"20160901"}%
  
   * Not so many activities during the summer holidays
   * [[https://its.cern.ch/jira/browse/MWREADY-133][MWREADY-133]] FTS 3.4.7 verification for ATLAS & CMS completed at CERN
   * [[https://its.cern.ch/jira/browse/MWREADY-136][MWREADY-136]] FTS 3.5.0 for ATLAS & CMS at CERN - on-going
   * [[https://its.cern.ch/jira/browse/MWREADY-135][MWREADY-135]] WN bundle for C7 ( on preparation, most probably ready by the end of the month)
   * [[https://its.cern.ch/jira/browse/MWREADY-137][MWREADY-137]] ARC-CE 5.1.1 on C7 to be tested by Raul at Brunel. The fix for opendap on C7 is breaking the ARC-CE InfoSys
   * Due to summer holidays, WLCG Workshop & CHEP preparations and aftermath, the proposed date for the next meeting is  *Wed Nov. 2nd @ 4pm CET*. Please email the e-group of the WG for comments.
   * We miss very much LHCb participation in the WG meetings and email exchanges. E.g. we know nothing on their !CentOS7 plans, despite 2 reminders during this meeting's preparation.

%ENDSECTION{"20160901"}%

2016-07-07

%STARTSECTION{"20160707"}%

The 18th meeting of the WG took place yesterday. Full minutes [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160706][here]]. Summary: 
   * Old and inactive tickets in jira will be closed by the MW Officer after last verification with Product owners and/or Volunteer site managers. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160706#Verification_status_report][See here which ones]].
   * We need to know about upcoming MW products' releases, we also have a permanent poll open for new Volunteer sites *especially* for MW Readiness verification on !CentOS7. [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160706#Special_topic][See here what we know so far for the near future]].
   * The agenda topic on WG Mandate review and meeting frequency discussion was postponed due to lack of participants in this meeting. For more people joining this effort, we hope to get some publicity at the WLCG Workshop in October. 
   * Due to summer holidays, WLCG Workshop & CHEP preparations and aftermath, the proposed date for the next meeting is  *Wed Nov. 2nd @ 4pm CET*. Please email the e-group of the WG for comments.

%ENDSECTION{"20160707"}%

2016-06-02

%STARTSECTION{"20160602"}%

   * The last WG meeting took place on May 18th see here the [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160518][Minutes]].
   * Based on the T0 and T1 feedback the meeting concluded that the testbeds at the volunteer sites will be the only ones required to run the pakiti client. This was also presented at the May 24th MB.  The need to have a tool that securely reveals what is installed at a site will be followed-up by the [[https://twiki.cern.ch/twiki/bin/view/EGEE/WLCGISEvolution][Information Systems' Evolution TF]]. Detailed replies from sites are present in [[LCG.MWReadinessMeetingNotes20160518#Replies_from_sites_on_the_deploy][here]].
   * Information on the !CentOS7 experiments' intentions is needed. Please prepare for the next meeting on...
   * proposed date July 6th at 4pm CEST.

%ENDSECTION{"20160602"}%

2016-04-28

%STARTSECTION{"20160428"}%
   * [[https://its.cern.ch/jira/browse/MWREADY-122][JIRA:MWREADY-122]] ATLAS  & CMS: CERN FTS 3.4.3 verification completed ( on both SL6 and CentOS7), couple of small issues found to be fixed in 3.4.4
   * [[https://its.cern.ch/jira/browse/MWREADY-123][JIRA:MWREADY-123]] CMS: dCache 2.15.4 verification at PIC ongoing 
   * [[https://its.cern.ch/jira/browse/MWREADY-30][JIRA:MWREADY-30]] Argus 1.7.0 on CentOS7 verification at CERN ongoing. Half of the production cluster is running this new version.
   * Input received so far on the pakiti client installation expansion - by CERN, JINR, !RAL, GRIF and NL_T1-  is now included in [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160518#Replies_from_sites_on_the_deploy][a dedicated section]] for discussion at the next [[http://indico.cern.ch/e/MW-Readiness_17][WG meeting]] of *May 18th*. Then we'll conclude on the issue. 
   * Background: Tier1s were invited at [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160316][the MWR]] and the [[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGOpsMinutes160317#Middleware_Readiness_WG][the Ops Coord]] meetings on 16 & 17/3 to tell the e-group wlcg-ops-coord-wg-middleware at cern.ch whether they agree to install the pakiti client on their production service nodes, so that the versions of MW run at the site be known to authorised DNs site managers taken from GOCDB and expert operations' supporters. The developer Lionel Cons stops further work on the tool. Site replies on their intention to expand the use of pakiti client can be found [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160518#Replies_from_sites_on_the_deploy][here]]. Maria D. will put 2 reminders to sites in the twikis of the April 7th and 28th Ops Coord meetings, as part of the MW Readiness WG report. 

%ENDSECTION{"20160428"}%

2016-04-07

%STARTSECTION{"20160407"}%

Tier1s were invited at [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160316][the MWR]] and the [[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGOpsMinutes160317#Middleware_Readiness_WG][the Ops Coord]] meetings on 16 & 17/3 to tell the e-group wlcg-ops-coord-wg-middleware at cern.ch whether they agree to install the pakiti client on their production service nodes, so that the versions of MW run at the site be known to authorised DNs site managers taken from GOCDB and expert operations' supporters. The developer Lionel Cons stops further work on the tool. Site replies on their intention to expand the use of pakiti client can be found [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160518#Replies_from_sites_on_the_deploy][here]]. Maria D. will put 2 reminders to sites in the twikis of the April 7th and 28th Ops Coord meetings, as part of the MW Readiness WG report. All the input received will be concatenated and attached to the MW Readiness WG [[http://indico.cern.ch/e/MW-Readiness_17][agenda]] of *May 18th*. Then we'll conclude on the issues. 

%ENDSECTION{"20160407"}%

2016-03-17

%STARTSECTION{"20160317"}%
   
   * The 16th MW Readiness WG meeting took place yesterday March 16th 2016. Agenda https://indico.cern.ch/e/MW-Readiness_16, notes LCG.MWReadinessMeetingNotes20160316, Summary:
   * *Tier1s are invited to tell the e-group wlcg-ops-coord-wg-middleware at cern.ch whether they agree to install the pakiti client on their production service nodes, so that the versions of MW run at the site be known to authorised !DNs  site managers taken from !GOCDB and expert operations' supporters.*
   * SRM-less DPM test on-hold until ATLAS pilot code is changed as per [[https://its.cern.ch/jira/browse/MWREADY-104][JIRA:MWR-104]] 
   * Excellent progress with gfal2 testing (various configurations) as per [[https://its.cern.ch/jira/browse/MWREADY-101][JIRA:MWR-101]] & [[https://its.cern.ch/jira/browse/MWREADY-117][JIRA:MWR-117]] 
   * Proposed date for the next meeting is *Wed May 18th at 4pm CEST.*


%ENDSECTION{"20160317"}%

2016-02-18

%STARTSECTION{"20160218"}%

The [[https://its.cern.ch/jira/secure/Dashboard.jspa?selectPageId=15100][JIRA dashboard]] shows per experiment and per site the product versions pending for Readiness verification. Changes since the Ops Coord. meeting of Jan. 21st are:
   * The 15th MW Readiness WG meeting took place on Jan. 27th. Please read the [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20160127#Summary][minutes' summary here]].
   * [[https://its.cern.ch/jira/browse/MWREADY-107][JIRA:MWREADY-107]] CMS and ATLAS: CERN FTS 3.4.1 verification completed, some issues identified and fixed in 3.4.2
   * [[https://its.cern.ch/jira/browse/MWREADY-114][JIRA:MWREADY-114]] CMS and ATLAS: CERN FTS 3.4.2 verification started 
   * [[https://its.cern.ch/jira/browse/MWREADY-108][JIRA:MWREADY-108]] CMS: GRIF-LLR dpm-xrootd 3.6.0 verification completed.
   * [[https://its.cern.ch/jira/browse/MWREADY-109][JIRA:MWREADY-109]] ATLAS: INFN-T1 Storm 1.11.10 verification ongoing, everything looks ok
   * [[https://its.cern.ch/jira/browse/MWREADY-101][JIRA:MWREADY-101]] CMS: GRIF-LLR gfal2 verification, lots of progress on testing gfal2 + Phedex + stageout
   * Suggested date for our WG meeting is *Thursday March 17th 2016 at 15h30pm CET*. *NB!!! Different day-of-the-week & different time!!! The WLCG Ops Coord slot is free. Comments?*

%ENDSECTION{"20160218"}%

2016-01-21

%STARTSECTION{"20160121"}%

The [[https://its.cern.ch/jira/secure/Dashboard.jspa?selectPageId=15100][JIRA dashboard]] shows per experiment and per site the product versions pending for Readiness verification. Changes since the Ops Coord. meeting of Jan. 7th are:
   * [[https://its.cern.ch/jira/browse/MWREADY-99][JIRA:MWREADY-99]] ATLAS  & CMS: CERN FTS 3.4.0 verification completed ( on both SL6 and CentOS7)
   * [[https://its.cern.ch/jira/browse/MWREADY-107][JIRA:MWREADY-107]] ATLAS & CMS: CERN FTS 3.4.1 available on the pilot for verification. 
   * [[https://its.cern.ch/jira/browse/MWREADY-104][JIRA:MWREADY-104]] ATLAS: LAPP Annecy  DPM 1.8.10 SRM-less verification ongoing, found a configuration problem with the new DPM-DSI 1.9.7-1
   * [[https://its.cern.ch/jira/browse/MWREADY-106][JIRA:MWREADY-106]] CMS: CERN EOS 4.0.8-citrine ongoing, fixed permissions for Phedex
   * [[https://its.cern.ch/jira/browse/MWREADY-108][JIRA:MWREADY-108]] CMS: GRIF-LLR dpm-xrootd 3.6.0 verification on going
   * The date of our WG meeting was changed, due to "popular vote" in the e-group. Please *join* our meeting next Wednesday *January 27th 2016 at 4pm CET*. Agenda http://indico.cern.ch/e/MW-Readiness_15. The minutes' twiki LCG.MWReadinessMeetingNotes20160127 is available for content input.

%ENDSECTION{"20160121"}%

2016-01-07

%STARTSECTION{"20160107"}%


The [[https://its.cern.ch/jira/secure/Dashboard.jspa?selectPageId=15100][JIRA dashboard]] shows per experiment and per site the product versions pending for Readiness verification. Changes since the Ops Coord. meeting of Dec. 17th are few due to the year end holidays. Details:
   * [[https://its.cern.ch/jira/browse/MWREADY-61][JIRA:MWREADY-61]] ATLAS: CNAF !LCG.StoRM setup completed, waiting for the release of v 1.11.10 to start verification
   * [[https://its.cern.ch/jira/browse/MWREADY-103][JIRA:MWREADY-103]] ATLAS: Triumf completed the dCache 2.10.47 verification.
   * [[https://its.cern.ch/jira/browse/MWREADY-104][JIRA:MWREADY-104]] ATLAS: LAPP Annecy *NEW Volunteer Site* will be testing DPM without SRM. The [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness#Volunteer_Sites][list of Volunteer Sites]] is up-to-date.
   * [[https://its.cern.ch/jira/browse/MWREADY-102][JIRA:MWREADY-105]] CMS: PIC dCache 2.14.5  verification ongoing 
   * Please *join* our next meeting on *January 20th 2016 at 4pm CET*. Agenda http://indico.cern.ch/e/MW-Readiness_15

%ENDSECTION{"20160107"}%

2015-12-17

%STARTSECTION{"20151217"}%

The [[https://its.cern.ch/jira/secure/Dashboard.jspa?selectPageId=15100][JIRA dashboard]] shows per experiment and per site the product versions pending for Readiness verification. Changes since the Ops Coord. meeting of Dec. 3rd:
   * [[https://its.cern.ch/jira/browse/MWREADY-91][JIRA:MWREADY-91]] CMS: PIC completed the dCache 2.13.12 verification, now switched to dCache 2.14.5.
   * [[https://its.cern.ch/jira/browse/MWREADY-97][JIRA:MWREADY-97]] ATLAS: BRUNEL and GRIF-IRFU completed BDII 5.2.23 verification for CENTOS7. EGI releasing this BDII version in UMD. 
   * [[https://its.cern.ch/jira/browse/MWREADY-99][JIRA:MWREADY-99]] ATLAS & CMS: FTS 3.4.0 verification using the FTS CERN pilot is ongoing
   * [[https://its.cern.ch/jira/browse/MWREADY-102][JIRA:MWREADY-102]] CMS: PIC started dCache 2.14.4 verification. They reported a problem to dCache, already fixed, and they are have now installed 2.14.5
   * [[https://its.cern.ch/jira/browse/MWREADY-103][JIRA:MWREADY-103]] ATLAS: Triumf to start dCache 2.10.47 verification.

In order to push for the transition from lcgutils to gfal2/gfal2-util ( lcgutils is deprecated since 2 years), we have started also to discuss with ATLAS & CMS about the usage of gfal2 in production. Still 99 % of the sites are using lcgutils for stagein/out of data. 
We therefore decided to start verifications of gfal2/gfal2-utils: 
   * [[https://its.cern.ch/jira/browse/MWREADY-100][JIRA:MWREADY-100]] ATLAS: Napoli will verify gfal2 and gfal2-utils. Already did some test demonstrating that everything is working fine.
   * [[https://its.cern.ch/jira/browse/MWREADY-101][JIRA:MWREADY-101]] CMS: Grif is a good candidate to verify gfal2 and gfal2-utils.

Reminder: Next meeting *January 20th 2016 at 4pm CET*. Agenda http://indico.cern.ch/e/MW-Readiness_15

%ENDSECTION{"20151217"}%

2015-12-03

%STARTSECTION{"20151203"}%

The http://indico.cern.ch/e/MW-Readiness_14 meeting yesterday, Dec. 2nd, was virtual. Summary:
   * New MW versions are now being under test via the ATLAS workflow. The dCache v.2.10.44 and v.2.14.0 and HTCondor v.8.4.1 verifications are already completed.
   * BDII v. 5.2.23 is the new MW product being verified at Brunel on !CentOS7 and completed at GRIF-IRFU.
   * The ARC-CE v.5.0.3 verification is completed for CMS.
   * The ARGUS Collaboration met twice since our last meeting: on Nov. 6th and Dec. 2nd.
   * Please comment on the suggested date for the next meeting: Wednesday 20th January 2016 at 4pm CET. Objections with alternative dates should be sent to wlcg-ops-coord-wg-middleware@cernSPAMNOT.ch 

%ENDSECTION{"20151203"}%

2015-11-19

%STARTSECTION{"20151119"}%

   * DPM 1.8.10 installed and verified @ Edinburgh for ATLAS
   * dCache 2.10.44 installed and verified @ TRIUMF for ATLAS
   * EOS testing @ CERN is paused. The new version Citrine has been installed in pre-prod, but is not yet ready for testing.
   * BDII verification on CENTOS7 will start next week @ Brunel
   * MW readiness app v0.3 deployed in prod https://wlcg-mw-readiness.cern.ch/releasenotes/
   * Next meeting Wednesday 2nd December at 4pm CET.
 
%ENDSECTION{"20151119"}%

2015-11-05

%STARTSECTION{"20151105"}%

Summary of the 13th WG meeting held on 28 October: 

   * The xrootd monitoring plugin for  dCache v. 2.13.x was installed and tested at PIC together with dCache 2.13.9
   * One host of the CERN FTS-3 pilot will be running !CentOS7. In this way ATLAS and CMS will be able to test FTS3, via their workflows, on both OS environments.
   * DESY has been contacted  to arrange ATLAS and CMS tests on their nightly rebuilt dCache instance
   * PIC-CERN !LCG.PhEDEx transfers failing are not yet understood; they are possibly due to a bug in one of the involved MW components (EOS, dCache, FTS-3, Globus, ...) or a misconfiguration somewhere. Experts at CERN are looking into this.
   * The MW Readiness App v.3 will move to production real soon now. Volunteer Sites will be called to comment on its functionality.
   * There was an ARGUS Collaboration meeting on Oct. 9th. The next one will be on Nov. 6th. The periodical sudden bursts of high load on the CERN ARGUS servers still persist and are not yet explained. A number of other issues from which CMS suffered are now understood. There is one more FTE now working in the ARGUS dev. team. Moving to the upcoming release for !CentOS7 may help solve issues, if any, due to historical dependencies: the latest builds use more recent versions of =jetty= etc.
   * Less than usual participants joined this meeting from the Volunteer Sites. Suggested date for the next one is Wednesday 2nd December at 4pm CET. Objections with alternative dates should be sent to wlcg-ops-coord-wg-middleware@cernSPAMNOT.ch 

%ENDSECTION{"20151105"}%

2015-10-22

%STARTSECTION{"20151022"}%

   * This [[https://its.cern.ch/jira/secure/Dashboard.jspa?selectPageId=15100][JIRA dashboard]] shows per experiment and per site the product versions pending for Readiness verification. Details:
      * [[https://its.cern.ch/jira/browse/MWREADY-84][JIRA:MWREADY-84]] CMS: at Brunel ARC-CE 5.0.3 test stalled - no jobs displayed.
      * [[https://its.cern.ch/jira/browse/MWREADY-85][JIRA:MWREADY-85]] ATLAS: at NDGF Rucio tests fail since ipv6 has been enabled. so dCache verifications are pending at NDGF
      * [[https://its.cern.ch/jira/browse/MWREADY-70][JIRA:MWREADY-90]] ATLAS: at TRIUMF dCache 2.10.42 verification completed.
      * [[https://its.cern.ch/jira/browse/MWREADY-70][JIRA:MWREADY-91]] CMS: at PIC dCache 2.13.9 verification ongoing.
      * [[https://its.cern.ch/jira/browse/MWREADY-87][JIRA:MWREADY-87]] ATLAS: at Edinburgh for DPM 1.8.10 verification - site manager silent.
      * [[https://its.cern.ch/jira/browse/MWREADY-82][JIRA:MWREADY-82]] ATLAS: at GLASGOW DPM 1.8.10 CentOS7 verification - missing tests' set-up.
      * [[https://its.cern.ch/jira/browse/MWREADY-81][JIRA:MWREADY-81]] CMS: at CERN EOS 0.3.129-aquamarine installed on the PPS - verification pending.
   * Issue found:
      * [[https://its.cern.ch/jira/browse/MWREADY-89][JIRA:MWREADY-89]]. When moving transfer tests to the FTS CERN Pilot ( with a different gfal2 configuration w.r.t  FTS@RAL ) we discovered an issue with DPM deployed with gridftp redirection. Reported to the devs
   * Reminder: Next meeting *October 28th at 4pm CET*. Agenda http://indico.cern.ch/e/MW-Readiness_13

%ENDSECTION{"20151022"}%

 
2015-10-01

%STARTSECTION{"20151001"}%

   * A puppet-pakiti module configuring the pakiti-client cron with the parameters needed for WLCG MW Readiness is available at https://gitlab.cern.ch/wlcg-mw-readiness/puppet-pakiti
   * !LCG.StoRM testing for ATLAS in process at INFN_T1. Failing jobs under investigation. [[https://its.cern.ch/jira/browse/MWREADY-61][JIRA:MWREADY-61]].
   * The !dCache 2.13.x xrootd monitoring plugin has been prepared by the developer Ilija Vukotic, tested by PIC  and pushed to the  WLCG repo.
   * DPM 1.8.10 verification for CMS completed at GRIF.  [[https://its.cern.ch/jira/browse/MWREADY-83][JIRA:MWREADY-83]].
   * DPM 1.8.10 verification on !CentOS7 started at GLASGOW. Now arranging a test set-up for the ATLAS workflow [[https://its.cern.ch/jira/browse/MWREADY-82][JIRA:MWREADY-82]].
   * EOS 0.3.129-aquamarine verification pending for CMS at CERN [[https://its.cern.ch/jira/browse/MWREADY-81][JIRA:MWREADY-81]].
   * Next meeting reminder *October 28th at 4pm CET*. Agenda http://indico.cern.ch/e/MW-Readiness_13

%ENDSECTION{"20151001"}%


2015-09-17
   
%STARTSECTION{"20150917"}%
The WG met yesterday. Full minutes in LCG.MWReadinessMeetingNotes20150916. Summary:
   * The new DPM version is being tested via the ATLAS workflow by the Edinburgh Volunteer site.
   * Many new sites showed interest to participate in MW Readiness testing with !CentOS7. It is useful to anticipate the MW behaviour in the event of new HW purchase. DPM validation on !LCG.CentOS/SL7 is already ongoing at Glasgow.
   * ATLAS and CMS are asked to declare whether the xrootd 4 monitoring plugin is important for them or not. As it is now, it doesn't work with !dCache v. 2.13.8
   * Despite the fact that FTS3 runs at very few sites we decided to test it for Readiness. In this context, ATLAS & CMS are now using the CERN FTS3 pilot in their transfer test workflows. 
   * PIC successfully tested !dCache v.2.13.8 for CMS.
   * CNAF has obtained Indigo-DataCloud effort to strengthen the ARGUS development team. The ARGUS collaboration will meet again early October. The problems faced at CERN with a CMS VOBOX are being investigated in ticket GGUS:116092.
   * The next MW Readiness WG vidyo meeting will take place on *Wednesday 28 October at 4pm CET.*

%ENDSECTION{"20150917"}%

2015-09-03

%STARTSECTION{"20150903"}%
   
   * Not so much to report about verifications cause not many MW versions were made available during the summer
   * Development of the MW Readiness App  continued in the past weeks, with graphical enhancements to the site, ssb integration, access to host packages, dev version is at https://mw-readiness-dev.cern.ch/ ( CERN only)
   * We would like to remind to volunteer sites that the deadline to move to the new pakit-client + server conf has passed ( end of August) and we still miss some upgrades. See instructions sent by mail in July at:
      * https://twiki.cern.ch/twiki/pub/LCG/MiddlewareReadiness/upgrade-instructions-07072015.txt 
   * An agenda of our next (16/9 at 4pm CEST) meeting is on page http://indico.cern.ch/e/MW-Readiness_12. Please send additional items to the e-group wlcg-ops-coord-wg-middleware at cern...
   * An update on the WG activities will be presented during the next week GDB.

%ENDSECTION{"20150903"}%


2015-07-30

%STARTSECTION{"20150730"}%

   * Thanks to Edinburgh and Grif for offering to verify selected MW products for Readiness over !CentOS7/SL7 this autumn. Progress/comments via  [[https://its.cern.ch/jira/issues/?jql=project%20%3D%20MWREADY][our JIRA tracker]].
   * A provisional agenda of our next (16/9 at 4pm CEST) meeting is on page http://indico.cern.ch/e/MW-Readiness_12. Please send additional items to the e-group wlcg-ops-coord-wg-middleware at cern...

%ENDSECTION{"20150730"}%

*PAST REPORTS*
%TWISTY{ mode="div" }%


2015-07-16

%STARTSECTION{"20150716"}%
   * The ATLAS and CMS Volunteer sites are active verifying new MW versions, especially [[https://www.dcache.org/downloads/1.9/release-notes-2.13.shtml][dCache v.2.13.3]] and !LCG.StoRM.
   * The MW Officer Andrea M. is now polling sites' interest in testing  !CentOS7/SL7 versions of MW.
   * The new pakiti-client v.3.0.1 is now available. It contains a new tag to distinguish publishing of packages run for MW Readiness. See explanation in [[https://its.cern.ch/jira/browse/MWREADY-67][jira ticket MWREADY-67]]. All documentation linked from [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness][our twiki]]. Direct link [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewarePackageReporter][HERE]].
   * Next vidyo meeting on Wed September 16th at 4pm CEST. 
%ENDSECTION{"20150716"}%

2015-07-02

%STARTSECTION{"20150702"}%
   * Argus Future meeting tomorrow July 3rd, at 11am CEST, focused on progress with EL7 support [[https://indico.cern.ch/event/406078/][Agenda]].
   * Latest version of pakiti-client (v3.0.1) with tag support has been pushed to EPEL stable, we will contact soon the volunteer sites for the upgrade.
   * !dCache testing for ATLAS at Triumf paused to perform  a re-configuration that will fix a problem  with the SRM space token.
   * PIC and Brunel are collaborating for !PhedeX tests. This allows PIC to better compare one site against another.
   * Reminder: The next MW Readiness WG vidyo meeting will take place on *Wednesday September 16th at 4pm CEST.* Please comment a.s.a.p. if this date is not good!
%ENDSECTION{"20150702"}%

2015-06-18

%STARTSECTION{"20150618"}%
   * The 11th meeting of the WG took place yesterday.
   * A lot of good work is on-going from most Volunteer sites. 
   * Special credit is due to  Edinburgh and GRIF for their detailed DPM testing on behalf of ATLAS and CMS respectively.
   * Similarly, great effort is invested by Triumf and NDGF for multiple !dCache versions testing for ATLAS and from PIC for !dCache testing for CMS.
   * Fine-tuning configuration at !CNAF for !LCG.StoRM testing for ATLAS.
   * New pakiti-client version 3.0.1 is imminent in EPEL Stable. The [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewarePackageReporter#Installation][updated documentation]] is available to all Volunteer Sites, together to a new configuration file to be used due new PKG DB servers deployment. This new pakiti-client version gives the possibility to specify a tag ( --tag option).  MW Readiness nodes should start publishing their packages with the tag *MWR*. Andrea M. will contact the sites for this upgrade. 
   * The MW Readiness App https://wlcg-mw-readiness.cern.ch/ is now available on a production instance. Check [[https://wlcg-mw-readiness.cern.ch/baseline/current/][here]] the Baseline MW versions' mgnt view.
   * EL7 support and the move to Java 8 are now urgent for ARGUS. The CERN testbed will be available real soon now for testing under heavy load and other scenarios.
   * The next MW Readiness WG vidyo meeting will take place on *Wednesday September 16th at 4pm CEST.* Please comment a.s.a.p. if this date is not good!

%ENDSECTION{"20150618"}%

2015-06-04

%STARTSECTION{"20150604"}%
   * The _MW Readiness App_ moved to production node https://wlcg-mw-readiness.cern.ch/. Remember [[http://indico.cern.ch/event/392665/contribution/3/material/slides/1.pdf][this presentation']] from the last WG meeting on May 6th. This tool will, eventually, replace today's, manually maintained, Baseline table and more.
   * The pakiti client is installed on the !dCache CMS instance for MW Readiness at PIC. The nodes are correctly published in the MW Package Collector and viewable from authorised people only.
   * *NB!!* Next vidyo meeting on June 17th at 4pm CEST. Draft agenda http://indico.cern.ch/e/MW-Readiness_11

%ENDSECTION{"20150604"}%

2015-05-21

%STARTSECTION{"20150521"}%
   * Very good work in the Readiness verification for dCache v.2.10.28 by Triumf. This is how this version is now part of the Baseline.
   * Progress is being made for the !LCG.StoRM verification for ATLAS at CNAF. The [[https://twiki.cern.ch/twiki/bin/view/AtlasComputing/MiddlewareReadinessWorkingGroup#Adding_test_endpoints_to_ATLAS][workflow documentation]] is at hand. All relevant experts in sync and follow-up is in [[https://its.cern.ch/jira/browse/MWREADY-61][jira ticket MWREADY-61]].
   * The MW Officer records in [[https://its.cern.ch/jira/browse/MWREADY-59][jira ticket MWREADY-59]] progress on the _MW Readiness App_ development. Remember [[http://indico.cern.ch/event/392665/contribution/3/material/slides/1.pdf][this presentation']] from the last WG meeting on May 6th. This tool will, eventually, replace today's, manually maintained, Baseline table.

%ENDSECTION{"20150521"}%

2015-05-07

%STARTSECTION{"20150507"}%
   * The 10th MW Readiness meeting took place yesterday May 6th [[http://indico.cern.ch/e/MW-Readiness_10][Agenda]].
   * After one year of full Readiness Verification activity, the WG is making a check-point of goals and priorities.
   * ATLAS and CMS were invited to review their [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness#Experiment_workflows][workflow twikis]]  for possible changes in the MW products to verify.
   * !LHCb and ALICE are invited to declare if and for which products they plan to contribute to the MW Readiness WG.
   * Possible stress tests were discussed, for products already verified from within experiment workflows. The decision was to leave this to the MW Officer, the experiment involved and the site on a case by case basis, as per the original definition in [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness#WLCG_MW_Officer_s_Tasks][the WG documentation, point 2.5.]].
   * EOS test for CMS at CERN just started.
   * ARGUS testbed at CERN is set-up and ready to start.
   * NDGF, PIC, CNAF and Triumf are reminded to install the pakiti client [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewarePackageReporter][Instructions]].
   * The [[http://indico.cern.ch/event/392665/contribution/3/material/slides/1.pdf][presentation on software being developed]] for our activities intends to show *product versions* being under test and their matching to the relevant rpms and, in addition, an automated way to display Baseline versions instead of the [[https://twiki.cern.ch/twiki/bin/view/LCG/WLCGBaselineVersions#Baseline_versions_of_services_an][current, manually updated, table]].
   * The next vidyo MW Readiness WG meeting will take place on *Wednesday June 17th at 4pm CEST* 

%ENDSECTION{"20150507"}%

2015-04-23

%STARTSECTION{"20150423"}%
   * Please remember the [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20150318#Actions][actions from our 18 March meeting]]. 
   * Work by the MW Officer Andrea Manzi and the Pakiti client (Package Reporter) expert developer Lionel Cons is starting on the collection and visualisation of product versions' installed per Volunteer Site. Details in [[https://its.cern.ch/jira/browse/MWREADY-56][JIRA:56]]
   * Please give input to Maria Dimou for the preparation of the next vidyo meeting on *Wed May 6th at 4pm CEST*.

%ENDSECTION{"20150423"}%


2015-04-02

%STARTSECTION{"20150402"}%

Excellent progress made by the Volunteer sites for selected MW product versions, as planned at our last meeting on March 18th. Thanks to the MW Officer Andrea Manzi for testing with the sites and to the pakiti client developer Lionel Cons for the good technical collaboration with the other EGI-funded experts. Details:
   * !dCache problem found in v.2.10.18 is solved in v.2.10.23. !dCache versions 2.11.14 and 2.12.3 also contain the fix.Triumf is now testing !dCache 2.10.23  while NDGF 2.12.3. ( both for ATLAS). 
   * !LCG.StoRM 1.11.8 has been verified at QMUL for ATLAS.(only one small issue found)
   * !CREAM-CE 1.16.5 is tested at INFN-Napoli for ATLAS.
   * EOS is in the pipeline for testing at CERN for CMS. Please add news in [[https://its.cern.ch/jira/browse/MWREADY-40][JIRA:MWREADY-40]].
   * Please do the [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20150318#Actions][Actions from our March 18th meeting]].
   * Remember *Wednesday May 6th at 4pm CEST* is the next MW Readiness WG meeting date.

%ENDSECTION{"20150402"}%


2015-03-19

%STARTSECTION{"20150319"}%

   * The [[http://indico.cern.ch/e/MW-Readiness_9][9th WG meeting]] took place yesterday. The Summary is available [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20150318#Summary][HERE]]. 
   * Please book your calendars for the next meeting *on May 6th at 4pm CEST at CERN and with vidyo!*.

%ENDSECTION{"20150319"}%


2015-03-05

%STARTSECTION{"20150305"}%

   * Verified Middleware since the previous meeting:
      * Storm 1.11.6 for ATLAS
      * dCache  2.11.8 for ATLAS
      * DPM-Xrootd 3.5.2 both for ATLAS and CMS
   * Ongoing verifications:
      * CREAM-CE 1.16.5 for CMS
      *  Storm 1.11.7 for ATLAS
      * dCache 2.10.18 for ATLAS ( possible issue found)
      * dCache 2.12.0 for ATLAS 
   * tests under setup
      * ARC-CE 5.0.0 ( under release ) for CMS
   * Remember our next meeting in a month *Wed March 18th at 4pm CET*. Please check the [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20150121#Actions][Action List]].

%ENDSECTION{"20150305"}%

2015-02-19 
%STARTSECTION{"20150219"}%

   * Thanks to the new sites to installed the MW Package Reporter. Instructions [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewarePackageReporter][here]]. 
   * The more Package Reporter installations we have, the better we can prototype and suggest a functional results' display view.
   * Very good progress is being made at QMUL for the !LCG.StoRM Readiness verification via the ATLAS workflow.
   * The MW Officer is working on the MW database view that will fetch and display candidate for Readiness verification release versions from the repositories.
   * Remember our next meeting in a month *Wed March 18th at 4pm CET*. Please check the [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20150121#Actions][Action List]].

%ENDSECTION{"20150219"}%

2015-02-05

%STARTSECTION{"20150205"}%

   * Full minutes of the last MW Readiness WG meeting are now availabe from LCG.MWReadinessMeetingNotes20150121
   * Thanks to the MW Officer, ATLAS, CMS and the _Volunteer Sites_ a lot of progress is being made for all [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20150121#Actions][Actions]], namely:
      * Condor-G verification is in the pipeline via the ATLAS pilot factory. [[https://its.cern.ch/jira/browse/MWREADY-39][JIRA:MWREADY-39]]
      * Brunel and Liverpool Universities offered to participate in the Readiness verification of the ARC CE. [[https://its.cern.ch/jira/browse/MWREADY-37][JIRA:MWREADY-37]].
      * The MW Officer starts investigating with ATLAS & CMS the use of the Prometheus test system, a small dCache instance (currently a single node), offered by DESY, for people who want to help test whether the next major release of dCache (currently 2.12) has any problems. [[https://its.cern.ch/jira/browse/MWREADY-36][JIRA:MWREADY-36]].
      * !LCG.StoRM 1.11.6 is in the pipeline for testing at QMUL. [[https://its.cern.ch/jira/browse/MWREADY-18][JIRA:MWREADY-18]].
      * Grif starts DPM testing with xrootd 4 for CMS.
      * !dCache 2.11.8 testing for ATLAS starts at NDGF [[https://its.cern.ch/jira/browse/MWREADY-38][JIRA:MWREADY-38]]. Triumf will probably soon follow.
      * EOS testing entered the MW Readiness activity for CMS [[https://its.cern.ch/jira/browse/MWREADY-40]]JIRA:MWREADY-40]].
      * Thanks to the _Volunteer Sites_ which installed the new version of the MW Package Reporter. More are very welcome. 

%ENDSECTION{"20150205"}%

2015-01-22
 
%STARTSECTION{"20150122"}%

   * The MW Readiness WG met yesterday Jan 21st. Agenda http://indico.cern.ch/e/MW-Readiness_8
   * Excellent participation and follow-up by the Volunteer Sites (Edinburgh, Napoli, Legnaro, QMUL, CNAF, Triumf, NDGF) and the MW Officer Andrea Manzi. Please follow the [[http://indico.cern.ch/event/354284/contribution/2/material/slides/1.pdf][slides]] for details.
   * The new version of the Package Reporter is ready, within the deadlines. The new design principles are in line with EGI security requirements. A maximum of code shared with Pakiti. The site is offered configuration options for the reporting. Please follow the presentation [[http://indico.cern.ch/event/354284/contribution/3/material/slides/0.pdf][here]] by the developer Lionel Cons for details. Very simple installation instructions are documented [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewarePackageReporter#Installation][here]].
   * Next meeting *Wed 18 March at 4pm CET*. Please note!


%ENDSECTION{"20150122"}%

2014-12-18
 
%STARTSECTION{"20141218"}%

   * The MW Readiness WG status presentation by the MW officer at the [[https://indico.cern.ch/event/272780/session/0/contribution/6/material/slides/][December 2014 GDB]] gives the status of the effort up to last week.
   * A slow-down in participation by sites is observed. 
   * The MW Readiness WG will participate in the ARGUS testing under load and/or with peculiar CA attributes. 

%ENDSECTION{"20141218"}%

2014-12-04
 
%STARTSECTION{"20141204"}%

   * DPM 1.8.9 verification completed, the bug discovered on dmlite lib has been fixed and dmlite 0.7.2 is on EPEL stable now.
   * dCache 2.11.0 verification  for ATLAS  completed
   * dCache 2.6.38 and 2.11.4  verifications for ATLAS are ongoing
   * Progresses can now be followed also on the MW readiness [[https://its.cern.ch/jira/secure/Dashboard.jspa?selectPageId=15100][ jira dashboard]]
   * the WLCG package reporter has been rebranded after the discussion with EGI security team as Pakiti v3 and released to EPEL ( under review). It will be used both by EGI security team and MW readiness.
   * Our [[https://twiki.cern.ch/twiki/bin/view/LCG/MiddlewareReadiness#Tasks_overview][Tasks overview]] is updated.
   * Full minutes of our last meeting are now [[https://twiki.cern.ch/twiki/bin/view/LCG/MWReadinessMeetingNotes20141119][published here]]. Please observe the actions.

%ENDSECTION{"20141204"}%

   * [[%ATTACHURL%/upgrade-instructions-07072015.txt][upgrade-instructions-07072015.txt]]: Mail with instructions for the pakiti-client upgrade and new server conf</verbatim>
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pptx	GDB_20140610_MW_Readiness_WG.pptx	r1	manage	2876.7 K	2014-06-10 - 16:47	MariaDimou	These slides are also in indico - 2014/06/11 GDB
pdf	LHCb_Comp_Workshop_MW_Readiness_WG_20140522.pdf	r3 r2 r1	manage	394.6 K	2014-05-22 - 09:57	MariaDimou	MW Readiness WG presentation for the LHCb Computing Workshop
pptx	LHCb_Comp_Workshop_MW_Readiness_WG_20140522.pptx	r2 r1	manage	2871.5 K	2014-05-21 - 18:57	MariaDimou	MW Readiness WG presentation for the LHCb Computing Workshop (.pptx)
txt	MW-Notes_Maarten_autumn2013.txt	r1	manage	2.6 K	2014-05-20 - 19:06	MariaDimou	Original notes on this effort by Maarten autumn 2013
txt	MW-Officer_Maria_Feb2014.txt	r1	manage	1.3 K	2014-05-20 - 19:10	MariaDimou	1st draft on WLCG MW Officer's tasks by Maria February 2014
png	MWR-architecture.png	r1	manage	38.9 K	2014-09-23 - 18:22	MariaDimou	MW Readiness architecture RPM Collector, its DB and the Package Reporter part.
pptx	WLCG_Workshop_Barcelona_20140707_MW_Readiness_WG.pptx	r2 r1	manage	2882.5 K	2014-07-04 - 10:57	MariaDimou	Slides to be uploaded to the WLCG Barcelona Workshop agenda https://indico.cern.ch/event/305362
pdf	WLCG_Workshop_Barcelona_20140707_MW_Readiness_WG_a.pdf	r1	manage	179.0 K	2014-07-04 - 19:34	MaartenLitmaath	Barcelona slides v1.1
pptx	WLCG_Workshop_Barcelona_20140707_MW_Readiness_WG_a.pptx	r1	manage	2880.5 K	2014-07-04 - 19:34	MaartenLitmaath	Barcelona slides v1.1
Topic revision: r209 - 2016-12-12 - MariaDimou
Account
- Log In
- Cern Search
- TWiki Search
- Google Search
Sandbox All webs
Edit
Attach
Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback