Summary of GDB meeting, November 12, 2014 (CERN)


Introduction - M. Jouvin

Future GDBs

  • March: need to decide asap for a GDB outside CERN. 2 offers received : ASGC (Taipei) and Amsterdam (EGI/NIKHEF). Feedback requested:
    • Are GDB outside Europe considered acceptable?
    • ASGC propose to host GDB during ISGC (co-location), the week after the normal GDB week. May be a good occasion to foster Asian contacts. A few of us going there anyway. But is it affordable (in particular from the budget point of view) to have an Asian GDB in March followed by a WLCG workshop in Japan (Okinawa) in April)?
    • More discussions on the mailing list in the coming days...
  • April: WLCG workshop before CHEP, GDB cancelled in April as it would be 3 days before

Future pre-GDBs (and post-GDB!)

  • Busy December
    • 2-day pre-GDB on HTCondor: developers and OSG experts present, for sites using HTCondor or considering to use it (not a tutorial on HTCondor). Please register if interested.
    • 1-day post-GDB (December 12): ARGUS future and support. For developers and security experts. Registration required
  • January
    • pre-GDB on Data Management: originally scheduled in December, postponed because of opportunity to organize the HTCondor pre-GDB
    • May be a post-GDB as a F2F meeting of the cloud traceability WG
  • Several topics on the list for pre-GDBs in the following months
    • Cloud issues (February/March)
    • Accounting: cloud/volunteer computing challenges
    • Volunteer computing (Spring)

Cloud traceability WG

  • Objective: do a gap analysis and propose incremental concrete solutions, compatible with current manpower
  • #13 people expressed their interest in participating, including OSG
    • Ian Collier accepted to lead it
  • Bootstrapping to happen soon: probably initially by email/Vidyo as there is no possibility for a F2F meeting before January

Actions in progress

  • perfSonar: sites must reinstall their instance asap, if not yet done
  • IPv6: T1s are requested to join the WG
    • Also still looking for more sites moving their production endpoints to dual stack
  • ginfo (GLUE2 client) testing: status ? feedback ?
  • H2020 VRE project: in progress, not yet clear if we'll be able to meet the deadline (January 14, 2015)
    • Includes DP activities but wider scope

New action: clarify of the relevance of gstat for WLCG

  • Tests obsoleted by glue-validator run as part of SAM
  • Not supported and in fact reporting false errors with WebDav endpoints
  • Can it be ignored by sites
  • Probably need a MB discussion/statement

Data / Storage

DPM Workshop Summary - O. Keeble

DPM Collaboration working pretty well

  • Most testing now undertaken by partners: CERN concentrates on development
  • A few partner contribution in term of development: admin and rebalancing tools (UK)

SRM: now WebDav emerged as a viable alternative for many metadata operations

  • S. Campana: ATLAS identified last week that WebDav for deletion, despite (or because) WebDav is (much) more efficient than SRM, it results in namespace high spike load
  • F. Furano: behaviour expected as a result of improvement performance! But happy to work on making possible to implement some throttling for this kind of use case

Several sites reports with different focus

  • UK: new backend support (HDFS, Lustre), ARGUS support
  • Italy: high availability in a distributed T2 context
  • France: full AAA/FAX participation, struggling with draining and balancing
  • Australia: part of Canada dynamic federations

Last DPM release (1.8.9): focus on sysadmin improvements

  • IO monitoring a la xrootd, using a dmlite profiler plugin
    • Monitors every activity going through the dmlite (almost everything except SRM)
    • dmlite profiler can be fed into GLED collectors
  • MySQL memcache now the recommended config
  • Puppet the official configuration tool, YAIM no longer supported
    • Can be used without a Puppet master
    • Distributed on puppetforge
  • gridftp redirection: based on "delayed passive mode", disabled by default
    • Requires gfal2 client: old clients can be penalized
    • Sites encouraged to enable it... and monitor it!
  • 20x improved performance of max file opening rate in http
    • No more errors as it was the case in 1.8.8
  • Improved draining and rebalancing

Xrootd federations

  • Can be configured by Puppet
    • vomsxrd must be configured to pass the full VOMS information
  • CMS file opening/reading tests: interesting results showing some sites who need some tuning
    • Not a site stress test per se

Future: main directions

  • Minimise maintenance costs: remove/minimize legacy code
  • SRM-less operation: need to develop an alternative for space reporting
    • Discussed with dCache and StoRM
    • RFC 4331: DAV extensions adding to new DAV props

P. Solagna: sites must be warned not to install 1.8.9 before deployment of the new SAM SRM probe else they will fail the (critical) test

  • New SAM probe deployment expected by the end of the month

Cristina requests better gfal2 documentation now that gfal/lcg_utils is deprecated.

  • Point taken

LHCb Experience with Dynamic Federations - F. Furano

Dynamic federation: frontend to present the contents offered by a set of endpoints

  • Without indexing or metadata persistency
    • 2 level caching: memory and memcache (memcache shareable between federators)
  • Reflects what is actually accessible
    • Handle properly multiple replica for a files through metalink
    • Multiple replica can be ordered by "geographic distance"
    • Federator requires only READ access to metadata
  • Can aggregate any HTTP/DAV server, including S3 backends
    • S3 access can be authenticated through the federation if the federator has the S3 keys
  • Can be used with browsers or applications
    • Including (X509 authenticated) file download from a browser
  • Can interact with a source catalogue (e.g. LFC) if needed to discover the namespace
    • Can be a static provider of replica SURL/TURL but accessibility will be checked in real time before being used
    • DynFed availability will be the underlying catalogue availability: better to avoid it if not needed
    • Can even be mounted in the federation namespace...

Early tests started by LHCb since September: aggregation of all LHCb storage elements

  • Current structure of LHCB namespace very clean and easy for a DynFed: no need for interacting with a catalogue
    • The same file has the same name on all sites, modulo a site prefix
  • Currently: 13 SEs out of 19
  • Not a replacement for catalogues: they are complementary objects
    • Federation is dynamic and reflects what is accessible by design
    • Catalogues are static and reflects what should be available

Xrootd and DynFed: Xrootd4 supports http/WebDav access and can be federated by the DynFed

  • Already tested

DynFed future development

  • Next RC in the coming weeks: smarter site detection, write support, logging, monitoring
  • WebFTS integration
  • Integration as a DataBridge in the BOINC infrastructure, using the S3 backend
  • Working on a Rucio-friendly prototype


  • How does translation of SRM TURLs happen as they are unknown until interacting with the storage element?
    • The slide would have been clearer, it's more SURL than TURL, but from the federator point of view SURL and TURL are the same thing: it's just a string to manipulate.

EGI Future Services and WLCG - P. Solagna

EGI core services

  • Accounting
  • Security: coordination mainly
  • Operations Portal
  • GGUS and associated 1st/2nd level incident management
  • UMD and staged rollout
  • SAM monitoring

Most relevant for WLCG : Security and UMD/Staged rollout

SLA define service targets in term of availability

  • Approved by EGI OMB
  • Actual availability proved to be significantly more than targets for most of the services

Operations and maintenance of the services is supported through EGI fees and contributions from the partners

  • Adding new capabilities requires new funding sources... thus specific projects
  • EGI-Engage is mostly about support of new communities but also includes a few developments for operation tools
    • Accounting: storage, cloud
    • GOCDB: support new requirement, extended data model to support new resource types
    • Operations Portal: improved VO management tools, integration of most important GStat features

New services planned by EGI-Engage

  • Service Registry: all the services offered by EGI
  • Marketplace: ability to acquire services through payment or for free, with associated SLAs

Other development projects around EGI (not necessarily H2020 projects)

  • Cloud platform projects: PaaS/SaaS
  • Data preservation on demand
  • Extension of EGI AppDB in collaboration with ELIXIR to discover datasets

Collaboration between EGI and WLCG

  • Do we need specific effort to gather requirements specific to WLCG
  • Participation to OTAG meetings dedicated to specific Ops tools and development
  • Accounting ? Staged rollout ? Operation tools?
  • Cloud: what possible collaboration between EGI FedCloud and WLCG?
    • Data services (preservation, open data...)?
    • Cloud broker?
  • Do we need a specific meeting/workshop with WLCG?


  • When will you get news on EGI-Engage
    • Early February, no news yet. Project expected to be funded, if accepted, since 1st of March.
  • Michel: many questions asked of wlcg, we need to answer them in particular about closer collaboration. Probably need to identify more specific topics along the ideas mentioned to move forward...
    • Maarten: collaboration already there, formally in places like the OpsCoord and informally
    • Peter: in EGI-Inspire there was lots of effort for ops tools, in EGI-Engage there will be less, that's why we need input in prioritisation.
    • Michel: WLCG probably doesn't have strong requirements/dependencies on Ops tools except accounting and staged rollout. WLCG decided to have its own monitoring infrastructure for example. MW baseline activities is one area of possible increased collaboration.
    • Maarten: in fact, already having a good collaboration between EGI staged rollout and WLCG MW Readiness, and we hope to push (semi-)automatically info to EGI from our validation efforts.
    • Michel: not in favor of a dedicated meeting at this point, propose to have follow-up discussions at GDB first. Let's iterate to better identify possible topics.

Actions in Progress

OpsCoord Report - A. Sciaba

WLCG operations survey to understand operational costs: questionnaire almost ready

  • Last day for feedback is today! But still possible in the next few days...
  • Already some feedback received and taken into account
  • Expectation of site answers rather than individual answers
  • Would like to have a good fraction of sites answering

WLCG Critical Services list must be reviewed

  • Current list is 2 years old
  • Some changes in the the urgency/impact
  • Some decommissioned services (e.g. WMS) and new services need to be covered (e.g. T0 Agile Infrastructure)

Experiment news

  • ALICE: high activity level, successful with new VOMS servers
  • ATLAS: Rucio/ProdSys2 commissioning successful, migration about to start

WLCG repo

  • RPMs now signed
  • lcg-utils obsolete and removed from the base line table, but not from any repositories


  • Integration into PanDA completed

SHA2 VOMS servers

  • ATLAS, CMS and LHCb still have to complete the migration of their core services to the new VOMS servers
  • LHCb: new VOMS servers open today to the world
  • ATLAS and CMS: work in progress, no problem expected
    • Should not be more than a problem of configuration update

Machine/job feature TF almost completed

  • Converged on an implementation for clouds
  • Final report at next GDB


  • Recommendation to deploy perfSonar in dual-stack

MW readiness

  • More SW covered: HTCondor, EOS, FTS3, dCache, StoRM
  • Developing a MW Readiness Dashboard


  • All sites must install 3.4+ asap
  • Network metrics will be soon available with instructions on how to use them in production

Multicore jobs - A. Forti

Passing job parameters to BS would simplify queue configuration and allow backfilling

  • CEs: ARC CE already ready, HTCondor just started to be discussed, most work on CREAM CE so far
    • CREAM CE current setup is legacy from EGEE, designed for publication of resources into BDII, many min values in BDII converted to max values in queue configurations
    • Local scripts needed are BS dependent without any good reason: already very similar for Torque and SGE, work in progress to merge them
  • Batch systems: Torque/MAUI is currently the most used BS
  • Question: how to express requirements to CEs. Currently GLUE1, do we need to move to GLUE2?
    • ARC CE and HTCondor don't use GLUE to express parameters
    • CREAM CE uses a _Min and _Max suffix to parameter names to express the requirement: is it applicable only to GLUE1?

Started to look at params available in different BS and useful: converged on 5 parameters

  • Core count, RSS, VMEM, CPU time, Wallclock time
  • SLURM and LSF not review yet

Memory monitoring/limiting remains difficult: VMEM no longer means RSS+swap in modern kernel, difficult to obtain/limit RSS+swap usage with standard tools (ps, ulimit)

  • Even worst with multicore jobs as shared memory is accounted multiple times
  • Some sites oversubscribing memory: particularly appropriate with multicore jobs, recipes available for MAUI and HTCondor
  • cgroups look as an attractive alternative: more accurate monitoring, smart soft limits without creating unused allocations
    • Easy to use with HTCondor
    • SLURM and UGE able to use it
    • No support in Torque/MAUI and SoGE/OGE
  • smaps can be an alternative for Torque/MAUI sites but this is not a standard tool
    • smaps can also be used by (pilot) jobs for pro-actively reacting to memory limits being reached and trigger termination of misbehaving applications, shipping all the information necessary for further debugging back before the batch system removes everything


  • Michel - how does smaps compare to cgroups?
    • Alessandra: smaps works everywhere, cgroups doesn't. If ATLAS can implement this limitation in application it's worth looking at. Else need to write our own implementation of a limiter...
    • Maarten: smaps is for monitoring, cgroups is for constraining. Both are useful. No coexistence problem.
  • Michel: as batch system support is so variable for cgroups, it should be folded into the batchsys discussions. At some point we have to balance the cost of writing a memory limiter compatible with our apps for Torque/MAUI with the cost/benefit of encouraging migration to something else. HTCondor pre-GDB may help to make progress on this.
    • Jeff: VMEM issues are linux general, not specific to Torque/MAUI. NIKHEF doesn't see problems for 'normal single core' apps and the batch sys does fine.
    • Alessandra: the difference is much worse for multicore
    • Alessandro dG: there are nevertheless ATLAS workflows on single core which suffer from NIKHEF approach, we have some jobs which go over their 20 GB of VMEM for a short period, it would be bad if they were killed.
    • Michel: must take into account that the vision is that multicore will be a significant part of the workflow, this has to be solved.
    • Simone: ATLAS now only uses 64bit builds which increases the discrepancy between rss and vmem. Limiting on vmem thus becomes more problematic.
    • Philippe: note that killing on rss limit makes you more vulnerable on an empty machine than a busy one. LHCb jobs have variable use of memory, with temporary peak and then lower usage after. Started to experiment with cgroups at RAL: looks promising.
  • Philippe: why cgroup are batch system dependent instead of being implemented globally at the system level?
    • Michel: cgroup are per-process, the batch system has to define the cgroup limit for the application, not necessarily the same for every job.
  • Michel: we are slowly converging despite another lively discussion! Details on use cases and exact problems must be discussed in the TF and summarized at one of the future GDB.

Computing Resource Provisioning

Summary of pre-GDB on Volunteer Computing - N. Hoimyr

Good attendance: ~30 local, 10 remote

Motivations for volunteer computing: mainly about getting volunteers, i.e. getting additional "free" resources

  • Require a significant outreach effort, so no completely free
  • Can also address some other use cases like opportunistic use of clusters

Review of volunteer computing MW : BOINC and XtremWeb/XWHEP

  • BOINC the main MW, several projects collecting huge resources
  • XWHEP: developed at LAL, used in smaller deployments, some distinctive/advanced features, in particular regarding security and authentication

LHC@home experience reported

  • Long experience with BOINC (since 2004) for SixTrack simulation (beam dynamics)
  • Mastered the challenge of maintaining the SW on several platforms
  • Currently 125K users registered: 10% actually contributing resources which is already a good result

Test4Theory@home: another early CERN use case that explored the use of virtualized environment to execute user payloads

ATLAS@home: started 1 year ago as a R&D project, using an ARC CE between PanDA and BOINC

  • ARC CE in charge of data management and all operations requiring grid credentials
  • grid credentials not sent to the volunteer PC
  • 2000 users already signed up despite the absence of significant outreach activity...
    • ATLAS setting pretty high requirements (CPU, mem) on volunteer PC: many volunteers cannot actually contribute
  • Already the 28th simulation site in ATLAS
  • BOINC just used to instantiate VMs

CMS@home: recent R&D based on the Test4Theory architecture but using a DataBridge to do data management and operations requiring a grid credential

  • DataBridge playing the role of the ARC CE in ATLAS@home
  • DataBridge is no more than http Dynamic Federation + mod_auth_sql to authenticate against BOINC
  • Testing with CRAB just started
  • Could provide the basis for a generalize vLHC@home platform

LHCb: several early attempts at using volunteer computing but not enough manpower to fix the issues preventing moving to production

All LHC experiments rely on CERNVM + CVMFS

  • ATLAS using precached application from CVMFS to optimize instantiation time, currently at the price of one image per SW version
    • Not foreseen as a major problem as used for low priority jobs that last several months, using the same SW version

Volunteer computing seen as an extension of the cloud resources used by experiments

  • Should use these resources before going to commercial cloud providers
  • Cost of supporting volunteer computing is minimal: mainly the outreach effort that could be shared by all users of a generalized platform

Secure Hybrid Cloud Infrastructure for Scientific Applications - S. Toor

DII-HEP: Datacenter Indirection Infrastructure for Secure HEP Data Analysis

  • Helsinki Institute of Physics + Computer Science Department of Univ. of Helsinki

Initial resources

  • Several CREAM CEs
  • Private cloud based on OpenStack + GlusterFS (used as the image store backend) + CVMFS
    • ARC CE for providing grid interface
  • Recently added the new CSC cloud in Kajaani

Test case : CMS analysis

  • So far reached 200 concurrent jobs on the cloud: 83% CPU efficiency
  • Performance analysis: 4% loss observed with virtualized environment
    • No other performance issues (e.g. higher I/O wait times) were seen with the CMS jobs so far
    • WNs are connected to the storage at 10 Gbps, which helps the I/O performance
  • GlusterFS (use to boot VMs): performance for a small number of VMs comparable to a local FS but remains constant when increasing the number of VMs starting simultaneously where it grows up significantly with a local file system
  • Most results presented at CHEP 2013

Implementing an elastic solution: based on EES (Execution Environment Service) from EMI Argus

  • Allow to bootstrap a VM on demand, through the grid interface, with the user grid credentials
  • Work done by John White, based on an initial implementation for OpenNebula
  • Development completed but not yet ready for production

To implement a secure cloud setup, used the Host Identity Protocol (HIP) from the mobile networks

  • HIP: provides a secure mechanism for IP multihoming and mobility (used for VM migration)
    • Help to build a secure multi-tenant, federated cloud infrastructure
  • Supports both IPv4 and IPv6
  • One daemon (hipd) in all the VMs: no modification required to the cloud MW/infrastructure or applications
  • hipd communication between several sites using IPSEC
  • Large scale test done: 10k simultaneous CMS jobs, mostly CPU bound but with access to remote data through xrootd
    • 99% success rate
    • No significant CPU penalty or increased network activity seen neither on CE, nor on WNs

Future work

  • finalizing the production version of EES plugin for OpenStack
  • Replacement of Jade cluster by a cloud setup

Cloud Adoption in WLCG - L. Field

Document work done with the following objective

  • Share information between VOs
  • Identify the missing bits and recommendations
  • Tool inventory

Document is ready for consultation

  • Laurence is the editor
  • Read it and send feedback!
  • A more detailed report at one of the next GDB, based on this feedback


  • Simon: what are the main gaps?
    • Laurence: in particular the accounting needs to be working before we can do large-scale deployment; fair share and quotas also need work, but are more in the area of optimization

HEP SW Foundation Progress Report - P. Mato

Since April workshop (see April GDB), 10 White Papers and the formation of an Interim Foundation Board (iFB)

  • iFB: White Papers authors + a few enthusiastic people!
  • Quickly converged on a bottom-up approach, inviting project to join
    • Build service based on needs emerging from discussions
  • Focus on scientific SW: data processing from detectors to physics publication
    • Still discussions on the exact scope
    • Cannot address everything at the beginning, exact scope will depend on project engagement
    • Handle packages at different stages, from well established packaged not developed anymore to R&D projects could improve the current SW base

Lightweight, transparent, open organization

  • Small startup team established, evolving membership
    • Meeting every week but minutes sent out almost immediately to the wider community
    • Monthly meeting with the wider iFB
  • Many (encouraged) informal discussions
  • Currently focusing on people/individuals: will start to deal with projects and organization at a later stage

Main activities done

  • Synthesis of the White Papers: the main achievement
    • Version 1.0 released yesterday: comments welcome!
    • Not a sum of all White Papers but a qualitative analysis and a proposal from startup team for a startup plans
    • Startup plan is not yet the foundation plan but a proto-plan to be able to make a real plan!
  • Started contacts with projects
    • Major goal: get feedback from projects on the startup plan and longer-term ideas
    • MC generator community identified as a major community who didn't attend the 1st workshop
    • Contact established with major projects like GEANT4 and ROOT
  • Established a website and communication channels (Google groups)
    • Attention paid to make the HSF official communication independent of institutes/organizations thus Google
    • Main place for communication exchange: news, meeting summaries, ...
    • First round at a structured inventory of SW packages relevant to the community
    • Want to foster contribution to the web site
    • Main email list open to everyone interested: registration possible with a message to (with subject = 'subscribe', no requirement for a Google group)
    • Also created a wider list, not restricted to HSF activities, not restricted to HSF activities:
  • Plan/prepare next workshop at SLAC (January 20-21)
    • Structured after main topics in White Papers Synthesis
    • Get feedback from the community, people and projects
    • Discuss new initiatives that could be launched under HSF umbrella
    • Agree on next steps
    • No need to commit to be a member of HSF
    • Importance of wide SW participation
    • Pre-announcement just sent out and must be circulated as much as possible


  • the HSF is on purpose a loosely-coupled foundation for now
  • explicit funding is not needed now, but might be later for certain activities
  • the HSF does not own or manage any contributing project, it just tries to coordinate between all the stakeholders
  • the HSF is open to community engagements
  • the covered SW normally is open source by construction
    • Fluka might thus not be included
  • a common license may be achievable
  • companies might be able to join under some conditions, that remains to be seen
Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2014-11-16 - MichelJouvin
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback