Summary of July GDB, July 18, 2018



Introduction - I. Collier


  • Main topics
  • Brief discussion of next WLCG-HSF Workshop early next year
    • Maarten noted the risks to travel with US east coat at that time
    • problem is over constrained
    • Now we consider the week commencing January 28th and the week commencing February 25th.
    • further discussion at next GDB

Nordugrid 2018 Conference (Oxana Smirnova)


  • Watch announcements for ARC6 training event(s)
  • Machine learning - interesting to look 10 years. Maybe moving to dedicated facilities for particular workloads e.g. analysis. Will middleware providers
    • machine learning often based on database which is not the (file-based) processing model so changes inevitable?
  • SPARK/Hadoop maybe non-traditional data layout where no 'jobs' as have been defined to date.

DPM and DPM workshop report (Fabrizio Furano)


  • Note on legacy code support end-date Jun 2019
    • Upgrade DPM to DOME flavour please.
  • JA:do quota tokens guarantee space?
    • Sum of token allocations may be less than total capacity - allocated by admin
    • 'write to free space' hides difficulties - QT's are precise and repeatable.
    • can be used for quota space accounting
  • ML: 'srm no longer needed'. Imagine that standard clients also moving away - could, for example do everything with gfal client
    • Goal is to have stability. Advise to move away from srm, not kick it off
    • e.g. big deletions are no longer faster via srm
  • ML: can use gfal client exclusively and under the hood there will be no srm - correct statement? A: yes.
  • LHC exp will have to discover the details, need to work with other communities to help move away.
  • Now there is a possibility to take srm away, how to move away for agreement …..
  • AF: to say 'not kick it away' is a subtlety, should not assume on an upgrade, need to push a little.
  • ML: need to close srm 'chapter' sometime - maybe early next year for at least our MoU sites to move to more to sustainable architecture - not before end run-2
  • IC: decision to set up task force to co-ordinate this
  • IC: what about security vulnerabilities on legacy branch?
    • Depending on effort necessary
    • IC: need a policy/statement for clarity
    • Likely that security patches accepted but on case-by-case basis.
    • ML: have in the past stated "it's unsupported" - but sometimes produced patch.
  • What was the statement the recent security vulnerability
    • ML: perhaps not handled very well - need to make it clear that cut will be necessary eventually
  • who can make this statement?
    • ML: push through EGI and WLCG - reminders spring next year. IC: spring too late for June.
    • IC: does not need decision here and now - start pushing earlier than last year


Authorisation WG report and discussion (Hannah Short)


WLCG Authorisation Requirements

Summary of the progress of the WLCG AuthZ WG during the pre-GDB on Tuesday July 17th

Description of the WG background, the activities and the next steps

Motivation : evolving landscape, federated identities, token based authorisation, GDPR, shielding the user from the x509 complexities

Outputs of the WG :

  • catalogue of JWT usage in infrastructures (document existing use and identify best options for WLCG) -> final changes pending
  • WLCG authorisation requirements - ready for comments, experiments are invited to comment, interviews of experiments are planned (the technology stack is known, the WG need to understand how token could fit in the experiment workflows)
  • WLCG common JWT profiles
    • ongoing
  • WLCG AAI pilot development
    • ongoing, two solutions appear to meet the majority of requirements EGI check-in and INDIGO IAM, both approaches need additional developments

Questions and comments

Q : slide 16, are the profile of the WLCG JWT token embedded ?

  • Hannah : that may have a significant overlap, they could be added one to the other

Q : it seems like a huge list of requirements ? have you prioritised ?

  • Hannah : not really. The requirements have been put together by the dev.

C : you said you are missing the input from the experiments

  • Hannah : we have started the gathering of the experiment requirements at the November 2017 pre-GDB, at least one experiment in the room, we need a real endorsement

Q (Ian) : next step : going to the computing coordinator ?

  • Hannah : it is a matter of priority on their side, having interviews is really the way to go

C (Dave) : we have to discuss more the CERN SSO we are in the process of cleaning up the CERN DB along the way we may encounter a lot of technical problems with the GDPR, we might lose access to the cern DB though there is a possibility we may have access to the groups from the cern DB

Hannah : it is an action of me, to check if we can get access to the group information of the CERN DB

C (Andrea) : if we could get membership info (person-experiment) from the cern SSO directly, this would simplify a lot ; we would streamline integration and avoid user being blocked ; the way to port this to cern SSO is not clear and this needs to be investigated in any way, for the new system, the admin must have a way to override the DB to avoid issues, like suspending a user ; we have to lay down the details and have feedback from users (site admins) on the overriding

C (Dave) : the HR DB knows who belong to which experiment , it is the reference and it has always been

C (Andréa) : we could override temporarily

C (Dave) : many other communities share the same thing it is very good that the JWT will be worked through with ARC

Hannah : we are early in this process hopefully our work will be valuable for AARC

Dave :

  • it is important that the experiments try the pilots
  • the technology exist, we need to know how to use it, like the refresh of the tokens

Andrea :

  • it will take some time to tune so that it works as well as certificates
  • this will need monitoring and other tools, too early to talk about it
  • now, we have two solutions that could be used, we need an active participation from the experiments and their feedback

Hannah : and we need to identify these people

Andrea : yes to have in the end the transition path and to make this infrastructure a reality

Dave : they really need to look at it, we are designing a new security infrastructure

Ian : we will look forward for future updates in a few months time

HSF: Update on R&D and activities (Graeme Stewart)


Graeme presents an review of the activities in the HEP Software Foundation. Some highlights:

  • CWP Papers: new articles in arXiv (“Careers and training” and “Machine Learning”)
  • Physics Event Generators: a re-engineering workshop is planned, possibly at CERN towards the end of the year
  • Intention to form new WGs to solved the HL-LHC problem: Simulation, Reconstruction and Analysis (based on the groups writing the CWP); do-ocracy spirit, feel free to participate ; these areas will be reviewed by the LHCC next year
  • SW Forum relaunched ; topics:
    • Already discussed: DD4hep geometry modelling package,
    • meeting 18th July: VeCcore and SOACiontainer ;
    • to come HEP analysis in the Python ecosystem and Tracking SW;
    • happy to receive suggestions for other topics
  • role of Python in HEP: more and more used, first workshop (PyHEP) organised right before CHEP, good attendance ; idea to build up a community and a workshop serie


Ian: good progress in the last 6 months

Oxana: about Python, what role does it play ? we should not leave C++

Graeme: if you do things in pure python, this is not the best in the world ; Python is good to glue things ; HEP will not become pure python, we will always need compiled code but python provides good benefits to our community.

Markus: I agree with Oxana regarding Python, C++ best suited for underlying code and python is good to glue things

Markus: the graph on slide 13 is misleading, it doesn’t say how big is the market

Graeme: Python is important for our community, I don’t propose to re-write everything in Python

Markus: can you say more about which concrete steps you envision to federate the schools ? it is difficult to find an appropriate school for students

Graeme: several school organisers were attending the Naples workshop and we started to discuss ; since this event, we had other issues and we will return to this discussion after the summer break

Markus: it is an important topic

Graeme: you should get involved

Ian : it is also notable that in the SKA talk this morning they recognise the effectiveness of the HSF.

Benchmarking WG activities and plans (Domenico Giordano)


In the last GDB (June) the WG presented the results of the pre-GDB. Today, the presentation highlights the work of the last few years and future plans. It covers:

  • the build of a new tool to run all benchmarks
  • the fast benchmark investigations -> they can’t replace long running benchmarks in procurement tasks
  • the study of the SPEC CPU2017: it is highly correlated (linear) with HS06 score
  • the comparison of HS06 and WLCG workloads with interaction counters and memory usage examples (ALICE and LHCb payloads don’t scale anymore with HS06, it still agrees within 10% for ATLAS and CMS)
  • the current plan for the future: the short term goal is to create a benchmark based on the SW suites from the experiments ; until it is finalised and packaged for distribution we keep the HS06 benchmark.
  • the study of spectre/meltdown impact on WLCG workloads

——— Questions and comments

Liz: you have to give the suite to the vendors, he runs the script and you get the number ; you said it was cumbersome, why ?

  • Domenico:
    • this must be usable by sites and HW vendors;
    • the input files are difficult to get and we are missing files like conditions data ;
    • it is also difficult to put this in continuous integration ;
    • I was in touch with cvmfs people, the idea is to snapshot this ;
    • they are working on that, just this summer. If we go for the experiments SW suite, the experiments will create and publish the containers and we will get to package the containers.

Liz: we at Fermilab are working on the squid caching solution

Liz: most experiments are running continuous integration, could you use that ?

  • Domenico: we have to attract these people in the WG, 90% is there, we need to solve the rest: conditions data

Liz: there are some other questions to be raised: shall we use a thick container ? with cvmfs mounted inside ? a layer container approach might be better to do ; the SFT group got some people from ATLAS and CMS

  • Domenico: I will try to follow the group meetings

Domenico: our main tasks is to compare benchmarks and experiments SW instead of building the suite itself

Helge: the WG consider the future benchmark as a mix of experiment workloads ?

  • Domenico: yes

Helge: I don't imagine experiments being willing to maintain the SW on a long term ; if the experiments notice a deviation we must be prepared to change the suite ; this needs a very careful thinking. The third party benchmark HS06 has done us well for 10 years.

Markus: it never represented our workflows

Domenico: I don't understand why we benchmark our resources with a suite that does something different than our payloads ; it is much better to benchmark with our own workloads ; we just want to fix how to distribute it. We will be able to change the benchmark when we think it is needed, not when the third party changes.

Comment (Helge ?): they are good aspects when there is an official spec, like HS06, we gave to the 3rd party all the burden of the maintenance, and now it comes on your side.

Ian: there is no easy way ; the fact that we can bundled this in container makes things different from the past

Helge: agreed

Ian: also, we are foreseeing different kind of resources to come in the landscape, like GPU, we will have to have different approaches

Ian: let's summarise: it is very useful to see the work of the past years ; we'll see after the summer about the progress. We see here some links with some people ; you need some inputs from people you did not have.

Wrap up: Ian

  • no GDB next month (August)
  • In September pre-GDB on DOMA

-- IanCollier - 2018-07-30

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r5 - 2018-08-14 - IanCollier
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback