TWiki> LCG Web>WLCGTEGDataManagement (revision 28)EditAttachPDF

TEG Data Management


We have been given the following topics by the WLCG MB (some aspects expanded on by Brian Bockelman; later additions from SM annotate what overlap we see between Storage and DM)
  • Review of the Data Management demonstrators from summer 2010. (DM/SM OVERLAP. Most topics in Amsterdam actually covered the DM (higher) layer. But many of them have implications in the SM, also, and some are actually infrastructure-related.)
  • Dataset management: Currently, the common tools operate at the "file level" (file transfer, file catalog), oblivious to the fact that each experiment has built a custom dataset mechanism on top of them. What commonalities could be extracted? Is it possible/wise/necessary for the WLCG to play some role at the dataset level? (DM)
  • Strategies for data federations in the WLCG. How do on-demand / caching architectures (c.f. ARC or Xrootd) fit into the larger WLCG data management ecosystem? (DM/SM OVERLAP. Enormous implications on SM, but DM could probably take a lead here, and in a latter part we could step in, e.g. how would you manage the storage implications. We would encourage the DM TEG to discuss and clarify things earlier on this wrt other topics.)
  • Wide area transfer protocols. GridFTP has been the "workhorse", but it has shown significant limitations: the striping mechanism is a nightmare for disks, and it inherits design issues from FTP that cause it to not work well with NATs. Recently, HTTP and Xrootd have been suggested as replacements. (DM/SM OVERLAP. But, again, probably, DM can take the lead here.)
  • Future strategies for FTS. FTS is again a workhorse for most of the experiments. How do we recommend it evolve in the future? Note: we can probably ask the FTS developers to come and present at one of our meetings. (DM/SM OVERLAP. FTS devs we agree should be invited ideally to a joint meeting.)
  • What are the evolving requirements for data accessibility and security? I believe ATLAS/CMS/LHCb depend on the 75 sites to each individually enforce the correct experiment-internal access policies to their data, while ALICE's model delegates the internal access policies back to the experiment. How pleased/displeased is each experiment, and is there an opportunity for "cross pollination"? (DM/SM OVERLAP. But we should understand what the Security TEG does on this.)
  • POOL: To my knowledge, ATLAS is the remaining user of POOL. Is it possible to relabel it an experiment-specific piece of software? (DM)
  • ROOT, Proof? How do these lower-level frameworks intersect with the WLCG, if anywhere? (DM/SM OVERLAP. We can discuss in the joint Dec meeting.)
  • Namespace management. Each experiment does namespace management very differently; this is often a tripping point in cross-experiment discussions (as an example, CMS does not use GUIDs and LFN<->PFN mappings can be done in constant time without a database-based catalog). Can we outline at the "philosophical" level what each experiment uses? (DM/SM OVERLAP)
  • Future directions of the LFC. How is it deployed, what features are used? What are experimental needs in the future? (DM)

From the set of Storage Management drivers from the MB, we found one with possible overlap:

  • Storage system interfaces to Grid (SRM future?). Interoperation (DM/SM OVERLAP. We need a list of what we need as from SRM functions. Joint discussion. Maybe also encourage all exps to have a team working on cloud tech.)

Please put any additional suggested topics below, and leave a signature so we know who added it.

Questions for Experiments

Initial list extracted from Dirk's presentation:

  • We assert that we have a working system, do not need a new system, but the current system (cost and complexity) exceeds what is needed. Would the experiment agree with this statement?
  • What elements are being used, and how? Which have been added and why?
  • Which components failed to deliver (larger) functionality needed? Does that mean we can deprecate and obsolete these components, or was it not truly needed?
  • If the middleware dropped complexity and removed functionality, does the experiment have the resources to adopt to the change?
  • What is the experiment's interest in revising data placement models? What kinds of revisions have gone through?
    • What is the experiment's interest in data federations?
  • What sort of assumptions does the experiment need to make for data federations work?
  • Could you work directly with clustered file systems at smaller sites?
  • Could you work directly with cloud file systems (i.e., GET/PUT)? Assume "cloud file systems" implies REST-like APIs, transfer via HTTP, no random access within files, no third-party transfer, and possibly no file/directory structure. See Amazon S3 for inspiration.
  • How thoroughly does the experiment use space management today?
  • Does your framework support HTTP-based access?
  • Is the archive / disk split an agreed-upon strategy in your experiment? Can HSM be dropped?
  • What role do you see for Data Federation versus Data Push? Is this useful at smaller sites?
  • For smaller sites, would caching work for your experiment?

Additions, based on conversations with Storage TEG:

  • Security / VOMS - what are experiments expectations / needs.
  • where they see their namespace management "philosophy" evolving in the future.
  • What volume of data do experiments plan to move / store (and what do they currently move /store).
  • What kind of file access performance are they expecting (any changes expected?) - what WAN data transfer rates?

Additions, based on conversations with the MB: Data management

  • Can we get rid of SRM in front of our storage services? (question also for Storage group)
  • Why do we need gridftp? Why can't we use http like the rest of the world. (NB: "because it can't do xx" is not a good answer - we are good at developing something from scratch and bad at adding functionality to open source sw).
  • Can we agree layered data management and storage architecture? This will help factorise various issues and help piecemeal replacement of layers. To be defined together with storage group
  • Is HSM model needed? (etc see Dirk's GDB slides): NO?
  • How will we manage a global namespace - whose problem is it? the Experiments? Is there a continued need for LFC?
  • Access control - what is really needed? The simpler the better!
  • As a general comment, we would like a better clarification/division of responsibilities between experiments and infrastructure(s). I.e. If the experiments want to manage most of the complexity then we should simplify the services that the infrastructure should provide.
    • Posed as a question: Where should the complexity and intelligence lie - the experiments or the infrastructure? How do you view the balance now, and how would you like to see this change (if at all)?

Answers from Experiments:

Draft Membership

  • Gerard Bernabeu
  • Giacinto Donvito
  • Pepe Flix
  • Sam Skipsey
  • Mattias Wadenstein
  • Brian Bockelman (chair)
  • Gergely Sipos
  • Jos van Wezel
  • Luca DellAgnello
  • Philippe Charpentier
  • Dirk Duellmann, (chair)
  • Vincent Garonne
  • Ricardo Graciani Diaz
  • Alexei Klimentov
  • Elisa Lanciotti
  • Maarten Litmaath
  • Markus Schulz
  • Graeme Stewart
  • Andrea Valassi
  • Maria Girone
  • Paul Rossman
  • Oliver Keeble
  • Simon Metson
  • Tim Bell
  • David Malon
  • Bernd Panzer
  • Michel Jouvin
  • Xiaomei Zhang
  • Jerome Pansanel
  • Riccardo Zappi
  • Doug Benjamin
  • Emmanuel Medernach
  • Daniele Bonacorsi
  • Alvaro Fernandez Casani
(TODO - map each name to stakeholder or organization)

Face-to-Face Meeting

More information here.

Logistics and Meetings

  • Indico category here
  • Email list archive here
  • Kickoff meeting details (not in indico due to permission issues):
    • Friday, Nov 4 at 9:00-100 FNAL, 15:00-16:00 CERN.
    • EVO URL; phone bridge ID is 425 7376.
    • Agenda will be mostly organizational. Items:
      • Short welcome, overview, and charge from chairs (15')
      • Discussion of organization of group (Specifically: need to determine how many/frequency of meetings, meeting agends, how many face-to-face meetings, dates, and locations) (15')
      • Assignment of homework: for WLCG experiments, what are the top needs / concerns / evolutionary plans? For CI providers, what are the plans over the next few years? What are you planning on evolving?
  • Doodle poll for the F2F (joint with Storage TEG):

Meeting Minutes

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatpptx DM-TEG-Status-8Nov-GDB.pptx r1 manage 55.2 K 2011-11-16 - 08:15 DirkDuellmann status report to GDB 8.11.11
Unknown file formatpptx DM_Kickoff.pptx r1 manage 52.7 K 2011-11-04 - 14:55 BrianBockelman  
Texttxt POOL_LHCb_statement.txt r1 manage 0.5 K 2011-11-05 - 08:51 AndreaValassi LHCb statement about POOL
Unknown file formatdocx TEG__Mandate.docx r1 manage 109.3 K 2011-11-04 - 14:55 BrianBockelman  
Unknown file formatdocx Technical_Evolution_Group__TWGs-v3.docx r1 manage 94.9 K 2011-10-18 - 13:49 DirkDuellmann  
Edit | Attach | Watch | Print version | History: r31 < r30 < r29 < r28 < r27 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r28 - 2012-01-10 - BrianBockelman
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback