Subject: Re: HES store comments From: David Adams Date: Fri, 22 Mar 2002 14:42:21 -0500 To: David Adams CC: Luc GOOSSENS , Atlas - Database Group Continuing my response... 13. Our placement categories are essentially an implementation of the sharing categories and, to a lesser extend, the placement categories introduced in the ADM (architecture DB model). I don't feel we have the freedom to drop them unless the notions of sharing and placement categories are dropped from the ADM. That said, personally I have become used to the idea of sharing categories and would be reluctant to drop them. I think it may be very useful to have a level of sharing that is between that on the EDO and the event. A file writer can largely ignore placement categories by creating a single category that allows all types and keys. Readers can ignore the categories by concatenating the contents of all categories. It would probably be useful to add a method to the file interface that does the latter. Yes, I think you understand (but I should probably make clearer) that a placement category is not required to hold all the tye-keys that its type allows. 23. While it is essential to have catalogs, I agree that they certain can and should be kept distinct from the file handling pieces that make up HES proper. It is important to have a rudimentary view of the catalogs because that defines what data must be held by the "self-describing" files in HES and defines how HES files reference data that is stored in the catalogs (e.g. JobOptions). 24. I think the concept of stream type is important. Physics analyses will be done on collections of files (datasets) for which the data should all be in the same state. The stream type is a way if labeling this state. Event if we dropped placement categories, it would be useful to keep the notion of a stream type. 26. Once you have stream types, input streams have to assign these types to files. 31. I agree it would be okay for two EDO's to have the same type-key as long as they have the same EDO ID. general 1: I think we agree that the usual unit of analysis is not the single file but is a collection of files in a common state. I call this a dataset. This is discussed in the HES document but not the talk. Input and output streams will generally correspond to datasets rather than single files. general 2: Now you are moving back into the world of cataloging. The first versions of input streams will only handle single files or small collections of files. I do imagine later versions could support large numbers of files with catalogs to locate desired EDO's. Or the appropriate files might be retrieved before the job is run. In any case I believe replica management can be layered on top of the HES file interface that we propose. Thank you very much for your comments. Please let me know if you want me to expand on any of the above or any other points. da David Adams wrote: Luc: Here are some off-the-cuff responses indexed by slide number: 3. We left the class definitions out of the information that a file is required to carry. This was a deliberate but not widely discussed choice. HES supports multiple file formats and leaves it up to them whether or not to include this information. We tried to to stay away from the transient world. The job of HES is to locate persistent objects. The individual file formats take care of conversion to and from transient representation. 5. Of course we added an integer index because we wanted a compact way to identify files but you may be right that the saving in space is not worth maintaining a global mapping. My experience is that string identifiers for files can get very long as people try to pack more and more information into them. This is a point to reconsider. 7. Including the file ID in the EDO ID makes it easy to guarantee the latter is unique as well as providing a pointer to the location of the original version of the object. The EDO ID serves as the EDO reference. I'm not sure what you mean by your last point. We know an EDO is a replica if its ID differs from the ID of the file that holds it. Note that a replica may be bitwise identical to the original if the two file formats are the same. 10. Again I'm not sure what you mean. It is a requirement that we be able to point to an object inside an EDO. HES takes the responsibility for defining this mechanism so that is is possible to point to objects in a different file format. 11. At least you recognize it as such. I hope this will be clearer in the next rewrite. 12. Event is a suprisingly subtle concept. One could define all the data within a file associated with one event ID to be an event. But this is just a collection of placement categories and we didn't see the need to make it explicit because of the likelihood of misinterpretation. I will continue in a later message... da Luc GOOSSENS wrote: Hi all, In preparation of today's DB phone meeting, here are my comments on HES (slides from sw week, draft 3, March 7) slide 3: making the files self describing is I think a very good idea this includes descriptions of all the types (classes) used in it and hence one would expect the relation with ADL to be clarified here slide 5: each file carries a unique ID -> I suppose this is a number as opposed to the lfn which will be a string this will save some space in references to this file BUT it generates another management task and need for a mapping table -> I suggest to drop it slide 7: the EDO ID specifies the ID of the file that owns it -> combined with (on slide 9) the statement that EDO replica's keep the original's owning file -> this implements a kind of master copy scheme -> why is that necessary? I think that conceptually the EDO ID can *not* include a reference to a file containing it. EDO references on the other hand could include the id of a file where a copy of the EDO can be found as a sort of hint. btw how does the system assure that replica's are indeed replica's? slide 10: I think it is possible that the persistent layer can do without sub-EDO references (i.e. to the persistency svc such a reference is just an object that happens to be the combination of an EDO reference and e.g. an integer) Is that desirable? slide 11: the local EDO index is probably a good optimization but perhaps it should be more presented as such slide 12: note that this slide explicitly says "there are no event objects" -> this means also no collections of references to event objects -> the reason is of course because this object would change all the time so it is much easier if it is not materialised slide 13: I propose to simply drop everything concerned with placement categories and use sets of type-key pairs directly btw the constraint that every event in a file has to have the same collection of PCs is pretty strong and weakly justified unless this needs to be interpreted as the *maximal* set of key-type pairs which is perhaps hinted at in slide 15 -> "allowed" type keys slide 23: file content catalog -> this largely overlaps with the meta data catalog from Grenoble. I propose to defer the job and production thread stuff to another document. I hope these two things can be kept separate. slide 24: like PCs I propose to drop the stream stuff a stream is just yet another name for a set of PCs I am sure we need a way to express groups of key-type pairs but this looks like a 3 level structure and as you know numbers other than 0,1 and n need proper justification. from slide 26 on things get a little bit too complex to my taste but I made a similar remark to our own architecture document slide 31: no type-key may be duplicated -> I guess you mean "may be duplicated but then of course needs to have the same value" btw at several occasions I have told the storegate people that the flat type-key space is much more liable to unintentional double use than the Gaudi hierarchical scheme but obviously they know better general: The fact that output does not fit into a single file is accidental this means that a logical set of events may be spread over different files both intentionally in the EDO dimension and accidentally in the event ID dimension I see no supra file mechanism to address this. This is of course related to the absence of events. There was the concept of dataset in the HES document but that was not completely clear to me either. I wonder if we can not think of a more effective scheme for replica management on the EDO level. This is in fact a non-trivial problem. Ideally, you would want to be able to fetch the optimal set of files that will resolve all the EDO references you will need with respect to some cost function. Please comment, Luc -- David Adams desk: 631-344-6049 Brookhaven National Lab fax: 631-344-5078 PAS group, Building 510A email: dladams@bnl.gov Upton, NY 11973-5000 http://www.usatlas.bnl.gov/~dladams