Subject: [IPFIX] IPFIX Structured Data - comments related to semantic

From: "Gowri Dhandapani (gowri)" <[email-masked]>
Date: Mon, 11 Oct 2010 09:14:29 -0700

Hi all,

Please find the mail below summarizing issues related to the
Semanticfield and our responses, also, included answers for Gerhard's
and Brian's comments regarding semantic. 

During the last IETF meeting, there was a concern regarding semantic of
subTemplateMultiList as described in the draft. To provide some context,
a subTemplateMultiList contains Data Records corresponding to multiple
Template Records. The semantic in the subTemplateMultiList provides the
top-level relationship among the series of Data Records corresponding to
different Template Records. For example, if a subTemplateMultiList has a
semantic 'exactlyOneOf' and the elements are (A1, A2), (B1, B2, B3),
(C1) - this means the Data Record contains exactly one of the lists (A1,
A2) or (B1, B2, B3) or (C1). 

The concern was that the semantic should mean all Data Records
regardless of the Template. Benoit, Paul, Gerhard and Brian had a follow
up discussion and reached a closure on this, the consensus at the end of
that meeting is:

*        Keep the semantic of subTemplateMultiList as defined, however
explain the semantic at the beginning of the draft. So, we added this in
Section 2.4: "The three Structured Data Information Elements carry some
semantic information so that the Collecting Process can understand the
relationship between the different list elements. The semantic in the
Structured Data Information Elements is provided in order to express the
relationship among the multiple top-level list elements. As an example,
if a list is composed of elements (A,B,C), the semantic expresses the
relationship among A, B and C, regardless of whether A, B and C are
individual elements or list of elements."  To clarify, in a
subTemplateMultiList the elements could represent lists in turn A->(A1,
A2), B->(B1, B2, B3) etc. So, if a subTemplateMultiList has a semantic
'exactlyOneOf' and the elements are (A1, A2), (B1, B2, B3), (C1) - this
means the Data Record contains exactly one of the lists (A1, A2) or (B1,
B2, B3) or (C1).   The keyword "top-level" indicates that the semantic
represents the relationship among elements (A, B, C) and not at the next
level if these elements expand to lists inturn. 

*        If we want the semantic of all Data Records in a
subTemplateMultiList, then according to Brian's suggestion we have added
the following text in Section 5.4: "Alternatively, if a semantic is
required to describe the relationship among all Data Records within a
subTemplateMultiList (regardless of the Template Record), an encoding
based on a subTemplateMultiList with one Data Record corresponding to a
single Template ID can be used." 

Also, please find responses to specific comments regarding semantic
below:

There is one suggestion about the specification itself. I think that it
would be easier to understand if the meaning of the semantic of
subTemplateList and subTemplateMultiList would be the same, namely that
it specifies the semantic among the Data Records. At the moment, the
draft specifies (although not very clearly) that the semantic refers to
goups of Data Records associated to different Templates in the case of
the subTemplateMultiList. I do not see a use-case for this. Also, no
useful example is provided. On the contrary, it is always said that the
semantic of subTemplateMultiList is not sufficient.     
 
Please refer to explanation above. 

	     4.5.3. subTemplateMultiList 
	 
	      Whereas each top-level element in a subTemplateList
Information 

what is meant by "top-level"? It means that there are levels below.
Remove?

Yes, in a subTemplateMultiList there are two levels - the list contains
(A, B, C) where A could in turn be a list (A1, A2, A3). The semantic
provides the relationship among A, B, C and not at the next level. So,
the keyword "top-level" is required to highlight the distinction.  

		      Element corresponds with a single Template ID and
therefore has 

	"corresponds to"

		      the same data type, sometimes it is useful for a
list to contain 

	Why "same data type"? Each element is a Data Record associated
to the
	same (Options) Template.
	Reference to "data type" removed. The section 4.5.3 now reads:
"Whereas each element in a subTemplateList Information Element
corresponds to a single Template, it is sometimes useful for a list to
contain elements corresponding to different Templates. "

	       0                   1                   2
3 
	       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8
9 0 1 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |    Semantic   |         Template ID X         |Data
Records...| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      | ... Length X  |        Data Record X.1 Content ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |                              ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |      ...      |        Data Record X.2 Content ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |                              ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |      ...      |        Data Record X.L Content ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |                              ...
|  
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |      ...      |         Template ID Y         |Data
Records...|  
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      | ... Length Y  |        Data Record  Y.1 Content ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |                              ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |      ...      |        Data Record Y.2 Content ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |                              ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |      ...      |        Data Record Y.M Content ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |                              ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |      ...      |         Template ID Z         |Data
Records...| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      | ... Length Z  |        Data Record Z.1 Content ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |                              ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |      ...      |        Data Record Z.2 Content ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |                              ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |      ...      |        Data Record Z.N Content ...
| 
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |                              ...
|  
	
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
	      |      ...      |    
	      +-+-+-+-+-+-+-+-+   
	       
	                   Figure H: subTemplateMultiList Encoding 
	                                        
	      Semantic 
	       
	          The Semantic indicates the top level semantic among
Template 

Remove "The" or "Semantic field"?

done.

remove "top level"
As explained above, "top-level" is required to mark the distinction. 
 
"among the Data Records"?
Yes, should be Data Records corresponding to different Templates -
corrected. 
 
Comment added later: Now as I have read section 5.4, I understand that
you want to restrict to the semantic among sequences of Data Records
belonging to different Templates. This does not become clear. Wouldn't
it be simpler to say that Semantic defines the semantic among all Data
Records, regardless of the Template?
Refer to explanation at the beginning of the mail. 

		          Records, i.e. the relationship among the
series of Data 
		          Records from the different Template Records
within this 

	"from" => "associated with"

Changed. 

	          Note: if a semantic is required to describe the
relationship 
	          among the different Data Records corresponding to a
single 
	          Template ID within the subTemplateMultiList, then an
encoding 
	          based on a basicList of subTemplateLists should be
used. Refer 

I would remove "basicList" because it does not support Templates. So, it
is not a general alternative.
"A basicList can contain individual IEs or any of the three Structured
Data IEs as list elements (note that the subTemplateList or
subTemplateMultiList contains multiple Data Records corresponding to one
or more Templates respectively).  
As an example, consider a basicList of subTemplateLists:
     basicList (semantic) { 
                               (subTemplateList1 w/ semantic-1), 
                               (subTemplateList2 w/ semantic-2)}. 
The top-level semantic would provide the relationship among the various
subTemplateLists while each subtemplateList can carry its own semantic
to describe the relationship among its list elements. Hope this is
clear.

	     5.4. Usage Guidelines for Equivalent Data Representations 
	 
	      Because basicList, subTemplateList, and
subTemplateMultiList are 
	      all lists, in several cases there is more than one way to 
	      represent what is effectively the same data structure.
However, 
	      in some cases, one approach has an advantage over the
other e.g. 
	      more compact, uses fewer resources, etc., and is therefore

	      preferred over an alternate representation. 
	 
	      A subTemplateList can represent the same simple list of
single-
	      value Information Elements as a basicList, if the Template

	      referenced by the subTemplateList contains only one
single-valued 
	      Information Element.  Although the encoding is more
compact than a 
	      basicList by two bytes, using a subTemplateList in this
case 
	      requires a new Template per Information Element.  The
basicList 
	      requires no additional Template and is therefore
RECOMMENDED in 
	      this case.  
	 
	      Although a subTemplateMultiList with one Element can
represent the 
	      contents of a subTemplateList, the subTemplateMultiList
carries 
	      two additional bytes (Element Length).  It is also
potentially 
	      useful to a Collecting Process to know in advance that a 
	      subTemplateList directly indicates that list element types
are 
	      consistent.  The subTemplateList Information Element is
therefore 
	      RECOMMENDED in this case. 
	 
	      The Semantic in a subTemplateMultiList indicates the top
level 

Semantic field

Done. 

	      semantic among Template Records, i.e. the relationship
among the 

Template Records => Data Records ?
 
See comment above in section 4.5.3.
Yes, corrected to Data Records. 

	      series of Data Records from the different Template
Records, within 
	      this Structured Data.  If a semantic is required to
describe the 
	      relationship among the different Data Records
corresponding to a 
	      single Template ID within the subTemplateMultiList, then
an 
	      encoding based on a basicList of subTemplateLists should
be used. 
	 
	      Note that the referenced Information Element(s) in the
Structured 
	      Data Information Elements can be taken from the IPFIX
information 
	      model [RFC5102], the PSAMP information model [RFC5477], or
any of 
	      the Information Elements defined in the IANA IPFIX
registry [IANA-
	      IPFIX]. 

	Brian's comments regarding semantic: 

	Agreed with Gerhard on this point, and this would be the most
important comment in my own review. Semantics, as I understand them,
bind to the set of _values_ within a list structure. The value in a
subTemplateMultiList are the records (each of which, indeed, may be
associated with a different Template). Binding semantics to templates
breaks this clean design. Section 5.6 almost seems like a paean to why
SubTemplateMultiList sucks, which if you're applying semantics at the
template level, not the record level, it seems to.

	I see the primary use of subTemplateMultiList as semantically
identical to subTemplateList, allowing the usage of different templates
representing semantically similar but situationally different data
(e.g., mixing v4 and v6 flows) within the same List. Here, Templates are
(as in IPFIX itself) an encoding mechanism, not a semantic one.

	Given this confusion, I've not reviewed the rest of the
subTemplateMultiList-relevant content. I would strongly suggest a
statement up front of the universality of semantics - applying to list
values in all cases - and a statement up front that subTemplateMultiList
is a convenience for allowing different encoding of semantically similar
information.

	Please refer to meeting consensus at the beginning of the mail. 
	 
	Thanks.