Subject: Re: [IPFIX] new work item candidate #4: extended types

From: "Boschi, Elisa" <[email-masked]>
Date: Thu, 26 Jul 2007 22:51:21 +0100

Gerhard,

> 
> On the one hand, it's a nice idea to have some IEs which allow
> describing the type and semantic of other IEs. On the other hand, my
> impression is that the usefulness is very restricted and not as broad as
> claimed in the introduction of the draft:
> 
>    ... Having to do some sort of tool-specific
>    configuration (or code modification) on every tool just to tell which
>    type each used Information Element has is a huge amount of work for
>    users and implementors of enterprise-specific Information Elements.
>    Many tools in fact only need to know the field's data type to work on
>    them. ...
>

One of the use cases we have in mind here would be a collaborative measurement infrastructure where each exporter is implemented by a different vendor and exports slightly different enterprise IEs. In that case, having type information in the message stream allows ad-hoc preliminary analysis of the enterprise specific data. 

Our claim that many tools only need to know the field's data types to work on them applies...so the usefulness of the draft is not too "restricted" 
 
> In my opinion, any deeper analysis and processing of metering data
> requires knowledge about the semantic. Just knowing the data type may be
> enough to display values correctly. Considering
> informationElementSemanticType, you might also know if the calculation
> of statistical properties (max, min, mean, variance etc.) makes sense or
> not. But any further analysis would probably require more semantical
> knowledge. That's why I assume that, in practice, analyzing and
> processing enterprise-specific data will mostly be performed with
> specialized analysis tools that have been implemented for this specific
> purpose. And for these tools, there is no need to export the IE information.
> 

...but those ad hoc tools won't be reusable, unless tool specific configuration or code modification are done for any used Enterprise Specific IE. And what about if the tools are not updated? 

A real life example that this draft would solve is mentioned in:
http://www1.ietf.org/mail-archive/web/ipfix/current/msg03040.html

...which also contains the motivation for this draft.

Regards,
Elisa

> 
> Juergen Quittek wrote:
>> Dear all,
>>
>> Candidates #4 to become a new IPFIX work item is described by
>> draft draft-boschi-ipfix-extended-type-00.txt.
>>
>>> 4. Extended Types (Elisa Boschi)
>>>    Needed for IPFIX files, most useful for Enterprise IEs; need to
>>>    specify what operations make sense for an IE.
>>>    Support shown for this as a WG item, as for File Format.
>>>    Milestones:  WG LC after IETF 70, to IESG before IETF  71.
>> It is a method for assigning data types to unknown IPFIX information
>> elements by using option records that map data types to element identifiers.
>>
>> It can be applied in cases where an IPFIX collector does not know the
>> data type of received information elements. This may in particular happen
>> if enterprise specific elements are used. Without this information the
>> collector would treat these elements as if they were of type octet array.
>> With this information it may be able to treat them more appropriately.
>>
>> The current version of the IPFIX file format draft recommends using this
>> method when IPFIX records are stored.
>>
>> Do you think this is a useful extension of the IPFIX protocol?
>> Do you think we should accept this as an IPFIX work item?
>> Please have a look at the draft and send your comments.
>>
>> Thanks,
>>
>>     Juergen
>>
>>
>>
>> _______________________________________________
>> IPFIX mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/ipfix
> 

************************************************************************************************** 
E-mail Confidentiality Notice and Disclaimer. 

This email and any files transmitted with it are confidential and are intended solely for the use 
of the individual or entity to which they are addressed. Access to this e-mail by anyone else is 
unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any
action taken or omitted to be taken in reliance on it, is prohibited. E-mail messages are not 
necessarily secure. Hitachi does not accept responsibility for any changes made to this message 
after it was sent. 

Please note that Hitachi checks outgoing e-mail messages for the presence of computer viruses. 
**************************************************************************************************
_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Juergen Quittek <[email-masked]>
Date: Fri, 27 Jul 2007 00:13:27 +0200

Hi Elisa,

Am 26.07.2007 23:51 Uhr schrieb "Boschi, Elisa" unter
<[email-masked]>:

> One of the use cases we have in mind here would be a collaborative measurement
> infrastructure where each exporter is implemented by a different vendor and
> exports slightly different enterprise IEs. In that case, having type
> information in the message stream allows ad-hoc preliminary analysis of the
> enterprise specific data.
> 
> Our claim that many tools only need to know the field's data types to work on
> them applies...so the usefulness of the draft is not too "restricted"

I have problems with fully accepting this claim.
You just know the data type.

When is is helpful to know the data type without having a clue about the
semantics?

Why would in such cases handling of opaque octet strings not be acceptle?

Thanks,

    Juergen

> ...but those ad hoc tools won't be reusable, unless tool specific
> configuration or code modification are done for any used Enterprise Specific
> IE. And what about if the tools are not updated?
> 
> This email and any files transmitted with it are confidential and are intended
> solely for the use of the individual or entity to which they are addressed.
> Access to this e-mail by anyone else is unauthorised. If you are not the
> intended recipient,any disclosure, copying, distribution or any action taken
> or omitted to be taken in reliance on it, is prohibited.
> E-mail messages are not necessarily secure.Hitachi does not accept
> responsibility for any changes made to this message after it was sent.
>

---

From: Juergen Quittek <[email-masked]>
Date: Fri, 27 Jul 2007 00:13:27 +0200

Hi Elisa,

Thanks,

    Juergen

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 27 Jul 2007 14:18:21 +0200

Elisa,

Boschi, Elisa wrote:
> One of the use cases we have in mind here would be a collaborative
> measurement infrastructure where each exporter is implemented by a
> different vendor and exports slightly different enterprise IEs. In that
> case, having type information in the message stream allows ad-hoc
> preliminary analysis of the enterprise specific data.

If "preliminary analysis" means displaying the values correctly or
calculating some simple statistics, I fully agree.

If you are aware of any more sophisticated analysis method one could
apply without any further semantical knowledge, please let me know. This
would make my desicion on this topic easier :)

> Our claim that many tools only need to know the field's data types to
> work on them applies...so the usefulness of the draft is not too
> "restricted"

There may be cases where displaying values or calculating mean,
variance, min, max etc. is just enough to "analyze" the metering data.
Yet, traffic analysis methods are usually much more sophisticated and
specific to the semantics of the metering data.

> configuration or code modification are done for any used Enterprise
> Specific IE. And what about if the tools are not updated?
> 

The mentioned example is only about displaying the values correctly. I
would not call this "analysis".

I'm certainly not against your draft or the idea to describe
enterprise-specific IEs. I just think that the usability is restricted,
and that it might not be of high relevance in practice.

Regards,
Gerhard

> This email and any files transmitted with it are confidential and are
> intended solely for the use of the individual or entity to which they
> are addressed. Access to this e-mail by anyone else is unauthorised. If
> you are not the intended recipient,any disclosure, copying, distribution
> or any action taken or omitted to be taken in reliance on it, is
> prohibited.

-- 
Dipl.-Ing. Gerhard Münz
Computer Networks and Internet
Wilhelm Schickard Institute for Computer Science
University of Tuebingen
Sand 13 (Room B309), D-72076 Tuebingen, Germany
Phone:  +49 7071 29-70534       Fax: +49 7071 29-5220
E-mail: [email-masked]
WWW:    http://net.informatik.uni-tuebingen.de/~muenz
_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 27 Jul 2007 14:18:21 +0200

Elisa,

Regards,
Gerhard

---

From: Brian Trammell <[email-masked]>
Date: Fri, 27 Jul 2007 08:48:12 -0500

All,

There seems to be some confusion on the list as to the intent of and  
method described by the extended type information draft. Speaking  
here as a coauthor of the draft and the editor of the file draft from  
which the method was taken, I'd like to see if I can clarify this.

In discussing the draft this week, we have realized what we aim to  
define here is a general method for representing the information  
element properties available in the IPFIX Information Model within an  
IPFIX Message stream using IPFIX Options. Information Elements can  
thus be described inline, framed and encoded the same way as the  
templates and data they're used to define.

The primary use case for this mechanism is the addition of type  
(signed, unsigned, octet array, address, etc.) and semantic (counter,  
flags, etc.) information to Enterprise Specific Information Elements  
to improve the interoperability of this feature. We first defined the  
mechanism in the file draft as it struck us as most useful to keep  
this information around in the archival case, so some number of years  
later when you pull data from tape received from an exporter that you  
no longer have access to you can at least do some exploratory data  
analysis. After discussions at the third IPFIX interop in Berlin and  
the subsequent IETF meeting in Prague, we decided that the mechanism  
might also be useful on the wire, so we separated the drafts.

The draft may need to provide for the encoding of a few more  
properties (e.g., units, name, perhaps a terse description string, as  
have been suggested in reviews of the draft and on the list) but  
these are at this point details, new IEs to be added to section 3,  
without changing the fundamental approach described.

To address two suggestions related to this draft I'm not sure are in  
scope:

First, information element mapping to define a direct equivalence  
between two IEs: yes, this may be useful, and indeed probably quite  
simple to implement using an approach like the one we describe, but  
as there is no concept of equivalence in the IPFIX Information Model,  
I don't think we should address it here.

Second, a complete "capability based" semantic system for enterprise  
specific IEs (or indeed, any IE), which would essentially enumerate  
the operations available on any given Information Element: this is  
summable, this is averageable, this is unionable, and so on. Two  
things to say here. First, a lot of the operations in this list are  
aggregation operations, so it would probably make more sense to  
define this mechanism when we address aggregation as an IE (whether  
in an aggregation draft or separately at the same time). Second, and  
again, this draft is simply a mechanism for representing the  
information model and its properties inline. As IPFIX-INFO does not  
handle operation enumerations, this draft should not either.

Juergen: I have one additional comment inline, below.

On Jul 26, 2007, at 5:13 PM, Juergen Quittek wrote:

>> measurement

You _do_ know the semantics, though, in many cases. These are the  
cases we're largely concerned with. The information model defines  
five semantics: quantity, totalCounter, deltaCounter, identifier, and  
flags; these are representable using this method. A collector that  
knows something is a total counter can do quite a lot with it - time- 
series binning, aggregation, correlation with other counters,  
graphical representations of these and other operations, etc. Even  
with just the type a collector knows how to decode and display a value.

Regards,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Fri, 27 Jul 2007 08:48:12 -0500

All,

Regards,

Brian