Subject: [ipfix-protocol] semantics of multiple identical Information Elements in a template

From: Brian Trammell <[email-masked]>
Date: Wed, 8 Mar 2006 14:54:52 -0500

Benoit,

Regarding multiple identical information elements, you proposed the  
following a couple of months ago:

> In section 8 "Template Management", after the following paragraph...
>
> If a specific Information Element is required by a Template but is  
> not available in observed packets, the Exporting Process MAY choose  
> to export Flow Records without this Information Element in a Data  
> Record defined by a new Template.
>
> I would add:
>
> If an Information Element is required more than once in Template,  
> the different occurrences of this Information Element SHOULD follow  
> the logical order of their treatments by the Metering Process. For  
> example, if a selected packet goes through two hash functions, and  
> if the two hash values are sent within a single template, the first  
> occurrence of the hash value should belong to the first hash  
> function in the Metering Process. For example, when exporting the  
> two source IP addresses of an IPv4 in IPv4 packets, the first  
> sourceIPv4Address Information Element occurrence should be the IPv4  
> address of the outer header, while the second occurrence should be  
> the inner header one.
>
> In section 9 "The Collecting Process's Side", at the very end, I  
> would add:
>      The Collector MUST support the use of Templates containing  
> multiple occurrences of
>      the similar Information Elements.
>
> Note: this text should solve the section 3.1.2.1 "Multiple IE of  
> same field type" open issue in draft-boschi-ipfix-implementation- 
> guidelines-00.txt Regards, Benoit.

I'm probably a bit late to the party on this, but yes, this seems  
exactly the right thing to do for IPFIX version 0x000A. However, I  
think we should be more explicit about the semantics implied by this  
approach, and more restrictive on the applicability of multiple  
information elements in order to avoid any grey areas that might be  
used by implementors to overload multiple IE support in mutually  
unintelligible ways.

Specifically, in investigating whether multiple information elements  
would be applicable for biflows -- exporting a forward direction  
counter in the first counter IE and a reverse direction counter in  
the second counter IE -- I realized you could argue that, because the  
reverse counters are related to the "second" direction seen at the  
Metering Process, you could use two IEs (say, octetTotalCount), have  
the first be the forward counter, the second be the reverse counter,  
and dispense with having to define any new information elements.

However, what would the limits be on this approach? Say I wanted to  
export forward and reverse non-key header fields, say,  
tcpSequenceNumber. Would this conflict with a Collecting Process  
expecting to receive data about some sort of (admittedly ill-advised)  
TCP-in-TCP tunneling scheme?

At this point I realized that overloading multiple IE support for the  
biflow case is probably not the right thing to do, at least with  
multiple IE support expressed as above. However, the language in the  
protocol document should make it explicit that only certain semantics  
are currently permitted with multiple identical information elements.

(As an aside, note that as this example is written, it _does_ obviate  
the need for all but one of the mplsLabelStackEntry IEs, since the  
label stack is "unwrapped" by the Metering Process in a defined order.)

Regards,

Brian

---

From: Lutz Mark <[email-masked]>
Date: Thu, 09 Mar 2006 10:19:40 +0100

Hi Brian,

find my comments inline.

>> If a specific Information Element is required by a Template but is not 
>> available in observed packets, the Exporting Process MAY choose to 
>> export Flow Records without this Information Element in a Data Record 
>> defined by a new Template.
>>
>> I would add:
>>
>> If an Information Element is required more than once in Template, the 
>> different occurrences of this Information Element SHOULD follow the 
>> logical order of their treatments by the Metering Process. For 
>> example, if a selected packet goes through two hash functions, and if 
>> the two hash values are sent within a single template, the first 
>> occurrence of the hash value should belong to the first hash function 
>> in the Metering Process. For example, when exporting the two source IP 
>> addresses of an IPv4 in IPv4 packets, the first sourceIPv4Address 
>> Information Element occurrence should be the IPv4 address of the outer 
>> header, while the second occurrence should be the inner header one.
>>
>> In section 9 "The Collecting Process's Side", at the very end, I would 
>> add:
>> Note: this text should solve the section 3.1.2.1 "Multiple IE of same 
>> field type" open issue in 
>> draft-boschi-ipfix-implementation-guidelines-00.txt Regards, Benoit.
> 
> exactly the right thing to do for IPFIX version 0x000A. However, I think 
> we should be more explicit about the semantics implied by this approach, 
> and more restrictive on the applicability of multiple information 
> elements in order to avoid any grey areas that might be used by 
> implementors to overload multiple IE support in mutually unintelligible 
> ways.

I disagree about this. I prefer to keep Benoit's proposal.
The IPFIX protocol draft just defines the protocol and does not
care about the semantics. For instance one can use IPFIX protocol
with vendor specific information elements and with multiple IEs
of the same type. The protocol shouldn't restrict this.

> TCP-in-TCP tunneling scheme?

IPFIX just defines the semantics of IEs (information model draft).
There are no predefined templates. For this reason there is no
predefined semantic for an IPFIX template and corresponding flow
records.

Best regards,
Lutz

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Brian Trammell <[email-masked]>
Date: Thu, 9 Mar 2006 11:54:07 -0500

Lutz,

Comments, as usual, inline...

On Mar 9, 2006, at 4:19 AM, Lutz Mark wrote:

>> think we should be more explicit about the semantics implied by  
>> this approach, and more restrictive on the applicability of  
>> multiple information elements in order to avoid any grey areas  
>> that might be used by implementors to overload multiple IE support  
>> in mutually unintelligible ways.
>

Defining IPFIX to be a semantics-free representation makes absolute  
sense when each IE appears only once. Note that even this case is not  
_really_ semantics-free -- "this IE value applies to this Flow  
Record" is an assertion carrying some meaning, however thin. It's  
simple to handle private enterprise IEs in this framework - you may  
not have any idea what a given IE means, but you can unambiguously

However, once you allow multiple IE instances per Flow Record, as we  
have here, it does not appear possible not to make the following  
choices and still have a universally understandable data format:

1. Do multiple IEs in a record represent a set?

2. Are multiple IE's ordered?

These first two points are necessary for proper functioning of  
generic collectors, especially those which might re-export. Benoit's  
proposal as I read it covers both, and defines multiple IEs as  
ordered arrays.

There are other questions I'm not sure have yet been explicitly  
answered:

3. How does the interpretation of each subsequent IE change with its  
position in the collection? From Benoit's proposal I assume that  
subsequent IEs in a single record apply to subsequent treatment or  
sensing at a metering process. Is any other interpretation supported?

4. Does this intepretation of each subsequent IE change with the  
count of identical IEs in a given template? An example may clarify  
this question. Presume a biflow collector that uses the second IE of  
each type to serve as the "reverse" counter. It's a generic biflow  
collector - it knows how to handle all the various ways flows may be  
placed in time; it handles both delta and total counters; it can even  
handle non-key header fields, so, for example, it can recognize  
forward and reverse TCP initial sequence numbers via duplicate  
tcpSequenceNumber fields. What should this hypothetical biflow  
collector do when it receives a record with three octetTotalCount  
fields? And how is this approach compatible with the hash example in  
3. - if a biflow record has six payload hashes, do the first three
apply to the forward direction flow, or are the directions interleaved?

>> However, what would the limits be on this approach? Say I wanted  
>> to export forward and reverse non-key header fields, say,  
>> expecting to receive data about some sort of (admittedly ill- 
>> advised) TCP-in-TCP tunneling scheme?
>

Understood. My objection (or rather, confusion) does not apply to  
templates as a whole, but to the semantics of _each individual field_  
within a flow record.

Regards,

Brian

---

From: Paul Aitken <[email-masked]>
Date: Thu, 09 Mar 2006 18:45:10 +0000

Brian,

> if a biflow record has six payload hashes, do the first three apply
> to the forward direction flow, or are the directions interleaved?

If it's not 100% clear and unambiguous, then the biflow design is 
clearly incomplete.

For my 2€:

Presumably the device meansures the same information in both directions
- perhaps it doesn't even care which direction is which.

So divide and conquer: let the device export one template and two records.

The template contains a direction element and three hash elements. 
Define their order as you will (eg, time or location).

Then one record shows "I observed these three hashes in direction A", 
while the other says the same about the other three hashes in direction B.

-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.

---

From: Brian Trammell <[email-masked]>
Date: Thu, 9 Mar 2006 13:53:34 -0500

Paul,

> If it's not 100% clear and unambiguous, then the biflow design is  

My point exactly. As currently specified, multiple identical IEs do  
not appear sufficient for the biflow use case. (The single-record  
biflow export presently advanced in the biflow draft does not make  
use of multiple IEs, for precisely this reason. Direction is  
explicitly specified in the definition of the information elements.)

Regards,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Thu, 9 Mar 2006 14:36:31 -0500

Lutz,

Oops. Interrupted myself in mid thought on this one. To finish...

On Mar 9, 2006, at 11:54 AM, Brian Trammell wrote:

> sense when each IE appears only once. Note that even this case is  
> not _really_ semantics-free -- "this IE value applies to this Flow  
...state that the value applies to the record.

Regards,

Brian

---

From: Lutz Mark <[email-masked]>
Date: Sun, 12 Mar 2006 23:15:51 +0100

Brian,

> My point exactly...

So let's make a well-defined design.

There are definitely multiple methods to export biflow data
via IPFIX protocol. In most cases the method Paul described
would be sufficient. The usage of special Information Elements
for the reverse direction fields is another method. And since
IPFIX allows the use of multiple IEs of the same type within
one flow record, there might be a smart method to exploit
this for the export of biflow records. I can think of further
methods to export biflow data.
Let's collect the alternatives in the biflow draft and
discuss their pros and cons and which method fits best
for different use cases.

Maybe we should also pick this issue and give a
recommendation in the implementation guidelines.

Best regards,
Lutz