Subject: Re: [IPFIX] Cascading Common Properties

From: Andrew Johnson <[email-masked]>
Date: Thu, 07 Dec 2006 19:35:58 +0000

Repeated email with correction to the example:
  ipv4DestinationAddress of 192.168.0.1 -> 192.168.0.2

Thanks to Danny Llewallyn for pointing it out.

Cheers

Andrew

-------- Original Message --------

Hi all

May I propose a relatively simple solution.  I think cascades of
identifiers is valid in IPFIX and useful.  I also think that they
are a lot simpler than people imagine.

NORMAL OPTIONS
==============
An option can be thought of as a substitution.  When a Data Record
is received containing Flow information then any identifiers in
that Record can be immediately substituted for the data associated
with it.

For example, a Data Record containing:
 commonPropertiesID = 5
 ipv4SourceAddress  = 192.168.0.1

If we have a definition of cpID 5, say ipv4DestinationAddress =
192.168.0.2) then the collector can immediately expand that into:
 ipv4DestinationAddress = 192.168.0.2
 ipv4SourceAddress      = 192.168.0.1

If cpId 5 is withdrawn or redefined at a later date then that
does NOT redefine this expanded Record.

If cpId 5 is not defined, the collector can hold off decoding
the Data Record for a short while before discarding it.

CASCADING OPTIONS
=================
If a Data Record is received which is definition then any
identifiers in that Record an be immediately substituted for
the data associated with it.  For example:
 commonProperties ID = 6   (scope)
 commonPropertiesID  = 5
 ipv4SourceAddress   = 192.168.0.1

Can be immediately expanded into:
 commonProperties ID    = 6   (scope)
 ipv4DestinationAddress = 192.168.0.2
 ipv4SourceAddress      = 192.168.0.1

If cpId 5 is withdrawn or redefined at a later date then that
does NOT redefine this expanded Record.

If cpId 5 is not defined, the collector can hold off decoding
the Data Record for a short while before discarding it.

A circular definition will sit on the waiting queue until it
is discarded.  A "collision" will immediately expand out to
contain the IE twice.  Today we can send Data Records with
the same IE twice so why not.  It should be noted that
ordering should be preserved.

Andrew

Brian Trammell wrote:
> All,
> 
> Agreed that common properties cascades 1. should be allowed, and 2. 
> addressed explicitly in the document.
> 
> To keep the discussion going, I'd like to propose solutions (or at least 
> choices of solutions) for the points Elisa raised:
> 
>>> duplication (as also written by Andrew)
>>>
>>> cpID A --> src addr A, src port A
>>> cpID B --> src addr A, cpID A
>>>
>>> collision
>>>
>>> cpID A --> src addr A, src port A
>>> cpID A --> src addr B, cpID A
> 
> I can think of two possible simple ways to address these concerns:
> 
> 1. The principles in 3.1.2 Multiple Data Reduction could be readily 
> extended to handle cascades: simply disallow any sort of collision, of 
> which duplication is merely a subset.
> 
> 2. Allow collision or duplication in the cascade, but ONLY to allow a 
> common property "lower" in the hierarchy to override values of one 
> "higher". This makes cascades potentially much more useful (giving us 
> workable inheritance) at the expense of a fair amount of complexity.
> 
> I don't personally have a strong opinion in either direction.
> 
>>> withdrawal
>>>
>>> what happens to a node in the tree if one of its children (or a node 
>>> in its subtree) is a cpID and is withdrawn? Does the collecting 
>>> process need to search for all the CP that are defined in terms of it 
>>> and invalidate them too?
> 
> Yes... it seems to me the only real difficulty with cascading common 
> properties (indeed, with cascading _anything_ really) is handling 
> withdrawal. We could of course allow any of the various policies for 
> cascading withdrawal and even provide a mechanism for setting the common 
> properties withdrawal cascade policy via options (and even allow those 
> options to be bound to common properties themselves, if we were feeling 
> particularly uncharitable toward CP implementors :) ) but I don't think 
> we want to do that for the sake of simplicity. So I think we want to 
> pick a single withdrawal cascade policy.
> 
> There is a question to answer explicitly before doing so: When a common 
> property is withdrawn or invalid, is the record referencing it 1. 
> invalid (as is implied by the document currently, and by most of the 
> messages in these threads) or 2. valid but simply missing data (as has 
> been proposed at least once, I forget by whom, but not particularly 
> deeply discussed yet). Note that if inheritance and overrides are 
> available in cascades as above, this second case is _not_ a valid choice.
> 
> In the first case, I would propose that 1. an Exporting Process SHOULD 
> withdraw all common properties "lower" in the cascade before withdrawing 
> the common properties they reference; 2. a Collecting Process MUST 
> invalidate all common properties defined in terms of a withdrawn common 
> property; and 3. a Collecting Process MUST ignore common properties 
> withdrawals sent by an Exporting Process that would withdraw common 
> properties previously invalidated by a cascade withdrawal as in 2. This 
> proposal is not particularly simple, but I think it is as simple as it 
> can be while holding to the principle of being liberal in what you 
> accept and conservative in what you send, and to the IPFIX design 
> requirement to minimize complexity at the EP.
> 
> In the second case, cascade withdrawal becomes simpler: simply ignore 
> the cascade. The Exporting Process MUST explicitly withdraw any common 
> property that is no longer valid, regardless of whether it's defined in 
> terms of others. Partially withdrawn cascades would then merely be 
> partially valid.
> 
> (For what it's worth, I personally lean toward the first case here.)
> 
> Thoughts?
> 
> Regards,
> 
> Brian
> 
> 
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ipfix

_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Brian Trammell <[email-masked]>
Date: Thu, 7 Dec 2006 17:44:23 -0500

Andrew,

Let me make sure I understand this by restating it a bit:

1. When the Collecting Process receives Common Properties that  
reference other Common Properties themselves, it MUST resolve the  
references to Common Properties. A Collecting Process MAY hold Common  
Properties for later resolution if the referenced Common Properties  
are not yet defined.

2. An Exporting Process MUST NOT export Common Properties that  
reference themselves directly, and a Collecting Process MUST drop any  
Common Properties that reference themselves directly and SHOULD log  
the error.

3. If a given Information Element appears both in a Record and by  
reference to Common Properties (whether in cascaded Common Properties  
or not), it MUST be treated as a multiple Information Element. The  
order of the multiple Information Elements MUST be determined from  
the arrangement of the commonPropertiesId and referenced Information  
Elements in the Data Record, and this order MUST be preserved.

Note that this third point would modify the present section 3.1.2  
multiple data reduction treatment of collisions.

(Also, I just realized something... if the three points above  
accurately describe what you're proposing, this could also be used  
to, somewhat inefficiently, build variable-length arrays in IPFIX...)

Hmmm.... I like it. It's simple, it's useful, it's implementable, and  
it requires fewer changes to the document at this point.

Unless I'm missing something. :)

Cheers,

Brian

On Dec 7, 2006, at 2:35 PM, Andrew Johnson wrote:

>> Agreed that common properties cascades 1. should be allowed, and  
>> 2. addressed explicitly in the document.
>> To keep the discussion going, I'd like to propose solutions (or at  
>> least choices of solutions) for the points Elisa raised:
>> 1. The principles in 3.1.2 Multiple Data Reduction could be  
>> readily extended to handle cascades: simply disallow any sort of  
>> collision, of which duplication is merely a subset.
>> 2. Allow collision or duplication in the cascade, but ONLY to  
>> allow a common property "lower" in the hierarchy to override  
>> values of one "higher". This makes cascades potentially much more  
>> useful (giving us workable inheritance) at the expense of a fair  
>> amount of complexity.
>>>> what happens to a node in the tree if one of its children (or a  
>>>> node in its subtree) is a cpID and is withdrawn? Does the  
>>>> collecting process need to search for all the CP that are  
>>>> defined in terms of it and invalidate them too?
>> Yes... it seems to me the only real difficulty with cascading  
>> common properties (indeed, with cascading _anything_ really) is  
>> handling withdrawal. We could of course allow any of the various  
>> policies for cascading withdrawal and even provide a mechanism for  
>> setting the common properties withdrawal cascade policy via  
>> options (and even allow those options to be bound to common  
>> properties themselves, if we were feeling particularly  
>> uncharitable toward CP implementors :) ) but I don't think we want  
>> to do that for the sake of simplicity. So I think we want to pick  
>> a single withdrawal cascade policy.
>> There is a question to answer explicitly before doing so: When a  
>> common property is withdrawn or invalid, is the record referencing  
>> it 1. invalid (as is implied by the document currently, and by  
>> most of the messages in these threads) or 2. valid but simply  
>> missing data (as has been proposed at least once, I forget by  
>> whom, but not particularly deeply discussed yet). Note that if  
>> inheritance and overrides are available in cascades as above, this  
>> second case is _not_ a valid choice.
>> In the first case, I would propose that 1. an Exporting Process  
>> SHOULD withdraw all common properties "lower" in the cascade  
>> before withdrawing the common properties they reference; 2. a  
>> Collecting Process MUST invalidate all common properties defined  
>> in terms of a withdrawn common property; and 3. a Collecting  
>> Process MUST ignore common properties withdrawals sent by an  
>> Exporting Process that would withdraw common properties previously  
>> invalidated by a cascade withdrawal as in 2. This proposal is not  
>> particularly simple, but I think it is as simple as it can be  
>> while holding to the principle of being liberal in what you accept  
>> and conservative in what you send, and to the IPFIX design  
>> In the second case, cascade withdrawal becomes simpler: simply  
>> ignore the cascade. The Exporting Process MUST explicitly withdraw  
>> any common property that is no longer valid, regardless of whether  
>> it's defined in terms of others. Partially withdrawn cascades  
>> would then merely be partially valid.

---

From: Brian Trammell <[email-masked]>
Date: Thu, 7 Dec 2006 17:44:23 -0500

Andrew,

Cheers,

Brian

---

From: Paul Aitken <[email-masked]>
Date: Thu, 07 Dec 2006 23:26:10 +0000

Brian,

> reference to Common Properties

Or indeed, not in the record at all but in multiple CP.

I'd just say "IE ordering MUST be preserved when expanding CP."

-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.

---

From: Paul Aitken <[email-masked]>
Date: Thu, 07 Dec 2006 23:26:10 +0000

Brian,

---

From: Andrew Johnson <[email-masked]>
Date: Fri, 08 Dec 2006 10:28:02 +0000

Hi Brian

As usual your understanding seems quite complete.  Some commentary
inline anyway.

BTW, I think Reducing Redundancy can, and should, be made to work with
the protocol as it stands today without any extra requirements on either
the Exporter or Collector.

Brian Trammell wrote:
> 1. When the Collecting Process receives Common Properties that reference 
> other Common Properties themselves, it MUST resolve the references to 
> Common Properties. A Collecting Process MAY hold Common Properties for 
> later resolution if the referenced Common Properties are not yet defined.

Absolutely.  Remember option data is received in a Data Record.  The same
rules should apply to Data Records that contain option data or flow data.

> 2. An Exporting Process MUST NOT export Common Properties that reference 
> themselves directly, and a Collecting Process MUST drop any Common 
> Properties that reference themselves directly and SHOULD log the error.

Well, I don't want to add any MUST, MUST NOT or SHOULDs beyond what the
protocol already says.  However if a Data Record is received that
references itself then this seems the fairest approach.  Perhaps it
should be added to the implementation guidelines as it is a general
issue for all options, not just CPs.

> reference to Common Properties (whether in cascaded Common Properties or 
> not), it MUST be treated as a multiple Information Element. The order of 
> the multiple Information Elements MUST be determined from the 
> arrangement of the commonPropertiesId and referenced Information 

Yes.

> multiple data reduction treatment of collisions.

Good :).  I would like all requirements that aren't taken from the
protocol to be removed, with the possible exceptions of ones, like you
mention above, that are really holes in the protocol document and should
probably also be covered in the implementation guidelines.

> (Also, I just realized something... if the three points above accurately 
> describe what you're proposing, this could also be used to, somewhat 
> inefficiently, build variable-length arrays in IPFIX...)
> 
> Hmmm.... I like it. It's simple, it's useful, it's implementable, and it 
> requires fewer changes to the document at this point.
> 

I don't think so.  Simple and implementable has always been my angle.
I also don't think that changing the document at this point should
really be much of a consideration.  It's blocked until PSAMP and it's
the first real test of options, so let's get it right.  The decisions
made here will have implications for all options, which we know aren't
dealt with properly/fully in the protocol.

regards

Andrew

>>> extended to handle cascades: simply disallow any sort of collision, 
>>> of which duplication is merely a subset.
>>>>> collecting process need to search for all the CP that are defined 
>>>>> in terms of it and invalidate them too?
>>> cascading withdrawal and even provide a mechanism for setting the 
>>> common properties withdrawal cascade policy via options (and even 
>>> allow those options to be bound to common properties themselves, if 
>>> we were feeling particularly uncharitable toward CP implementors :) ) 
>>> but I don't think we want to do that for the sake of simplicity. So I 
>>> think we want to pick a single withdrawal cascade policy.
>>> common property is withdrawn or invalid, is the record referencing it 
>>> 1. invalid (as is implied by the document currently, and by most of 
>>> the messages in these threads) or 2. valid but simply missing data 
>>> (as has been proposed at least once, I forget by whom, but not 
>>> particularly deeply discussed yet). Note that if inheritance and 
>>> overrides are available in cascades as above, this second case is 
>>> _not_ a valid choice.
>>> SHOULD withdraw all common properties "lower" in the cascade before 
>>> withdrawing the common properties they reference; 2. a Collecting 
>>> Process MUST invalidate all common properties defined in terms of a 
>>> withdrawn common property; and 3. a Collecting Process MUST ignore 
>>> common properties withdrawals sent by an Exporting Process that would 
>>> withdraw common properties previously invalidated by a cascade 
>>> withdrawal as in 2. This proposal is not particularly simple, but I 
>>> think it is as simple as it can be while holding to the principle of 
>>> being liberal in what you accept and conservative in what you send, 
>>> and to the IPFIX design requirement to minimize complexity at the EP.
>>> the cascade. The Exporting Process MUST explicitly withdraw any 
>>> common property that is no longer valid, regardless of whether it's 
>>> defined in terms of others. Partially withdrawn cascades would then 
>>> merely be partially valid.

---

From: Andrew Johnson <[email-masked]>
Date: Fri, 08 Dec 2006 10:28:02 +0000

Hi Brian

Yes.

regards

Andrew