Subject: Re: [IPFIX] An advanced for Reducing Redundancy

From: "Boschi, Elisa" <[email-masked]>
Date: Wed, 6 Dec 2006 18:37:11 -0000

Paul,

			 >> One more that might be a bit obtuse.  The technique for 
			 >> defining separate cpIds for source and destination Common 
			 >> Properties is mentioned in the draft.  For common combinations 
			 >> a third category of cpId could be used which maps to two other 
			 >> cpIds. 
			 >> 
			 >> i.e.: 
			 >>  cpID C -> cpID A + cpID B 
			 >>  cpID A -> source address A, source port A 
			 >>  cpID B -> destination address B, source port B 
			 > 
			 > 
			 > Which raises the question of what duplicate cpID means, eg: 
			 > 
			 >   cpID D -> cpID C + cpID A 
			 > 
			 > 
			 > Also, we need to be careful to avoid circular definitions, eg: 
			 > 
			 >   cpID E -> cpID F [ + cpID ... ] 
			 >   cpID F -> cpID E [ + cpID ... ] 
			 > 
			 > 
			 > Probably an exporter SHOULD NOT send these, and a collector MUST 
			 > detect and discard them, and log. 
			
			>>> Agree, but I propose a SHOULD instead of a MUST: 
			
			>>> "An Exporter SHOULD NOT send Common Properties defined in >>> terms of other Common Properties (i.e. when one of the properties >>> is a CommonPropertiesID), and a Collector SHOULD detect and >>> discard them and log." 

	> Awww man, nooooo! 
	
	> I want to be able to build CP ID trees without having to create CommonCommonProperties or CommonCommonPropertiesProperties! 

so to avoid that you propose to define CommonProperties in terms of other Common Properties which in turn are defined in terms of CommonProperties which...
Is it really better?

If we allow trees, there are a few things more to worry about that should be addressed in the draft.

For example: 

duplication (as also written by Andrew)

cpID A --> src addr A, src port A
cpID B --> src addr A, cpID A

collision

cpID A --> src addr A, src port A
cpID A --> src addr B, cpID A

withdrawal

what happens to a node in the tree if one of its children (or a node in its subtree) is a cpID and is withdrawn? Does the collecting process need to search for all the CP that are defined in terms of it and invalidate them too?

		>> I think you misunderstand Paul's position.  I believe Paul thinks that 
		>> the collector should catch circular definitions.  The idea of defining 
		>> common Properties in terms of other common properties seems completely 
		>> valid to me. 

	> Exactly. 

**************************************************************************************************
 
 E-mail Confidentiality Notice and Disclaimer.
 
  This email and any files transmitted with it are confidential and are intended solely for 
the use of the individual or entity to which they are addressed. Access to this e-mail by 
anyone else is unauthorised. If you are not the intended recipient, any disclosure, 
copying, distribution or any action taken or omitted to be taken in reliance on it, is 
prohibited. 
  E-mail messages are not necessarily secure.  Hitachi does not accept responsibility 
for any changes made to this message after it was sent. 
  Please note that Hitachi checks outgoing e-mail messages for the presence of 
computer viruses.

**************************************************************************************************
_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Benoit Claise <[email-masked]>
Date: Thu, 07 Dec 2006 16:10:41 +0100

Elisa, Paul,
>>>> >>> "An Exporter SHOULD NOT send Common Properties defined in >>> 
>>>> terms of other Common Properties (i.e. when one of the properties 
>>>> >>> is a CommonPropertiesID), and a Collector SHOULD detect and >>> 
>>>> discard them and log."
>>
>>
>> > Awww man, nooooo!
>>
>> > I want to be able to build CP ID trees without having to create 
>> CommonCommonProperties or CommonCommonPropertiesProperties!
>>
> so to avoid that you propose to define CommonProperties in terms of 
> other Common Properties which in turn are defined in terms of 
> CommonProperties which...
> Is it really better?
>
Hence the new section I proposed earlier: Section 8.5 "Cascaded Common 
Properties"

Regards, Benoit.
> what happens to a node in the tree if one of its children (or a node 
> in its subtree) is a cpID and is withdrawn? Does the collecting 
> process need to search for all the CP that are defined in terms of it 
> and invalidate them too?
>
>
> ------------------------------------------------------------------------
>

---

From: Benoit Claise <[email-masked]>
Date: Thu, 07 Dec 2006 16:10:41 +0100

Elisa, Paul,

Regards, Benoit.

---

From: Lutz Mark <[email-masked]>
Date: Thu, 07 Dec 2006 16:18:18 +0100

Benoit, Elisa, Paul,

> Hence the new section I proposed earlier: Section 8.5 "Cascaded Common 

If we discuss and find a solution for the mentioned and probably
some more special cases in the draft, I wonder if this
will take more space than the description of the methods of
redundancy reduction.

Although I would prefer to allow the use of Common Property IDs
within the Common Properties Records, I guess that
dealing with all the special cases will blow up
the implementation disproportionately.

If we just say "the Exporter SHOULD NOT send trees of
Common Properties" and "the Collector SHOULD discard them",
then we can leave out the discussion from the draft and
ease the implementation of the protocol.
(However everybody may use CommonPropertiesIDs as
Common Properties, if there are special demands.)

Thoughts?

Regards,
Lutz

---

From: Lutz Mark <[email-masked]>
Date: Thu, 07 Dec 2006 16:18:18 +0100

Benoit, Elisa, Paul,

Thoughts?

Regards,
Lutz

---

From: Brian Trammell <[email-masked]>
Date: Thu, 7 Dec 2006 10:48:12 -0500

All,

Agreed that common properties cascades 1. should be allowed, and 2.  
addressed explicitly in the document.

To keep the discussion going, I'd like to propose solutions (or at  

>> cpID A --> src addr B, cpID A

I can think of two possible simple ways to address these concerns:

1. The principles in 3.1.2 Multiple Data Reduction could be readily  
extended to handle cascades: simply disallow any sort of collision,  
of which duplication is merely a subset.

2. Allow collision or duplication in the cascade, but ONLY to allow a  
common property "lower" in the hierarchy to override values of one  
"higher". This makes cascades potentially much more useful (giving us  
workable inheritance) at the expense of a fair amount of complexity.

I don't personally have a strong opinion in either direction.

>> withdrawal
>>
>> what happens to a node in the tree if one of its children (or a  
>> node in its subtree) is a cpID and is withdrawn? Does the  
>> collecting process need to search for all the CP that are defined  
>> in terms of it and invalidate them too?

Yes... it seems to me the only real difficulty with cascading common  
properties (indeed, with cascading _anything_ really) is handling  
withdrawal. We could of course allow any of the various policies for  
cascading withdrawal and even provide a mechanism for setting the  
common properties withdrawal cascade policy via options (and even  
allow those options to be bound to common properties themselves, if  
we were feeling particularly uncharitable toward CP implementors :) )  
but I don't think we want to do that for the sake of simplicity. So I  
think we want to pick a single withdrawal cascade policy.

There is a question to answer explicitly before doing so: When a  
common property is withdrawn or invalid, is the record referencing it  
1. invalid (as is implied by the document currently, and by most of  
the messages in these threads) or 2. valid but simply missing data  
(as has been proposed at least once, I forget by whom, but not  
particularly deeply discussed yet). Note that if inheritance and  
overrides are available in cascades as above, this second case is  
_not_ a valid choice.

In the first case, I would propose that 1. an Exporting Process  
SHOULD withdraw all common properties "lower" in the cascade before  
withdrawing the common properties they reference; 2. a Collecting  
Process MUST invalidate all common properties defined in terms of a  
withdrawn common property; and 3. a Collecting Process MUST ignore  
common properties withdrawals sent by an Exporting Process that would  
withdraw common properties previously invalidated by a cascade  
withdrawal as in 2. This proposal is not particularly simple, but I  
think it is as simple as it can be while holding to the principle of  
being liberal in what you accept and conservative in what you send,  
and to the IPFIX design requirement to minimize complexity at the EP.

In the second case, cascade withdrawal becomes simpler: simply ignore  
the cascade. The Exporting Process MUST explicitly withdraw any  
common property that is no longer valid, regardless of whether it's  
defined in terms of others. Partially withdrawn cascades would then  
merely be partially valid.

(For what it's worth, I personally lean toward the first case here.)

Thoughts?

Regards,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Thu, 7 Dec 2006 10:48:12 -0500

All,

Thoughts?

Regards,

Brian

---

From: Lutz Mark <[email-masked]>
Date: Thu, 07 Dec 2006 17:50:10 +0100

Brian,

> To keep the discussion going, I'd like to propose solutions (or at least 
> choices of solutions) for the points Elisa raised:

thanks for your assistance.

> extended to handle cascades: simply disallow any sort of collision, of 
> which duplication is merely a subset.
> 

I think it's not so easy since the protocol allows to have
multiple IEs of the same type within a template.

Regards,
Lutz

---

From: Lutz Mark <[email-masked]>
Date: Thu, 07 Dec 2006 17:50:10 +0100

Brian,

Regards,
Lutz

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 07 Dec 2006 19:08:40 +0000

Hi all

May I propose a relatively simple solution.  I think cascades of
identifiers is valid in IPFIX and useful.  I also think that they
are a lot simpler than people imagine.

NORMAL OPTIONS
==============
An option can be thought of as a substitution.  When a Data Record
is received containing Flow information then any identifiers in
that Record can be immediately substituted for the data associated
with it.

For example, a Data Record containing:
  commonPropertiesID = 5
  ipv4SourceAddress  = 192.168.0.1

If we have a definition of cpID 5, say ipv4DestinationAddress =
192.168.0.2) then the collector can immediately expand that into:
  ipv4DestinationAddress = 192.168.0.1
  ipv4SourceAddress      = 192.168.0.1

If cpId 5 is withdrawn or redefined at a later date then that
does NOT redefine this expanded Record.

If cpId 5 is not defined, the collector can hold off decoding
the Data Record for a short while before discarding it.

CASCADING OPTIONS
=================
If a Data Record is received which is definition then any
identifiers in that Record an be immediately substituted for
the data associated with it.  For example:
  commonProperties ID = 6   (scope)
  commonPropertiesID  = 5
  ipv4SourceAddress   = 192.168.0.1

Can be immediately expanded into:
  commonProperties ID    = 6   (scope)
  ipv4DestinationAddress = 192.168.0.1
  ipv4SourceAddress      = 192.168.0.1

If cpId 5 is withdrawn or redefined at a later date then that
does NOT redefine this expanded Record.

If cpId 5 is not defined, the collector can hold off decoding
the Data Record for a short while before discarding it.

A circular definition will sit on the waiting queue until it
is discarded.  A "collision" will immediately expand out to
contain the IE twice.  Today we can send Data Records with
the same IE twice so why not.  It should be noted that
ordering should be preserved.

Andrew

Brian Trammell wrote:
> cascading withdrawal and even provide a mechanism for setting the common 
> properties withdrawal cascade policy via options (and even allow those 
> options to be bound to common properties themselves, if we were feeling 
> particularly uncharitable toward CP implementors :) ) but I don't think 
> we want to do that for the sake of simplicity. So I think we want to 
> pick a single withdrawal cascade policy.
> 
> There is a question to answer explicitly before doing so: When a common 
> property is withdrawn or invalid, is the record referencing it 1. 
> invalid (as is implied by the document currently, and by most of the 
> messages in these threads) or 2. valid but simply missing data (as has 
> been proposed at least once, I forget by whom, but not particularly 
> deeply discussed yet). Note that if inheritance and overrides are 
> available in cascades as above, this second case is _not_ a valid choice.
> 
> In the first case, I would propose that 1. an Exporting Process SHOULD 
> withdraw all common properties "lower" in the cascade before withdrawing 
> the common properties they reference; 2. a Collecting Process MUST 
> invalidate all common properties defined in terms of a withdrawn common 
> property; and 3. a Collecting Process MUST ignore common properties 
> withdrawals sent by an Exporting Process that would withdraw common 
> properties previously invalidated by a cascade withdrawal as in 2. This 
> proposal is not particularly simple, but I think it is as simple as it 
> can be while holding to the principle of being liberal in what you 
> accept and conservative in what you send, and to the IPFIX design 
> requirement to minimize complexity at the EP.
> 
> the cascade. The Exporting Process MUST explicitly withdraw any common 
> property that is no longer valid, regardless of whether it's defined in 
> terms of others. Partially withdrawn cascades would then merely be 
> partially valid.
>

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 07 Dec 2006 19:08:40 +0000

Hi all

NORMAL OPTIONS
==============

CASCADING OPTIONS
=================

Andrew

---

From: Brian Trammell <[email-masked]>
Date: Thu, 7 Dec 2006 17:30:25 -0500

Lutz,

Ah, yes. I keep forgetting about this feature (largely, I think,  
because though we do support it now, we don't use it :) ). While I  
still think it would be useful to support inheritance with common  
properties cascades, it doesn't appear that the changes necessary to  
actually support it (i.e. you'd need some way to note one IE can  
override another) would be worth the effort. So, point 2. of proposal  
withdrawn.

Of course, as Andrew points out, this does not actually solve the  
problem with reconciling multiple IEs with common properties. See my  
reply to his message, forthcoming...

Regards,

Brian

On Dec 7, 2006, at 11:50 AM, Lutz Mark wrote:

>> 1. The principles in 3.1.2 Multiple Data Reduction could be  
>> readily extended to handle cascades: simply disallow any sort of  
>> collision, of which duplication is merely a subset.
>> 2. Allow collision or duplication in the cascade, but ONLY to  
>> allow a common property "lower" in the hierarchy to override  
>> values of one "higher". This makes cascades potentially much more  
>> useful (giving us workable inheritance) at the expense of a fair  
>> amount of complexity.
>

---

From: Brian Trammell <[email-masked]>
Date: Thu, 7 Dec 2006 17:30:25 -0500

Lutz,

Regards,

Brian

---

From: Paul Aitken <[email-masked]>
Date: Thu, 07 Dec 2006 23:32:59 +0000

Lutz,

> ease the implementation of the protocol.

We should be careful not to prohibit things just because we don't fully 
understand them - because then we make life difficult for the future, 
when someone has a better understanding but has to work around 
unnecessary limitations that we've imposed.

So if we decided not to proceed with this, then I would prefer not to 
include any text either explicitly for or against it. There are 
precedents for this in the existing drafts, if you read them carefully.

-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.

---

From: Paul Aitken <[email-masked]>
Date: Thu, 07 Dec 2006 23:32:59 +0000

Lutz,