Subject: Re: [IPFIX] WG last call on draft-ietf-ipfix-reducing-redundancy-01.txt until December 1st

From: Benoit Claise <[email-masked]>
Date: Tue, 05 Dec 2006 13:50:26 +0100

Hi Brian,

Thanks for your comments.
See inline. Note that, for clarity purposes, I only kept a few comments
>
>
>
> Second, there appears to be some spurious mention of templates in 
> section 7, presumably as the cpId management text evolved from 
> template management text:
Trying to keep the parity with [IPFIX-PROTO], I simply adapted the text 
... a little bit too quickly :(
>
>
> 7 para 7 OLD:
>
> If a Collecting Process receives a Common Properties Withdrawal 
> message, the Collecting Process MUST delete the corresponding Common 
> Properties associated with the specific SCTP association and specific 
> Observation Domain, and stop decoding IPFIX Messages that use the 
> withdrawn Templates.
>
> 7 para 7 NEW:
>
> If a Collecting Process receives a Common Properties Withdrawal 
> message, the Collecting Process MUST delete the corresponding Common 
> Properties associated with the specific SCTP association and specific 
> Observation Domain.
I would like to add, "and stop decoding IPFIX Messages that use the 
withdrawn Common Properties"
This would be in line with [IPFIX-PROTO] "The Collecting Process's Side" 
section:

   If a Collecting Process receives a Template Withdraw Message, the 
   Collecting Process MUST delete the corresponding Template Records 
   associated with the specific SCTP association and specific 
   Observation Domain, and stop decoding IPFIX Messages that use the 
   withdrawn Templates. 

>
> 7.1 para 3 OLD:
>
>    The Collecting Process MUST NOT decode any further Data Records which
>    are associated with the expired Common Properties.  If a Common
>    Property is refreshed with a Template Record that differs from the
>    previous received Template Record, the Collecting Process SHOULD log
>    a warning and replace the previous received Common Property with the
>    new one.  The Template lifetime at the Collecting Process MUST be at
>    least 3 times higher than the Template refresh timeout configured on
>    the Exporting Process.
>
> 7.1 para 3 NEW:
>
>    The Collecting Process MUST NOT decode any further Data Records which
>    are associated with the expired Common Properties.  If a Common
>    Property is refreshed with a definition that differs from the
>    previous definition, the Collecting Process SHOULD log
>    a warning and replace the previous received Common Property with the
>    new one.  The Common Property lifetime at the Collecting Process 
> MUST be at
>    least 3 times higher than the Common Property timeout configured on
>    the Exporting Process.
In this case, you insert a new term "Common Property timeout". I propose:
   The Collecting Process MUST NOT decode any further Data Records which
   are associated with the expired Common Properties.  If a Common
   Property is refreshed with a definition that differs from the
   previous definition, the Collecting Process SHOULD log
   a warning and replace the previous received Common Property with the
   new one.  The Common Property lifetime at the Collecting Process MUST 
be at
   least 3 times higher than the refresh timeout of the Template used to 
export the
   Common Property definition, configured on the Exporting Process.

Regards, Benoit.
>
>
>
>
> Regards,
>
> Brian
>
> On Nov 13, 2006, at 2:17 PM, Juergen Quittek wrote:
>
>> Dear all,
>>
>> As agreed at the IPFIX session in San Diego last week
>> this is the WG last call on the Internet draft on
>> Reducing redundancy in IPFIX and PSAMP reports,
>> draft-ietf-ipfix-reducing-redundancy-01.txt.
>>
>> The WG last call starts today and will be closed on
>> December 1st.
>>
>> Please review the draft and send your comments!  Please also
>> send a message to the mailing list if you reviewed it and
>> have no comment to make!
>>
>> Thanks,
>>
>>    Juergen
>>
>>
>> _______________________________________________
>> IPFIX mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/ipfix

_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Benoit Claise <[email-masked]>
Date: Tue, 05 Dec 2006 13:50:26 +0100

Hi Brian,

Regards, Benoit.

---

From: Brian Trammell <[email-masked]>
Date: Tue, 5 Dec 2006 11:31:16 -0500

Benoit and all,

See inline, as always...

On Dec 5, 2006, at 7:50 AM, Benoit Claise wrote:

>> message, the Collecting Process MUST delete the corresponding  
>> Common Properties associated with the specific SCTP association  
>> and specific Observation Domain, and stop decoding IPFIX Messages  
>> that use the withdrawn Templates.
>>
>> 7 para 7 NEW:
>>
>> message, the Collecting Process MUST delete the corresponding  
>> Common Properties associated with the specific SCTP association  
>> and specific Observation Domain.
> This would be in line with [IPFIX-PROTO] "The Collecting Process's  
> Side" section:
>
>   If a Collecting Process receives a Template Withdraw Message,  
> the   Collecting Process MUST delete the corresponding Template  
> Records   associated with the specific SCTP association and  
> specific   Observation Domain, and stop decoding IPFIX Messages  
> that use the   withdrawn Templates.

Hmmm.... I agree with the point, but I'm not sure this is the right  
language here. A CP would stop decoding Messages (actually, Sets)  
described by withdrawn Templates because Templates are necessary for  
decode. Common Properties are a bit higher in the stack - the Sets  
can be decoded, but some data would be missing for each record. How  
about something like:

"and stop interpreting Data Records referring to those Common  
Properties. The receipt of Data Records referring to Common  
Properties that have been withdrawn MUST be ignored and SHOULD be  
logged by the Collecting Process."

>>    new one.  The Common Property lifetime at the Collecting  
>> Process MUST be at
>   least 3 times higher than the refresh timeout of the Template  
> used to export the

(see previous message where this was clarified to "Common Property  
refresh timeout")

An alternative would be to define the Common Property refresh timeout  
elsewhere in the document, and refer to it here, as we do with  
Template refresh in the protocol. Since the Common Property lifetime  
and the Template lifetime are both scoped to the Transport Session  
lifetime for connection-oriented transports, there may be no reason  
to presume that the Template refresh and Common Property refresh  
timeout should ever be different (indeed, things may break if they  
are; I've not thought it all the way through); however, I do think it  
is clearer if the Common Property refresh mechanism has its own  
terminology rather than tying it directly to the template timeout.

Thoughts?

Regards,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Tue, 5 Dec 2006 11:31:16 -0500

Benoit and all,

See inline, as always...

Thoughts?

Regards,

Brian

---

From: Benoit Claise <[email-masked]>
Date: Tue, 05 Dec 2006 23:29:08 +0100

Hi Andrew,
>
>
> If there are references to PSAMP drafts, doesn't that
> block this from becoming an RFC until the PSAMP ones
> are RFCs? 
Sure.
> Do we have to mention PSAMP so much anyway,
> as it's just a specialised use case of IPFIX?
Right, I think that the previous version was somehow lacking some 
information.
Yes we can use this mechanism for all sort of flows, even for:
"So the PSAMP export can be seen as special IPFIX Flow Record containing 
information about a single packet." dixit PSAMP-PROTO
However, by having the clear distinctions...

  3.1    Per Flow Data Reduction...............................11  
  3.1.1 Unique Data Reduction..................................11  
  3.1.2 Multiple Data Reduction................................12  
  3.2    Per Packet Data Reduction.............................13
... I believe that we gained in clarity... Maybe not for you, as you 
know enough about IPFIX and PSAMP to understand how to apply this 
techniques, but for the rest of the community.
>
> Since a flow is defined as a group of packets that share
> a set of common properties, i.e. key fields, is there
> any danger that the commonPropertiesID will be wrongly
> assumed to refer to only the complete set of key fields?
By definition, I think this is impossible. The flow key definition says 
"Each of the fields which are used to define a Flow are termed Flow 
Keys.". So there is nothing in common between flows. However, I 
understand your concern for novice readers.

OLD section 1 "Introduction"

     Regardless of the fields content, flow records with common  
     properties export the same fields in every single flow record.   
     These common properties may represent values common to a collection  
     of flows or packets, or values that are invariant over time.  The  
     reduction of redundant data from the export stream can result in a  
     significant reduction of the transferred data.

NEW section 1 "Introduction"

  Regardless of the fields content, flow records with common  
  properties export the same fields in every single flow record.   
  These common properties may represent values common to a collection  
  of flows or packets, or values that are invariant over time.  Note  
  that the common properties don't represent the list of flow keys, which  
  are used to define a flow definition: however, the common properties  
  may contain some of the flow keys. The reduction of redundant data from  
  the export stream can result in a significant reduction of the   
transferred data.

>
> Is there any need to define a new term for "Specific
> Properties"? These are just part of a normal Data Record.
> For example:
> "... The rows are then referenced by the Specific Properties
> by using the appropriate value for the Common Properties
> identifier."
>
> could simply be:
> "... The rows are then referenced by Data Records by
> using the appropriate value for the Common Properties
> identifier."
In your example, the "Data Records" doesn't mean the full data records 
before applying the redundancy reduction mechanism but only a part of it.
So it might be confusing... That's the trick we found.

However, I could agree that the draft would work without a formal (upper 
case) definition.... So do you have the same idea for the Common Properties?
>
> Are option withdrawal messages part of IPFIX?
Yes. IPFIX says: "The Set ID field MUST contain the value 2 for Template 
Set withdrawal and the value 3 for Options Template Set withdrawal."
Unless you want to say an option data record withdrawal message?

Note: I discovered that the draft still contains: "Option Data Record"
This term has been removed from IPFIX-PROTO. So it should be removed in 
this draft.
> Are they meaningful in IPFIX? i.e. per the PROTO will an option
> with scope but no data be deemed invalidated by the Collector.
> Scope without data is the same as never having recieved the
> mapping in the first place. I think that it would be up to the
> implementation but it seems like a good idea so if we can't add
> it to the PROTO can we put it in the implementation guidelines?
>
> How do you propose to withdraw common properties when they are
> associated with a template rather than the Common Properties index?
I don't ;) This is the reason why the use the Common Properties index is 
a cleaner design.
>
>
> General Comments
> ================
> I think, sections 3.1.2, 8.2 and 8.3 would be better grouped
> into a 3.2 Advanced Techniques section.
>
> I think 3.2 Per Packet Data Reduction, should be removed and
> any information that is not in section 3.1 should be added to
> to the PSAMP example.
> I think the the per-packet example is a good example of
> per flow reducing redundancy and section 11.1.1 can be removed.
See one of my previous remark about:

  3.1    Per Flow Data Reduction...............................11  
  3.1.1 Unique Data Reduction..................................11  
  3.1.2 Multiple Data Reduction................................12  
  3.2    Per Packet Data Reduction.............................13

Again, it's because you know the boundary between IPFIX and PSAMP
>
> Labels for the figures would be better if they were centred.
Agreed.
>
> There is all sorts of extra spacing at the end of the lines.
>
>
>
> Specific Comments
> =================
> 2. Terminology
> Tabbing is inconsistent in definition of "Selector":
> (i) the Packet Content; (ii) information derived from the packet's 
> treatment at the Observation Point; (iii) any selection state that may 
> be maintained by the Selection Process.
>
> 3.1.2 Multiple Data Reduction
>
> I don't think this section is explained very well. From my original
> email to Elisa and Lutz (18th Nov 2005):
> Andrew Johnson wrote:
>> Flows
>> =====
>> Flow A: flowId 1, flowId 11, 30 packets, 6000 bytes
>> Flow B: flowID 1, flowId 12, 50 packets, 9500 bytes
>> Flow C: flowId 2, flowId 11, 60 packets, 8000 bytes
>> Flow D: flowID 2, flowId 12, 10 packets, 1500 bytes
>>
>> Options
>> =======
>> flowId 1 - destinationIpv4address 10.0.0.1, destinationTcpPort 80
>> flowId 1 - destinationIpv4address 10.0.0.5, destinationTcpPort 80
>>
>> flowId 11 - sourceIpv4address 10.0.1.10, sourceTcpPort 1932
>> flowId 12 - sourceIpv4address 10.0.1.15, sourceTcpPort 1842
>
>
> This shows how 4 (2+2) options can be expanded to 4 combinations of
> Data Record to reduce redundancy without the need to define 4 complete
> sets of Common Properties and therefore not save any bandwidth. The
> more options you have the better this works. 20 options (10+10) would
> give you up to 100 possible pairs common properties.
>
> This isn't really made clear in the draft text, with only "flow1"
> having two indices.
I agree the examples in figure 4/5 don't show this.
Maybe a new section in section 8 "Export and Evaluation Considerations"?
> The example in 11.1.2 doesn't even have any!
Agreed. This should be corrected.
>
> While I agree that using a template that doesn't include the cpID
> is a good idea to efficiently communicate flows with not so common
> properties, however I would call this an "Advanced Technique" to be
> listed in the section I proposed earlier.
>
>
> 3.2 Per Packet Data Reduction
>
> The text reads:
> If filtering is applied to select a series of packets, using the PSAMP 
> specifications to export per-packet information might be relatively 
> inefficient if the filtered fields (the common attributes) are 
> exported in every single record.
>
> However in section 6.5.2.5 (Property Match Filtering) of PSAMP-PROTO
> it is explained how the filter information MUST be sent. This is
> designed so that the selector ID will match the one from the Data
> Record (via the selectionSequenceId) and perform exactly the function
> that is being proposed here for the commonPropertiesId.
Good point. I would remove this paragraph and move to this place the one 
starting with:

  An example of the per packet data reduction is the measurement of  
  One-Way Delay (OWD), where the exact same specific packet must be  
  observed at the source and destination of the path to be measured.  
  By subtracting ...

>
>
> 4. Specifications for bandwidth saving information export
>
> What is being specified here that isn't true for ALL options as
> described in IPFIX-PROTO?
What do you suggest?
>
> I think section 3 does a better job of explaining how all this
> works. I'm really not sure what section 4 adds.
The idea was to put the specs in section 4 while the section 3 was the 
high level solution.
> (Aside: figure 9 refers to Common Sroperties).
Yep. thanks.
>
>
> 6. commonPropertiesID Management
>
> The text reads:
> The commonPropertiesID values MAY be assigned sequentially, but it�s
> It seems the "it's" is using an invalid character rather than an
> apostrophe.
>
>
> 7. The Collecting Process Side
>
> The text reads:
> Like TemplateIDs the commonPropertiesIDs are generated dynamically by 
> the Exporting Process. Hence a restart of the Exporting Process may 
> imply a renumbering of commonProperiesIDs.
> The collector can't know if the Exporter Process restarted, this is
> why we rescoped the commonPropertiesID to the Transport Session, so
> the above text is now redundant.
I know it's redundant, but don't you think it helps. For example, we 
don't explain anywhere (if I recall correctly) that "the 
commonPropertiesIDs are generated dynamically by the Exporting Process"
>
>
> The text reads:
> Common Properties IDs are unique per SCTP association and per 
> Observation Domain. If the Collecting Process receives a common 
> Properties ID which has already been received but which has not 
> previously been withdrawn (i.e. a commonPropertiesID from the same 
> Exporter Observation Domain with the same Template ID received on the 
> SCTP association), then the Collecting Process MUST shutdown the 
> association.
This is similar to [IPFIX-PROTO]

  Template IDs are unique per SCTP association and per Observation  
  Domain.  If the Collecting Process receives a Template which has  
  already been received but which has not previously been withdrawn  
   (i.e. a Template Record from the same Exporter Observation Domain  
    with the same Template ID received on the SCTP association), then 
    the Collecting Process MUST shutdown the association.

>
> and:
> If the Collecting Process receives a Common Properties Withdrawal 
> message for a Common Properties that it has not received before on 
> this SCTP assocation, it MUST reset the SCTP association, discard the 
> IPFIX Message, and SHOULD log the error as it does for malformed IPFIX 
> Messages.
> I think this might go against IPFIX-PROTO.
This is similar to [IPFIX-PROTO]

  If the Collecting Process receives a Template Withdraw message for a  
  Template Record it has not received before on this SCTP assocation,  
   it MUST reset the SCTP association, discard the IPFIX Message, and  
   SHOULD log the error as it does for malformed IPFIX Messages.
>
>
> The text reads:
> If the Common Properties are not refreshed before that lifetime has 
> expired, the Collecting Process MUST discard the corresponding 
> definition of the commonPropertiesID and any current and future 
> associated Data Records. In which case, an alarm MUST be logged.
> What happened to storing it until the option turned up. Furthermore
> you shouldn't dictate the the record is discarded, just because you
> can't tell a piece of potentially trivial information about a flow
> doesn't mean it's meaningless. The Common Properties in question
> could be anything.
This is similar to [IPFIX-PROTO]

  If the Template   (or other definition) is not refreshed before that 
lifetime has  
  expired, the Collecting Process MUST discard that definition and any  
  current and future associated Data Records.  In which case, an alarm  
  MUST be logged. 
>
> Besides, we're not changing the protocol here, right?
Right. So this is not an IPFIX redundancy reduction issue but an IPFIX 
protocol issue... as we don't want to change the protocol with this draft.
>
>
> 8.1 Transport Protocol Choice The text reads:
> The proposed method is most effective using a reliable transport 
> protocol for the transfer of the Common Properties. Therefore the use 
> of SCTP or TCP is recommended.
>
> Note that SCTP isn't necessarily reliable.
SCTP -> PR-SCTP with the reliable mode
>
>
> 8.3 commonPropertiesID vs. TemplateID scope
> Additional flow properties can be tied to almost any I.E. For example
> a particular interface could be associated with the one IP address it
> serves.
That's true but I still think the design is cleaner with the Common 
Properties Id.
See the remark about the Common Properties ID withdrawal

>
>
> 8.4 Efficiency Gain
> The text reads:
> "...it can�t predict whether future..."
>
> Weird apostrophe character again.
>
>
>
> Examples:
>
> 1. There is no padding between records, it's against IPFIX-PROTO,
> unless you use the special padding I.E.
> 2. The templates define the counts as 4 octets but the records show
> them as 2.
> 3. Since 11.1.1 and 11.2 don't demonstrate anything different, can
> we have just one example?
> 4. The tabbing is sometimes a little of on the run though of the I.E.s,
> and I think we can loose the commas. e.g.
>
> OLD
> - the source IPv4 Address, sourceIPv4Address [IPFIX-INFO], with a type 
> of 8 and a length of 4 octets
> NEW
> - the source IPv4 Address: sourceIPv4Address [IPFIX-INFO] with a type 
> of 8 and a length of 4 octets
>
>
> 5. What is a packet ID anyway? In general PSAMP uses sampling to gather
> detailed information about a small proportion of the network traffic.
> In most cases there won't be many common factors between the reports.
> The real exception is hash-based filtering, which is what I think the
> example uses.
>
>
> 6. Do we have to use enterprise specific things here? What about:
> observationTimeMilliseconds (323),
> digestHashValue (326),
> ipTotalLength (224)
>
Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Tue, 05 Dec 2006 23:29:08 +0100

Hi Andrew,
Sure.

Agreed.

What do you suggest?
Yep. thanks.

Regards, Benoit.

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 06 Dec 2006 00:41:01 +0000

Hi all,

some comments inline:

Brian Trammell wrote:
> decode. Common Properties are a bit higher in the stack - the Sets can 
> be decoded, but some data would be missing for each record. How about 
> something like:
> 
> Properties. The receipt of Data Records referring to Common Properties 
> that have been withdrawn MUST be ignored and SHOULD be logged by the 
> Collecting Process."

I think even this is in danger of putting demands on a collector that
go beyond what the IPFIX specifies.  I think Common Properties should
be expected to work with a collector that is compliant with what IPFIX
PROTO says today, and this is something new.  I agree that a flow with
a CPid that doesn't map to anything is a bit meaningless and a collector
COULD be configured to throw away meaningless flows.  However, in
general I would expect a collector to keep flows even without the
associated data that can be gleaned from options.

>>   least 3 times higher than the refresh timeout of the Template used 
>> to export the
> elsewhere in the document, and refer to it here, as we do with Template 
> refresh in the protocol. Since the Common Property lifetime and the 
> Template lifetime are both scoped to the Transport Session lifetime for 
> connection-oriented transports, there may be no reason to presume that 
> the Template refresh and Common Property refresh timeout should ever be 
> different (indeed, things may break if they are; I've not thought it all 
> the way through); however, I do think it is clearer if the Common 
> Property refresh mechanism has its own terminology rather than tying it 
> directly to the template timeout.
> 
> Thoughts?

Again I am concerned that Reducing Redundancy is making demands of the
protocol beyond what is expected today.  I don't think options are
particularly well dealt with and I agree that treating them in a similar
way to templates is probably the best approach, but a collector that is
written without Reducing Redundancy in mind should be expected to work
with these additional requirements.  That means that we can't add some
new lifetime for CP, or even a more general one that covers all options.

That said, it would be fair to add additional requirements on the
Exporter to implement the scheme, and resending the options data
regularly be a good thing.  Since the the timeout values for the
template and common properties will be based on the expected
reliability of the UDP stream, then it makes sense to me that they
would be the same.

Cheers

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 06 Dec 2006 00:41:01 +0000

Hi all,

some comments inline:

Cheers

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 06 Dec 2006 01:45:24 +0000

>> block this from becoming an RFC until the PSAMP ones

OK.

> techniques, but for the rest of the community.

It seems that 3.2 doesn't add anything to 3.1.1 other than providing
another example.  Perhaps remove the per-packet stuff from the main
part of the draft but keep it as a detailed example at the end.

>  Regardless of the fields content, flow records with common   properties 
> export the same fields in every single flow record.    These common 
> properties may represent values common to a collection   of flows or 
> packets, or values that are invariant over time.  Note   that the common 
> properties don't represent the list of flow keys, which   are used to 
> define a flow definition: however, the common properties   may contain 
> some of the flow keys. The reduction of redundant data from   the export 
> stream can result in a significant reduction of the   transferred data.

My concern wasn't really directed at the draft itself, but just the
name "common properties".  From IPFIX-PROTO:
  "A Flow is defined as a set of IP packets passing an Observation 
   Point in the network during a certain time interval.  All packets 
   belonging to a particular Flow have a set of common properties."

And then from IPFIX-INFO:
   Description of commonPropertiesId:
      An identifier of a set of common properties that is unique per
      Observation Domain.  Typically, this Information Element is used
      to link to information reported in separate records.

The terminology is almost identical.  The commonPropertiesId refers
to "a set of common properties" that is quite different from ones
that determine a Flow.

> case) definition.... So do you have the same idea for the Common 

Considering there is a commonPropertiesID, I am quite happy with you
referring to Common Properties.

There seems to be a bit of a conceptual difference here though.  You
seem to imagine that an Exporting Process has a set of IPFIX Data
Records which it then applies some processing to to get some new
special IPFIX Data Records and some Common Property options.  The
original set of Data Records may never actually exist, depending on
how the scheme is implemented.  For me, the only set of IPFIX Data
Records to consider are the ones which are actually sent.

> Unless you want to say an option data record withdrawal message?

There is a distinct difference between withdrawing the template and
withdrawing the data record associated with an option template.  I
think that the concept works though, although perhaps not by design.

> I don't ;) This is the reason why the use the Common Properties index is 

I think withdrawal messages should work in general for options
data, so why not send an empty record with the template as scope?

> Again, it's because you know the boundary between IPFIX and PSAMP

I just don't think the repetition is necessary.  Per-packet is
just a good example of the technique, the Per Packet section
doesn't seem to add any new information.

>  observed at the source and destination of the path to be measured.   By 
> subtracting ...
> 
>>
>>

Well it seems that section three really explains almost the same
thing 3 times.  The section 4 explains most of it again (with a
bit more info) another 3 times.

I would prefer section 3 to simply explain the technique.  Walk
through how the information from a Flow is split into two parts,
and then add the bit from section 4 how these two parts are
exported.

Using more than one CPid in a record should be part of the
advanced techniques once the basics are made clear.  Per packet
should just be a detailed example at the end.

> The idea was to put the specs in section 4 while the section 3 was the 

Rereading it, I see that now, but there seems to be a fair amount
of repetition.

> commonPropertiesIDs are generated dynamically by the Exporting Process"

The commonPropertiesIDs are unique within the transport session.  If the
session is restarted then you've lost that scope and therefore can't
keep using the same commonPropertiesID mappings.

I don't see why the commonPropertiesIDs have to be dynamically generated
by the Exporting Process.  The could be shared between several EPs or
pre-calculated based in historical data from that Observation Domain.

> with the same Template ID received on the SCTP association), then    the 
> Collecting Process MUST shutdown the association.
> 
>>
>> and:

These are similar to what IPFIX-PROTO says about templates, but that
doesn't mean the text can just be used to talk about Data Records sent
for Options Templates.

Option data withdrawal messages are not mentioned in IPFIX-PROTO,
although I think they can work.  However, in general options data with
the same scope as an earlier record will just overwrite the older
record.

> lifetime has   expired, the Collecting Process MUST discard that 
> definition and any   current and future associated Data Records.  In 
> which case, an alarm   MUST be logged.

This "(or other definition)" stuff is only mentioned for the Collecting
Process and it really isn't clear if it should apply to Common
Properties.  However, storing the data records in a queue awaiting
the decode information is really only something that is expected to
be done at the start of the session.  Is it mentioned in this draft?

> protocol issue... as we don't want to change the protocol with this draft.

What's a protocol issue?  I think applying all the requirements of
templates to all options is changing the protocol.

> See the remark about the Common Properties ID withdrawal

You can withdraw any option record, so I don't think it's a big deal.

>> observationTimeMilliseconds (323),

Regards

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 06 Dec 2006 01:45:24 +0000

OK.

Regards

Andrew

---

From: Benoit Claise <[email-masked]>
Date: Thu, 07 Dec 2006 13:33:59 +0100

Andrew,
>> "So the PSAMP export can be seen as special IPFIX Flow Record 
>> containing information about a single packet." dixit PSAMP-PROTO
>> By definition, I think this is impossible. The flow key definition 
>> says "Each of the fields which are used to define a Flow are termed 
>> Flow Keys.". So there is nothing in common between flows. However, I 
>> These common properties may represent values common to a 
>> collection      of flows or packets, or values that are invariant 
>> over time.  The      reduction of redundant data from the export 
>> that the common properties don't represent the list of flow keys, 
>> which   are used to define a flow definition: however, the common 
>> properties   may contain some of the flow keys. The reduction of 
>> redundant data from   the export stream can result in a significant 
>> reduction of the   transferred data.
>
What about something such as?
Description of commonPropertiesId:
     An identifier of a set of common properties between different Flows.
     The that is unique per Observation Domain. 
     Typically, this Information Element is used
     to link to information reported in separate records.
>> In your example, the "Data Records" doesn't mean the full data 
>> records before applying the redundancy reduction mechanism but only a 
>> part of it.
>> However, I could agree that the draft would work without a formal 
>> (upper case) definition.... So do you have the same idea for the 
>> Common Properties?
>
Ok. So let's see how the draft looks like if we remove the "Specific 
Properties" formal definition but if we keep the "Common Properties" one.
> how the scheme is implemented.  
Sure, this is an implementation detail, but conceptually, we want to 
reduce "something". How do you call this "something"?

   This document describes a bandwidth saving method for exporting flow 
   or packet information using the IP Flow Information Export (IPFIX) 
   protocol.

Anyway, I think that we're diverging from your initial issue which was about "Specific
Properties"...

> For me, the only set of IPFIX Data

>> Yes. IPFIX says: "The Set ID field MUST contain the value 2 for 
>> Template Set withdrawal and the value 3 for Options Template Set 
>> withdrawal."
Agreed.
>> This term has been removed from IPFIX-PROTO. So it should be removed 
>> in this draft.
>>
>> I don't ;) This is the reason why the use the Common Properties index 
>> is a cleaner design.
>
I guess that you wanted to say:
    I think withdrawal messages should work in general for options
    data, so why not send an empty record with the _common property_ as 
scope?
Well, I prefer to have a clear design so that this draft is really 
independent, with its withdrawal mechanism.
Otherwise, you're telling in this I-RFC the standard track IPFIX 
protocol should consider an empty record as a withdrawal. This could be 
true a true assumption, but this is not clearly explained/specified!
>>> treatment at the Observation Point; (iii) any selection state that 
>>> may be maintained by the Selection Process.
>>>
>>> If filtering is applied to select a series of packets, using the 
>>> PSAMP specifications to export per-packet information might be 
>>> relatively inefficient if the filtered fields (the common 
>>> attributes) are exported in every single record.
>>>
>> Good point. I would remove this paragraph and move to this place the 
>> one starting with:
>>
>> The idea was to put the specs in section 4 while the section 3 was 
>> the high level solution.
>
Let me ask the question differently. Do you see this repetition as problem?
>>> Like TemplateIDs the commonPropertiesIDs are generated dynamically 
>>> by the Exporting Process. Hence a restart of the Exporting Process 
>>> may imply a renumbering of commonProperiesIDs.
Fine.
>>> Exporter Observation Domain with the same Template ID received on 
>>> the SCTP association), then the Collecting Process MUST shutdown the 
>>> association.
>>> this SCTP assocation, it MUST reset the SCTP association, discard 
>>> the IPFIX Message, and SHOULD log the error as it does for malformed 
>>> IPFIX Messages.
See my previous remark about an independent draft.
>
>
>>  If the Collecting Process receives a Template Withdraw message for 
>> a   Template Record it has not received before on this SCTP 
>> assocation,    it MUST reset the SCTP association, discard the IPFIX 
>> Message, and    SHOULD log the error as it does for malformed IPFIX 
>> Messages.
>>>
>>>
>>> The text reads:
>>>   If the Common Properties are not refreshed before that lifetime 
>>> has   expired, the Collecting Process MUST discard the corresponding 
> Properties.  
In this draft, we require a mechanism such as the following:
"If the Common Properties are not refreshed before that lifetime has   
expired, the Collecting Process MUST discard the corresponding   
definition of the commonPropertiesID and any current and future   
associated Data Records. In which case, an alarm MUST be logged. "
Because there is no withdraw message with UDP:
" If UDP is selected as the transport protocol, the commonPropertiesID
  Template Withdraw Messages MUST not be used, as this method is
  inefficient due to the unreliable nature of UDP."
So I'm wondering why we are discussing this?

Let me come back to your questions:
Q: What happened to storing it until the option turned up.
A:

   If the Collecting Process has received the Specific Properties Data 
   Record before the associated Common Properties Data Record, the 
   Collecting Process MAY store the Specific Properties Data Record and 
   await the retransmission or out-of-order arrival of the Common 
   Properties Data Record.  

Q: Furthermore you shouldn't dictate the the record is discarded, just 
because you
can't tell a piece of potentially trivial information about a flow
doesn't mean it's meaningless.
A: without the Common Properties, we can't decode the records. So we 
must discard them

Q: The Common Properties in question could be anything.
A: I think we should limit this draft to Common Properties
> However, storing the data records in a queue awaiting
>> Right. So this is not an IPFIX redundancy reduction issue but an 
>> IPFIX protocol issue... as we don't want to change the protocol with 
>> this draft.
>
>>> protocol for the transfer of the Common Properties. Therefore the 
>>> use of SCTP or TCP is recommended.
>>>
Regards, Benoit

---

From: Benoit Claise <[email-masked]>
Date: Thu, 07 Dec 2006 13:33:59 +0100

Andrew,

Agreed.
Fine.

Regards, Benoit

---

From: Benoit Claise <[email-masked]>
Date: Thu, 07 Dec 2006 13:51:16 +0100

Hi Paul,
> The draft should discuss whether it's possible to define common 
> property trees, where common properties are defined in terms of other 
> common properties.
We could add a new section ( 8.5 Cascaded Common Properties.) with a few 
sentences.

My personal view is:
    - There is nothing in the protocol that prevents this
    - So this should be allowed
    - However, I'm not sure it makes a lot of sense (expressed 
differently: I haven't found the perfect example yet ;) )

Regards, Benoit.
>
> If so, it should also discuss what to do when one or more necessary 
> common properties are missing, and how to avoid circular definitions.
>

---

From: Benoit Claise <[email-masked]>
Date: Thu, 07 Dec 2006 13:51:16 +0100

Hi Paul,

My personal view is:

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Thu, 07 Dec 2006 14:02:47 +0100

Brian,
>>> Properties associated with the specific SCTP association and 
>>> specific Observation Domain, and stop decoding IPFIX Messages that 
>>> use the withdrawn Templates.
>>>
>>> 7 para 7 NEW:
>>>
>>> Properties associated with the specific SCTP association and 
>>> specific Observation Domain.
Yes, this is more accurate. Thanks.
> lifetime for connection-oriented transports, there may be no reason to 
> presume that the Template refresh and Common Property refresh timeout 
> should ever be different (indeed, things may break if they are; I've 
> not thought it all the way through); however, I do think it is clearer 
> if the Common Property refresh mechanism has its own terminology 
> rather than tying it directly to the template timeout.
>
> Thoughts?
Fine with me if this makes the draft clearer.
I would also add a note somewhere in the draft (not sure where) a 
guideline to express your concern. "there may be no reason to presume 
that the Template refresh and Common Property refresh timeout should 
ever be different"

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Thu, 07 Dec 2006 14:02:47 +0100

Brian,

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Thu, 07 Dec 2006 15:12:21 +0100

Gerhard,
> Dear all,
>
> Some meccorrections for draft-ietf-ipfix-reducing-redundancy-01:
>
> fig. 7:
> Shouldn't "index of properties A" be in front of "scrAddrA"?
>   
Well spot. This would be inline with figure 5
> p. 18:
> it's => it is
> etc... => etc.
>
> p. 21:
> "In that case, for a flow of 1000 packets the amount of data can be
> decreased _by_ more than 33 percent"
> can't => cannot
>   
This comes from section 11.2 "For a flow of 1000 packets this sums up to 
20018 bytes. This is a decrease of more than 33 percent."
Maybe the proposal is to:
1. Change section 11.2 to "For a flow of 1000 packets this sums up to 
20018 bytes. This is a decrease of 33 percents".
2. Modify the first paragraph in section 8.4:

    The passive One-Way Delay measurement example in section 11.2, 
    which is a very special application where the proposed method is
    very beneficial,  uses IPFIX to export measurement data for each
    received packet. In that case, for a flow of 1000 packets the amount
    of data can be decreased by 33 percents"

3. Add an extra paragraph in section 8.4 to address "under conditions" 
(i.e. your last remark)
    See below.

> p. 22:
> Somehow mark the notion of time in the table, e.g. with horizontal lines
> between distinct export instances? Or add timestamp fields.
>
> p. 23:
> Field Count = 3 ?
>   
Good catch.
Figure 14 should be corrected.
>
> Some comments on Section 8:
>
>    The main advantage of the method specified in this document is the
>    reduction in the amount of measurement data that has to be
>    transferred from the Exporter to the Collector.
>
> I would like to add here "under some conditions" (or something similar)
> since the proposed method does not necessarily lead to a reduction of
> the amount of data.
>
>    The transfer of the commonPropertiesIDs originates some overhead.
>    Note that IPFIX allows reduced-size encoding of Information
>    Elements.  In cases where the range of the commonPropertiesID can be
>    restricted, reduced-size encoding can be applied also to the
>    commonPropertiesID, and would result in a further bandwidth
>    efficiency gain.
>
> I would say: the overhead of the commonPropertiesID IE is very
> significant if 64 bit are used. The length of the commonPropertiesID
> field also gives you a necessary condition if you ever want to reach a
> reduced amount of data: The length must be shorter than the length of
> the replaced IEs, otherwise you even increase the amount monitoring data!
> As a consequence, reduced-size encoding should be used whenever possible.
>   
Agreed.
>    The example in section 11.2 below uses IPFIX to export measurement
>    data for each received packet.  In that case, for a flow of 1000
>    packets the amount of data can be decreased more than 33 percent.
>
> The example in 11.2 refers to OWD measurements, which is a very special
> application where the proposed method is very beneficial. Maybe you
> should mention "OWD measurements" here, since 33 percent may not be the
> general case.
>
>    While the goal of this specification is to reduce the bandwidth, the
>    efficiency might be limited.
>
> As I said before, you can even increase the amount of data with the
> proposed method. So I suggest to explain under which conditions the
> approach achieves its goal.
> How many times do I need to refer to the common properties until the
> amount of data is reduced for the first time?
> This should be a simple calculation. Even if this number cannot be
> predicted in advance in pratice, it might be useful to estimate when I
> should use the approach.
>   
Proposal: add the following two paragraphs at the end of the section 8.4 
Efficiency Gain

    While the goal of this specification is to reduce the bandwidth,
    this technique might even lead to an increase in bandwidth usage
    under certain conditions.
    Taking into account the overhead of exporting the commonPropertiesID
    values, if the commonPropertiesID values are not used in future Data
    Records, this technique would actually increase the export
    bandwidth. A typical case would be the assignments of Common
    Properties based on past observed traffic, hoping that future flows
    would contain the same characteristics.

    Note that the efficiency gain would be directly proportional the
    re-use of the defined commonPropertiesID values (In other words, the
    more we re-use a commonPropertiesID value, the better the efficiency
    gain), with a theoretical limit where all the Data Records would use
    a single commonPropertiesID.

Feel free to modify the proposal as you wish.

> Gerhard
> Juergen Quittek wrote:
> ------------------------------------------------------------------------
>

---

From: Benoit Claise <[email-masked]>
Date: Thu, 07 Dec 2006 15:12:21 +0100

Gerhard,

Good catch.
Agreed.

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 07 Dec 2006 15:33:59 +0100

Benoit, Elisa,

>> decreased _by_ more than 33 percent"

Actually, I thought I found a typo, thinking that "_by_" should be added
("decreased by"), but maybe I'm wrong and it's good English without.
Sorry if my comment was not clear.

> 3. Add an extra paragraph in section 8.4 to address "under conditions"

As you like. I'm also fine this the original phrase.

>> As a consequence, reduced-size encoding should be used whenever possible.

Elisa proposed a "RECOMMENDED" here, but see my answer:
http://www1.ietf.org/mail-archive/web/ipfix/current/msg03774.html
"You should add that the length of the commonPropertiesID field has to
be shorter than the length of the replaced fields, otherwise the amount
of data is increased.
You could note that a big difference between the length of the replaced
fields and the length of the commonPropertiesID field increases the
efficiency gain."

> Feel free to modify the proposal as you wish.

Looks good to me.

Regards,
Gerhard

-- 
Dipl.-Ing. Gerhard Münz
Computer Networks and Internet
Wilhelm Schickard Institute for Computer Science
University of Tuebingen
Sand 13 (Room B309), D-72076 Tuebingen, Germany
Phone:  +49 7071 29-70534       Fax: +49 7071 29-5220
E-mail: [email-masked]
WWW:    http://net.informatik.uni-tuebingen.de/~muenz
_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 07 Dec 2006 15:33:59 +0100

Benoit, Elisa,

Looks good to me.

Regards,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 07 Dec 2006 15:03:28 +0000

> Andrew,
[SNIP]
>      to link to information reported in separate records.

I suggested something quite similar to Elisa:
  An identifier of a set of properties that is common across more
  that one Flow.  Typically, this Information Element is used to
  link to information reported in separate records.  The identifier
  is unique per Transport Session and Observation Domain.

Note: the uniqueness of the cpID needs redefined to include the
Transport Session.

> Properties" formal definition but if we keep the "Common Properties" one.

Sounds good.

> reduce "something". How do you call this "something"?

There are already several names for the something.  Most common is Flows
or data.  Sometimes even "Flow Record" but I think that term is too
easily confused with "Data Record".  Data Record has a very specific
meaning and is not what is meant here.  A Data Record is defined by
an IPFIX Template.  The Flows that are sent from the Metering Process
to the Exporting Process have no set format.

>> Records to consider are the ones which are actually sent.
[SNIP]
> independent, with its withdrawal mechanism.

Then using the template ID to identify Common Properties can be
mentioned in passing, which I guess it is, in 8.3, however it is
also discussed in a little detail in section 4.1.1 (and figure 9),
and section 7.  It seems that these details should be removed.

> true a true assumption, but this is not clearly explained/specified!

Not at all.  Either withdrawing options in this way is valid in IPFIX,
in which case Reducing Redundancy can use it, or it a valid technique
for anybody.  Now I'm not declaring that this is a technique for all
is within the scope of the draft.  If red-red is going to use this
technique though we should decide in the WG that it is valid and say
so in the implementation guidelines or the "Managing Scope" draft.

> Let me ask the question differently. Do you see this repetition as problem?

Yes.  I think the draft would be cleared if how options work was left
to IPFIX-PROTO.  red-red should reference that, then build describe
exactly how the basic Common Properties work for the simple case, from
concept to execution.

The other variations on this can then build upon the proper foundation
of the simple case.  This way the reader can understand how the system
works before you introduce the concept of including more than one
commonPropertiesId in a Template/Data Record.

[SNIP]
> See my previous remark about an independent draft.

What?

AFAIK red-red is an information draft aimed at explaining how a technique
you can use with an IPFIX compliant Exporter/Collector.  IPFIX-PROTO says
nothing about withdrawing options data and if this draft says:
  If the Collecting Process receives a common Properties ID which has
  already been received but which has not previously been withdrawn ...
  then the Collecting Process MUST shutdown the association.

Then it is expanding on what IPFIX-PROTO says and is no longer just a
way of using IPFIX but an extension to it.  If we accept that withdrawing
options is part of IPFIX today then this is about the only extension
being asked for.

I can see why you want it, but the problem you are trying to solve here
is the same for ALL options, not just Common Property definitions.

[SNIP]
> associated Data Records. In which case, an alarm MUST be logged. "

Why is it required for common properties and not other options?
Why is it ok to use a mechanism that is not part of the protocol?

> So I'm wondering why we are discussing this?

Because you're trying to extend the protocol because you don't think
it's good enough.  While I agree that the protocol could do a better
job of handling options, I think this extension is out of scope of
the draft.

Fine.

> A: without the Common Properties, we can't decode the records. So we 

Surely that depends on what the Common Properties show.  Perhaps the
particular properties that are common aren't important.  Similarly,
if a normal full Data Record was received without something the
exporter thought was necessary, it could be discarded.

> A: I think we should limit this draft to Common Properties

It wasn't a question and your answer makes no sense.  I'm saying
that the properties defined by a particular cpID could be something
that is not important to the Collector.  Perhaps it's the
classOfService field and the collector doesn't use it.

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 07 Dec 2006 15:03:28 +0000

[SNIP]

Sounds good.

[SNIP]

[SNIP]

What?

[SNIP]

Fine.

Andrew

---

From: Elisa Boschi <[email-masked]>
Date: Thu, 07 Dec 2006 17:06:56 +0100

All,

>>      The that is unique per Observation Domain.      Typically, this 
>> Information Element is used
it does. the current commonPropertiesID definition is:

commonPropertiesID

An identifier of a set of common properties that is locally unique per 
Observation Domain and Transport Session. This ID can be used to link to 
information reported in separate records. See [IPFIX-INFO] for the 
Information Element definition.

This is also the definition that we agreed to have in IPFIX-INFO during 
last meeting

(note that the definition had not been updated in the terminology 
section of -01, while it had in section 6 - now it's corrected)

Can we actually modify IPFIX-INFO at this stage?

Elisa

>>>> records before applying the redundancy reduction mechanism but only 
>>>> a part of it.
>>    This document describes a bandwidth saving method for exporting 
>> flow    or packet information using the IP Flow Information Export 
>> (IPFIX)    protocol.
>>
>>>> I don't ;) This is the reason why the use the Common Properties 
>>>> index is a cleaner design.
>>>
>>     data, so why not send an empty record with the _common property_ 
>> as scope?
>> protocol should consider an empty record as a withdrawal. This could 
>> be true a true assumption, but this is not clearly explained/specified!
>
>>>>> the SCTP association), then the Collecting Process MUST shutdown 
>>>>> the association.
>>>>  already been received but which has not previously been 
>>>> withdrawn    (i.e. a Template Record from the same Exporter 
>>>> Observation Domain     with the same Template ID received on the 
>>>>> the IPFIX Message, and SHOULD log the error as it does for 
>>>>> malformed IPFIX Messages.
>> "If the Common Properties are not refreshed before that lifetime 
>> has   expired, the Collecting Process MUST discard the 
>> corresponding   definition of the commonPropertiesID and any current 
>> and future   associated Data Records. In which case, an alarm MUST be 
>> logged. "
>
>>    If the Collecting Process has received the Specific Properties 
>> Data    Record before the associated Common Properties Data Record, 
>> the    Collecting Process MAY store the Specific Properties Data 
>> Record and    await the retransmission or out-of-order arrival of the 
>> Common    Properties Data Record.  
>
> Fine.
>
>
>> Q: Furthermore you shouldn't dictate the the record is discarded, 
>> just because you
> ************************************************************************************************** 
>
> E-mail Confidentiality Notice and Disclaimer.
> This email and any files transmitted with it are confidential and are 
> intended solely for the use of the individual or entity to which they 
> are addressed.  Access to this email by anyone else is unauthorised.  
> If you are not the intended recipient, any disclosure, copying, 
> distribution or any action taken or omitted to be taken in reliance on 
> it, is prohibited.
>
> Email messages are not necessarily secure.  Hitachi Europe does not 
> accept responsibility for any changes made to this message after it 
> was sent.  Please note that Hitachi Europe checks outgoing email 
> messages for the presence of computer viruses.

---

From: Elisa Boschi <[email-masked]>
Date: Thu, 07 Dec 2006 17:06:56 +0100

All,

commonPropertiesID

Elisa

---

From: Benoit Claise <[email-masked]>
Date: Fri, 08 Dec 2006 17:49:45 +0100

Andrew,
>  is unique per Transport Session and Observation Domain.
Agreed. I actually saw Elisa's email after I proposed this :(
One small correction though, after re-reading the Flow and Flow Record 
definitions.
We should be using

    The commonPropertiesID is an identifier of a set of common properties
    that is locally unique per Observation Domain and Transport Session.
    Typically, this Information Element is used to link to information
    reported in separate _Data Records_.

>> Properties" formal definition but if we keep the "Common Properties" 
I went through the process of trying "Specific Properties" into 
"specific properties" and I have to admit that some of the sentences and 
figures look strange and even sometimes misleading. For example, "Common 
and specific Properties" is strange. So I have to agree with Elisa with 
keeping the definition in order to clarify the document. Anyway you 
replied in your last email "I just don't think it is a useful 
definition, but I don't feel that strongly.". So I think we can close 
this one.
> to the Exporting Process have no set format.
Acknowledged.
In the draft, I choose Flow Record, as the definitions say "The Metering 
Process generates Flow Records"
I checked every occurrence of Data Record and Flow Record in the draft 
and you're right there are some minor corrections to be done.

OLD
   Regardless of the fields content, flow records with common 
   properties export the same fields in every single flow record. 
NEW
   Regardless of the fields content, flow records with common 
   properties export the same fields in every single data record. 

OLD
   The Common Properties Flow Records MUST be sent on a reliable SCTP 
   stream. 
NEW
   The Common Properties Data Records MUST be sent on a reliable SCTP 
   stream. 

OLD
   The Collecting Process can either store the Flow Records as they 
   arrive, without reconstructing the initial Flow Record, or
   reconstruct the initial Flow Record.
NEW
   The Collecting Process can either store the Data Records as they 
   arrive, without reconstructing the initial Flow Record, or
   reconstruct the initial Flow Record.

OLD
   In addition there 
   might be less storage capacity required at the Collector side if the 
   Collector decides to store the Flow Records as they arrive, without 
   reconstructing the initial Flow Record.  
NEW
   In addition there 
   might be less storage capacity required at the Collector side if the 
   Collector decides to store the Data Records as they arrive, without 
   reconstructing the initial Flow Record.

OLD:    
   The assignment of Flow Records to common attributes could be done 
   via the commonPropertiesID and alternatively via the templateID 
   Information Element.   
NEW:    
   The assignment of common attributes could be done 
   via the commonPropertiesID and alternatively via the templateID 
   Information Element. 

OLD
   On the other hand, these methods require additional resources on 
   both the Exporter and the Collector.  The Exporter has to manage 
   Common Properties information and to assign commonPropertiesId 
   values to Flow Records. 
NEW 
   On the other hand, these methods require additional resources on 
   both the Exporter and the Collector.  The Exporter has to manage 
   Common Properties information and to assign commonPropertiesId 
   values. 
 
If you see some more required changes, let us know.

For figures 2 to 6, I think the "record" should be "Data Record".
   Figure 2: Common and Specific Properties exported in the same record 
   Figure 3: Common and Specific Properties exported in different records    
   Figure 4: Common and Specific Properties exported in the same record 
   Figure 5: Multiple Common (above) and Specific Properties (below) 
   exported in different records 
   Figure 6: Common and Specific Properties represented in one record
Alternatively, if we want to avoid the term record or Data Record in figures, let's use the trick of figure 7: 
   Figure 7: Common and Specific (packet) Properties exported 
   separately 
So let's use "exported together" versus "exported separately"

[SNIP]
> and section 7.  It seems that these details should be removed.
I understand the issue that we put at the same level the common 
properties with commonPropertiesID and with templateID, while we don't 
speak about the template withdrawal in this draft.
Here is a proposal:
We remove the templateID from section 3 and 4, we put it into section 
"8.3 commonPropertiesID vs. TemplateID scope"
And we mention in that paragraph that:
- the template withdrawal, as specified in [IPFIX-PROTO], can be used to 
withdraw Template
- the template ID could be used for common properties if the common 
properties remain constant for the live of the template.
  Otherwise we enter into some problems such as: what's happening when 
we want to change the common properties in the template.
- that we don't need template withdrawal in this is case, as the 
template withdrawal is covered by [IPFIX-PROTO] already.
- If the Template is withdrawn then the associated data is invalid too.
> exactly how the basic Common Properties work for the simple case, from
You should be speaking about the beginning of section 4.
As many clarifications to the scope concept had to be made on the 
mailing list, we thought that adding this extra paragraph of 
explanations would benefit the readers...
> is the same for ALL options, not just Common Property definitions.

This draft, even if this will be an Informational RFC, is an extension 
on how to use IPFIX in this particular case of redundancy reduction.
So, yes, this draft can be more specific. We're trying to solve the only 
case of the Common Property.
> Why is it required for common properties and not other options?
There are differences between the data records sent by option data record.
Example1: NetFlow V9 exports the ifIndex, IfDescr matchings in Option 
Template Records
Example2: We could think to export the refresh timeout in Option 
Template Records
Example3: The Common Properties ID for redundancy reduction.
Even if the 3 examples use Option Template Records, the fundamental 
difference is in the meaning of the I.E.s:
    - example 1 and 2 contain I.E.s that are self explaining: they mean 
something by themselves. We don't need those for the data records to be 
meaningfull
    - example 3: the Common Properties ID MUST be present in order to 
decode the initial Flow Record
This implies that:
    1. we can't treat all options template record the same way.
    2. "If the Common Properties are not refreshed before that lifetime 
has expired, the Collecting Process MUST discard the corresponding 
definition of the commonPropertiesID and any current and future 
associated Data Records. In which case, an alarm MUST be logged. "

> Why is it ok to use a mechanism that is not part of the protocol?
Because this draft is an extension.
If we didn't have to specify some more specifications compared to 
[IPFIX-PROTO], then an extra example in [IPFIX-AS] would be enough.
> exporter thought was necessary, it could be discarded.
Exactly, this is the reason why I made a distinction between the 
different examples in my previous comment.
> classOfService field and the collector doesn't use it.
We agree, aren't we? See my previous remark (example 1 and 2)

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Fri, 08 Dec 2006 17:49:45 +0100

Andrew,

Acknowledged.

OLD

OLD

OLD

OLD
   In addition there 

OLD:    

OLD

[SNIP]

Regards, Benoit.

---

From: Andrew Johnson <[email-masked]>
Date: Fri, 08 Dec 2006 17:07:43 +0000

Benoit Claise wrote:
>     reported in separate  Data Records .

This looks fine.

> definition, but I don't feel that strongly.". So I think we can close 

Yes, I think the draft _could_ be written without out but if it's a lot
of work then it's probably not worth the effort.

> If you see some more required changes, let us know.

These changes all look fine.

OK.

> - If the Template is withdrawn then the associated data is invalid too.

I think your suggestion is a good one, but I think using the Template ID
may be a bad idea for Common Properties and perhaps it shouldn't be
suggested at all in the draft.  I will start a new thread to specifically
discuss this issue.

> explanations would benefit the readers...

I think you can still refer to section 3.4.2 of [IPFIX-PROTO], but
by all means clarify what you need to.

> case of the Common Property.

Yes, I think I'll have to agree that some extensions will be needed,
even if some of the techniques aren't really prohibited in IPFIX-PROTO,
and fall into a grey area.

Please detail the extensions in an Appendix as most of the methods used
in the draft are from IPFIX-PROTO and there are only a few areas that
you expand upon.  It would be good for the implementer of an IPFIX
compliant device to clearly see what the differences are.

> We agree, aren't we? See my previous remark (example 1 and 2)

We agree.  With an identifier such as cpID, if it's not defined then
you have no idea what you are missing.  With data associated with
an IE that is valid by itself (ifIndex for example) then you don't
need the definition, and in fact wouldn't know you were missing it.

regards

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Fri, 08 Dec 2006 17:07:43 +0000

This looks fine.

OK.

regards

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Sat, 09 Dec 2006 10:22:53 +0000

Hi all

There are a few issues with using the Template ID as scope for Common
Properties.  I am tempted to say that we shouldn't use the Template ID for
scope for anything, except perhaps the template refresh time, or other
data that is associated with the template, but not the records defined by
that template.

Issue 1:
The Common Properties usually contain information that is vital to the
understanding of the flow.  So much so, that the draft implies that any
Data Record that is received containing an undefined commonPropertiesID
should be discarded.

Now when the template ID is used then the Collecting Process has no way
to know that this Template has further associated Common Properties.  As
such, it will continue to decode the Data Records, possibly writing them
to disk, without knowing that it is missing vital information.

Issue 2:
Some collectors will simply write all the Data Records, options and flows,
to disk and allow reconciliation later.  For current options that use
things like the ifIndex, this is fine.  Using the TemplateID will now
require them to store the ObservationDomainID and TemplateID with every
received record.  These identifiers are really for use in the Transport
Session and are not supposed to be stored externally after decode.

Issue 3:
When sending information associated with the Template, such as the
refresh time for the template, then you would expect to send an option
Data Record containing the TemplateID as scope and the refresh time as
the data.  So there are two issues here:
  a) Is information sent with a TemplateID as scope associated with the
     template or with the Data Records associated with the template?

  b) If these features were used simultaneously then because they would
     be sending options Data Records with the same scope then they would
     overwrite each others data.  (The way around this is to combine the
     data, sending a Data Record with the Template ID as scope and the
     Common Properties AND the refresh time as data).

Thoughts?

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Sat, 09 Dec 2006 10:22:53 +0000

Hi all

Issue 1:

Issue 2:

Issue 3:

Thoughts?

Andrew

---

From: [email-masked]
Date: Sat, 9 Dec 2006 18:08:05 +0000

Hey Andrew,
Some of your statements are starting to get at why argus is doing what its doing, with regard to scoping and source identification.  However, I'm afraid that you guys have gone down the current path too far?

Carter

Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax  

-----Original Message-----
From: Andrew Johnson <[email-masked]>
Date: Sat, 09 Dec 2006 10:22:53 
To:[email-masked]
Subject: [IPFIX] Template ID as scope for Common Properties

Hi all

Issue 1:

Issue 2:

Issue 3:

Thoughts?

Andrew

---

From: [email-masked]
Date: Sat, 9 Dec 2006 18:08:05 +0000

Hey Andrew,

Carter

Carter Bullard
QoSient LLC

-----Original Message-----
From: Andrew Johnson <[email-masked]>
Date: Sat, 09 Dec 2006 10:22:53 
To:[email-masked]

Hi all

Issue 1:

Issue 2:

Issue 3:

Thoughts?

Andrew

---

From: Juergen Quittek <[email-masked]>
Date: Sun, 10 Dec 2006 11:16:40 +0100

Hi Andrew,

> to know that this Template has further associated Common Properties.  As

Sorry, I don't get the point here.  Could you elaborate it?

> Session and are not supposed to be stored externally after decode.

Maybe this issue could be solved by a structured IPFIX file format?

Thanks,

    Juergen

-- 
Juergen Quittek        [email-masked]       Tel: +49 6221 4342-115
NEC Europe Ltd.,       Network Laboratories        Fax: +49 6221 4342-155
Kurfuersten-Anlage 36, 69115 Heidelberg, Germany   http://www.netlab.nec.de

---

From: Juergen Quittek <[email-masked]>
Date: Sun, 10 Dec 2006 11:16:40 +0100

Hi Andrew,

Thanks,

    Juergen

---

From: Andrew Johnson <[email-masked]>
Date: Sun, 10 Dec 2006 12:10:58 +0000

> Sorry, I don't get the point here.  Could you elaborate it?

Certainly.  In the draft, although it is not explicitly said, if a Data
Record is recieved that contains a cpID that has not been defined then it
should be considered meaningless or at least incomplete and discarded.
>From section 7.1:

The argument is that the Flow is complete without both the Specific
Properties and the Common Properties.  So if a Data Record is received
that has a cpID that isn't defined then the Collecting Process knows
that something is missing and will discard the Flow (perhaps after
waiting a short time to see if the definition is simply late).

Now, when the TemplateID is used to associate the Common Properties
with Data Records that are defined by the Template, and that definition
is lost before reaching the Collector, then it has no way of knowing
that this vital information is lost and will continue to decode the
Data Records without the Common Properties.

> Maybe this issue could be solved by a structured IPFIX file format?

We can't dictate the output of the Collecting Process just so that a
Collector can use Reducing Redundancy.  The Templates and Template IDs
are part of the transport mechanism and I don't think they should form
part of the data that is output by the Collecting Process.

I think Issue 3a below is a big one.  Template IDs should refer to
Templates and not Data Records that are defined by those Templates.

regards

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Sun, 10 Dec 2006 12:10:58 +0000

regards

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Sun, 10 Dec 2006 12:14:05 +0000

[email-masked] wrote:
> Hey Andrew,
> Some of your statements are starting to get at why argus is doing what
> its doing, with regard to scoping and source identification.  However,
> I'm afraid that you guys have gone down the current path too far?

So what is Argus doing and why?

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Sun, 10 Dec 2006 12:14:05 +0000

Andrew

---

From: Carter Bullard <[email-masked]>
Date: Sun, 10 Dec 2006 23:19:02 -0500

Hmmmm, well argus has had its IP flow exchange protocol for about
15 years now, and we decided over the years not to do many of the things
that the IPFIX WG is considering.  We've done a number of control  
channel
data channel strategies, compression schemes, template oriented (not to
the extent of IPFIX) messaging between probe and collector, etc .... and
found that being clever was not really very useful.  Data reduction  
methods
can be interesting, but you have to be careful, especially with scoping,
when there are multiple readers, and they are interested in just a  
subset
of the data.

We decided not to use the approach proposed in the WG.

Carter

---

From: Carter Bullard <[email-masked]>
Date: Sun, 10 Dec 2006 23:19:02 -0500

Carter

---

From: Carter Bullard <[email-masked]>
Date: Mon, 11 Dec 2006 11:38:22 -0500

I guess its not fair just to state, "we don't use it", and leave it  
at that.

One of the primary requirements of argus records, is that we can
provide complete accountability of the source and disposition of argus
records.  If an intermediate process modifies a record, it must "own"  
the
record, which means it must either originate the record, or re- 
originate it.
This involves inserting the modifier's source identifier, timestamps,  
sequence
numbers, etc ... (or at least references to these things) and provide  
some
information about the nature of the changes.

This is important for when you want to use flow records in applications
that have a high requirement for reliability and data integrity,  
which is
important for billing and most security applications, at least in our
experience.

This accountability issue tended to add a lot of overhead, so that
there weren't predictable improvements in data reduction, and in some
cases, we got data expansion, which of course was not the goal.

Data reduction works well for data at rest, not necessarily data in  
motion,
especially when reliability has a higher priority, at least that is  
some of
the argus experience.

Carter

---

From: Carter Bullard <[email-masked]>
Date: Mon, 11 Dec 2006 11:38:22 -0500

Carter

---

From: Andrew Johnson <[email-masked]>
Date: Mon, 11 Dec 2006 19:04:00 +0000

Hi Carter

Thanks, that's quite interesting.

Within IPFIX we're specifying a bunch of ways you might want to send
your information, in the case of your application you could use IPFIX,
you just have some stricter requirements about how you use it.  You
may even want some new IEs so you can do something specific.

In the case of a middle-box modifying records then the Observation
Domain would be considered to have changed and it is likely that the
middle-box would have to start inserting identification of the
original Observation Domain (aka source) into the records.

That said, the whole options thing isn't particularly clear right
now, and the need for more guidance has been identified.  I'm hoping
to get the ball rolling in the next couple of months...

Cheers

Andrew

Carter Bullard wrote:
> I guess its not fair just to state, "we don't use it", and leave it at that.
> 
> records.  If an intermediate process modifies a record, it must "own" the
> record, which means it must either originate the record, or re-originate it.
> numbers, etc ... (or at least references to these things) and provide some
> that have a high requirement for reliability and data integrity, which is
> Data reduction works well for data at rest, not necessarily data in motion,
> especially when reliability has a higher priority, at least that is some of
> the argus experience.
> 
> Carter
> 
> 
> 
>> that the IPFIX WG is considering.  We've done a number of control channel
>> when there are multiple readers, and they are interested in just a subset
>> of the data.   
>>
>>> [email-masked] <mailto:[email-masked]> wrote:
>>>> Hey Andrew,
>> [email-masked] <mailto:[email-masked]>

---

From: Andrew Johnson <[email-masked]>
Date: Mon, 11 Dec 2006 19:04:00 +0000

Hi Carter

Cheers

Andrew

---

From: Carter Bullard <[email-masked]>
Date: Mon, 11 Dec 2006 14:21:23 -0500

Hey Andrew,
I was/(am?) quite active in IPFIX, and tried early to help, but ....   
We've found that the
template strategy is pretty unusable for argus.  We have so many  
possible combinations
of objects that, in worst case, it would take 15K template  
descriptions to describe the
variations in records that a collector might see during the life of a  
probe/collector session.
There are architectural/philosophical problems.  Argus is pure pull,  
not push, we have
integrated TLS in the protocol itself, support for arbitrary data  
flow merging and splitting,
and multicast transport.  Stuff IPFIX wasn't interested in.  That  
makes using IPFIX a bit of
an issue, and I'm still befuddled about the SCTP requirement.  Many  
sites are starting to
treat SCTP as a firewall bypass technology, and as a result,  are not  
forwarding it.

Carter

On Dec 11, 2006, at 2:04 PM, Andrew Johnson wrote:

>> This involves inserting the modifier's source identifier,  
>> timestamps, sequence
>>> found that being clever was not really very useful.  Data  
>>> reduction methods

---

From: Carter Bullard <[email-masked]>
Date: Mon, 11 Dec 2006 14:21:23 -0500

Hey Andrew,

Carter

---

From: Andrew Johnson <[email-masked]>
Date: Mon, 11 Dec 2006 22:43:07 +0000

> I was/(am?) quite active in IPFIX, and tried early to help, but ....  

Yeah, I do remember you from a recent WG meeting (Vancouver?).

> of objects that, in worst case, it would take 15K template descriptions 
> to describe the

In theory it shouldn't much worse than TLV.  The Templates are the TL
and the Data Record the V.  OTOH if your data really does vary for
nearly every record then perhaps IPFIX isn't the best transport
protocol. It also doesn't sound as if you're sending well defined Flows
in the same way I'm used to.

> There are architectural/philosophical problems.  Argus is pure pull, not 
> push, we have
> integrated TLS in the protocol itself, support for arbitrary data flow 
> merging and splitting,
> and multicast transport.  Stuff IPFIX wasn't interested in.  

Yet... :)  I'm sure we'll see a number of extensions to IPFIX over the
next  couple of years.  I have to agree that there are a number of
areas that IPFIX doesn't handle particularly well, but it should
handle the applications it's currently targeted well enough.

> ... That makes using IPFIX a bit of

I think it's still early days for SCTP.  I expect it'll gain
acceptance after a while.  Meanwhile, I don't think it will be an
issue in the enterprise environment.  For the moment IPFIX will
mostly be used inside the firewall.

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Mon, 11 Dec 2006 22:43:07 +0000

Andrew

---

From: Brian Trammell <[email-masked]>
Date: Tue, 12 Dec 2006 10:48:15 -0500

> waiting a short time to see if the definition is simply late).

("Flow is incomplete", right?)

> Data Records without the Common Properties.

I've always been uneasy about scoping random stuff to templates, largely
because of a general feeling of ambiguity; this is the most succinct
explanation of one of those ambiguities that I've seen on the list...

> part of the data that is output by the Collecting Process.

Indeed, there is no concept in the file draft (nor are we planning to
have one) of requiring or recommending the persistence of identifiers
(whether they be part of the framing mechanism as tId, or part of the
record encoding as cpId) at a Collecting Process which uses IPFIX files
for storage; such recommendations would be out of scope.

Now, there _is_ a case which we will most probably address in the file
draft before Prague where odId and tId are persistent; the "netcat
collector" case where the process listening to the network is merely
copying the messages into IPFIX Files. Such a CP could be implemented
using nc(1), hence the nickname, but would not be strictly compliant (it
would be unable to terminate the association or connection in those
cases the protocol requires, because it's not actually interpreting the
messages). This is quite a special case (designed for unit testing, or
testing during the early stages of development of an IPFIX Device) and
should not be used as justification for template IDs as common property
scope.

>>>      Common Properties AND the refresh time as data).

hmmm... yes, were I not already not a fan of scoping common properties
to template ID, 3a above would convince me.

So, tally up another in the "no common properties scoped to templates"
column. Exporting Common Properties via template ID is too ambiguous to
be useful, and going to the effort to resolve the ambiguities seems
wasteful given that we have a nice shiny new IE (cpId) custom-defined
for the reducing-redundancy case.

Regards,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Tue, 12 Dec 2006 10:48:15 -0500

Regards,

Brian

---

From: Lutz Mark <[email-masked]>
Date: Thu, 14 Dec 2006 19:32:09 +0100

Hi Andrew,

> to disk, without knowing that it is missing vital information.

good point. This applies only for UDP. If the draft allows
Common Properties with TemplateID scope, this issue has to
be discussed.

We already had some discussion if one should limit this draft to
SCTP and TCP only, because on UDP the need to resend the Common
Properties does reduce the gain. We came to the conclusion that
the user has to weight what protocol to use and whether to use
Common Properties at all.

> received record.

I hope not! Of course it is additional effort for the collector to
support Common Properties. This method as well as the method using
the commonProperiesID IE.

> These identifiers are really for use in the Transport

Of course.

If the collector stores the ipfix data, it has to store all relevant
data needed by the post precession application. There are certainly
multiple ways to keep the mapping between the data records an the
Common Properties without storing TemplateIDs.

>     template or with the Data Records associated with the template?

If this is unclear, we have again a new issue for the implementation
guidelines.

Best regards,
Lutz

---

From: Lutz Mark <[email-masked]>
Date: Thu, 14 Dec 2006 19:32:09 +0100

Hi Andrew,

Of course.

Best regards,
Lutz

---

From: Andrew Johnson <[email-masked]>
Date: Mon, 18 Dec 2006 12:35:16 +0000

> Common Properties with TemplateID scope, this issue has to

It doesn't really just apply to UDP.  The Collector has to be prepared for
the Exporter to make mistakes.  In several places in the protocol some
things are highlighted that the Collector has to watch for and reset the
the SCTP association if it happens.

> Common Properties at all.

Of course.  The overhead of resending templates and options periodically
with UDP is almost negligible when you are sending a reasonable amount
of traffic.

> the commonProperiesID IE.

Why?  Using the IE the Collector can simply store the the common properties
in a lookup table (it will need to add some timestamp info) and do the
lookups when needed.  Some Collectors will do this today already for
options.  Overall, it's a more efficient way to store the info and demands
less processing overhead for the Collector.

> Common Properties without storing TemplateIDs.

Perhaps, but using the common properties ID is something that will
work with existing Collectors.  At the very least it needs noted that
the Collector may need to be extended in this way.

> If this is unclear, we have again a new issue for the implementation

I think IPFIX-INFO is clear.  It defines the templateID as:
  An identifier of a Template that is locally unique within an
  Observation Domain.

So the templateID refers to the template.  This is pretty logical
really and I don't think there is any need to redefine it so we can
save a couple of bytes occasionally.

regards

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Mon, 18 Dec 2006 12:35:16 +0000

regards

Andrew