Subject: [IPFIX] Draft Standard version of RFC5101

From: Benoit Claise <[email-masked]>
Date: Sun, 27 Mar 2011 16:26:50 +0200

Dear all,

Here is the list of things to be changed in the Draft Standard version 
of RFC5101.
Along the years, I've been collecting (from different people a list of 
what could be improved in IPFIX. Many points come from discussions with 
Paul Aitken.

Some of points below should be discussed.

1. what must be changed is obviously 
http://www.rfc-editor.org/errata_search.php?rfc=5101

2. the email thread 
http://www.ietf.org/mail-archive/web/ipfix/current/msg05652.html 
suggests to change the definition from "IP Traffic Flow or Flow" to 
"Traffic Flow or Flow", and from "IP packets" to "packets" when RFC5101 
will go from Proposed Standard to Draft Standard.

3. Different Template Record management between the transport protocol 
(feedback from the IPFIX interop)

SCTP transport

    If the Collecting Process receives a Template that has
    already been received but that has not previously been withdrawn
    (i.e., a Template Record from the same Exporter Observation Domain
    with the same Template ID received on the SCTP association), then the
    Collecting Process MUST shut down the association.

So an identical Template Record is resent (like in UDP), the Collecting 
Process MUST shut down the association
First problem: we don't have the same statement for TCP, which only says:

    If the Collecting Process receives a malformed IPFIX Message, it MUST
    discard the IPFIX Message and SHOULD log the error.

So this is not consistent as we don't know what "malformed" means.
Thinking some more about it, is this really a mistake if SCTP and TCP 
would behave as UDP, i.e. accept the exact same (Options) Template 
Record to be resent? Proposal with something such as: It's NOT 
RECOMMENDED that _identical _Template Record are resent in SCTP and 
TCP.... but it will not cause the SCTP association or TCP association to 
be shut down...
Before applying this change, we have to double-check if there are no 
complications for draft-ietf-ipfix-export-per-sctp-stream-08

4.

OLD:

Exporting Process ID
                         The identifier of the Exporting Process for
                         which lack of reliability is reported.  There
                         are three Information Elements specified in
                         [RFC5102  <http://tools.ietf.org/html/rfc5102>] that can be used for this purpose:
                         exporterIPv4Address, exporterIPv6Address, or
                         exportingProcessId.  This Information Element
                         MUST be defined as a Scope Field.

NEW:

Exporting Process ID
                         The identifier of the Exporting Process for
                         which lack of reliability is reported.  There
                         are three Information Elements specified in
                         [RFC5102  <http://tools.ietf.org/html/rfc5102>] that can be used for this purpose:
                         exporterIPv4Address, exporterIPv6Address, or
                         exportingProcessId.  This Information Element,
                         _or these Information Elements if multiple are
_                         _needed,_  MUST be defined as a Scope Field

Justification: both the address and process ID are needed to identify a 
specific exporter in the case  of multiple exporters at the same address.

5.

10.3.6.  Template Management

    Following a configuration change that can modify the interpretation
    of the Data Records (for example, a sampling rate change) a new
    Template ID MUST be used, and the old Template ID MUST NOT be reused
    until its lifetime (see Section 10.3.7) has expired.

In section 10.3.7.  Collecting Process, we read:

    The Template lifetime at the Collecting Process MUST be at least 3
    times higher than the Template refresh timeout configured on the
    Exporting Process.

In Flexible NetFlow, template refresh is in units of packets, so that 
template export scales with the volume of exported data.
If we're no longer exporting any packets then the "3 times higher" 
threshold will never be reached... so the templates will never expire.
eg, if we're exporting templates every 1,000 packets, then the lifetime 
on the collector should be 3,000 packets. Yet once we're no longer using 
the template, we'll be exporting 0 packets...
So the RFC should specify the units, or give an alternative mechanism.

6.

Section 8:
    If the measurement parameters change such that a new Template is
    required, the Template MUST be withdrawn (using a Template Withdraw
    Message and a new Template definition) or an unused Template ID MUST
    be used.  Examples of the measurement changes are: a new sampling
    rate, a new Flow expiration process, a new filtering definition, etc.

A new sampling rate etc may not require a new template at all; it may be 
expressed with the same template, just different data. The template 
belongs to the transport layer and really has nothing to do with the 
sampler change!

Proposal: remove "a new sampling rate"

7.

Section 4.3.  The Exporting Process Reliability Statistics Option Template

    time first flow dropped
                         The timestamp of the first Flow was dropped by
                         the Metering Process.  For this timestamp, any
                         of the "flowStart" timestamp Information
                         Elements flowStartMilliseconds,
                         flowStartMicroseconds, flowStartNanoseconds, and
                         flowStartDeltaMicroseconds can be used.

    time last flow dropped
                         The timestamp of the last IP packet that was
                         ignored by the Metering Process.  For this
                         timestamp, any of the "flowEnd" timestamp
                         Information Elements flowEndMilliseconds,
                         flowEndMicroseconds, flowEndNanoseconds, and
                         flowEndDeltaMicroseconds can be used.

Firstly, these definitions are inconsistent since the names and the 
first definition say "flow" while the second definition says "IP 
packet". Obviously "IP packet" != "flow" :-o

Secondly, "The timestamp of the first Flow was dropped by the Metering 
Process." is bad English: at least it's missing "that".

8.

Section 4.3.  The Exporting Process Reliability Statistics Option Template

These statistics where defined before we introduced the concept of 
Transport Session.
  This full section 3.4 should be re-specified for "4.3 The Transport 
Session Reliability Statistics Option Template"

9.

Section 4.2.  The Metering Process Reliability Statistics Option Template

Suppose a packet cannot be classified before it's dropped. Which
Metering Process (meteringProcessId) should report this?

Suppose the packet pertains to several monitors. If some or all of
them cannot record the packet (eg, cache full), should they all report
the loss? That might look like some random number of packets had been
lost, when in fact it's only one.

10.

Do we want to have a new IE  for the template lifetime, to be sent in an 
Options Template Record.

Regards, Benoit.

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Mon, 28 Mar 2011 13:39:05 +0200

Hi,

When progressing from Proposed to Draft only editorial clarifications
editorial changes are allowed. You can also point to portion of the
specification that you would like to obsolete because the implementation
experience show that they are no longer used. However, you cannot make
changes that are not 100% compatible with the version of the protocol
defined at Proposed. If such changes are needed the document needs to
recycle at Proposed. 

Are you sure that all the changes that you are suggesting are editorial
clarifications? For example change #3, change #8 and change #10? 

Thanks and Regards,

Dan

________________________________

	From: [email-masked] [mailto:[email-masked]] On
Behalf Of Benoit Claise
	Sent: Sunday, March 27, 2011 4:27 PM
	To: [email-masked]
	Subject: [IPFIX] Draft Standard version of RFC5101

	Dear all,
	
	Here is the list of things to be changed in the Draft Standard
version of RFC5101.
	Along the years, I've been collecting (from different people a
list of what could be improved in IPFIX. Many points come from
discussions with Paul Aitken.

	
	2. the email thread
	
	3. Different Template Record management between the transport
protocol (feedback from the IPFIX interop)
	
	SCTP transport
	
	   If the Collecting Process receives a Template that has
	   already been received but that has not previously been
withdrawn
	   (i.e., a Template Record from the same Exporter Observation
Domain
	   with the same Template ID received on the SCTP association),
then the
	   Collecting Process MUST shut down the association.
	
	So an identical Template Record is resent (like in UDP), the
Collecting Process MUST shut down the association
	First problem: we don't have the same statement for TCP, which
only says:
	
	   If the Collecting Process receives a malformed IPFIX Message,
it MUST
	   discard the IPFIX Message and SHOULD log the error.
	So this is not consistent as we don't know what "malformed"
means.
	Thinking some more about it, is this really a mistake if SCTP
and TCP would behave as UDP, i.e. accept the exact same (Options)
Template Record to be resent? Proposal with something such as: It's NOT
RECOMMENDED that identical Template Record are resent in SCTP and
TCP.... but it will not cause the SCTP association or TCP association to
be shut down...
	Before applying this change, we have to double-check if there
are no complications for draft-ietf-ipfix-export-per-sctp-stream-08
	
	4. 
	
	OLD:
	
	Exporting Process ID
	                        The identifier of the Exporting Process
for
	                        which lack of reliability is reported.
There
	                        are three Information Elements specified
in
	                        [RFC5102
<http://tools.ietf.org/html/rfc5102> ] that can be used for this
purpose:
	                        exporterIPv4Address,
exporterIPv6Address, or
	                        exportingProcessId.  This Information
Element
	                        MUST be defined as a Scope Field.
	NEW:
	
	Exporting Process ID
	                        The identifier of the Exporting Process
for
	                        which lack of reliability is reported.
There
	                        are three Information Elements specified
in
	                        [RFC5102
<http://tools.ietf.org/html/rfc5102> ] that can be used for this
purpose:
	                        exporterIPv4Address,
exporterIPv6Address, or
	                        exportingProcessId.  This Information
Element, 
	                        or these Information Elements if
multiple are 
	                        needed, MUST be defined as a Scope Field
	Justification: both the address and process ID are needed to
identify a specific exporter in the case  of multiple exporters at the
same address.

	5.

	   Following a configuration change that can modify the
interpretation 
	   of the Data Records (for example, a sampling rate change) a
new 
	   Template ID MUST be used, and the old Template ID MUST NOT be
reused 
	   until its lifetime (see Section 10.3.7) has expired. 

	   The Template lifetime at the Collecting Process MUST be at
least 3 
	   times higher than the Template refresh timeout configured on
the 
	   Exporting Process. 

	In Flexible NetFlow, template refresh is in units of packets, so
that template export scales with the volume of exported data. 
	If we're no longer exporting any packets then the "3 times
higher" threshold will never be reached... so the templates will never
expire. 
	eg, if we're exporting templates every 1,000 packets, then the
lifetime on the collector should be 3,000 packets. Yet once we're no
longer using the template, we'll be exporting 0 packets... 
	So the RFC should specify the units, or give an alternative
mechanism. 

	6. 
	
	Section 8: 
	   If the measurement parameters change such that a new Template
is 
	   required, the Template MUST be withdrawn (using a Template
Withdraw 
	   Message and a new Template definition) or an unused Template
ID MUST 
	   be used.  Examples of the measurement changes are: a new
sampling 
	   rate, a new Flow expiration process, a new filtering
definition, etc. 
	
	A new sampling rate etc may not require a new template at all;
it may be expressed with the same template, just different data. The
template belongs to the transport layer and really has nothing to do
with the sampler change! 

	7.
	
	Section 4.3.  The Exporting Process Reliability Statistics
Option Template
	   time first flow dropped 
	                        The timestamp of the first Flow was
dropped by 
	                        the Metering Process.  For this
timestamp, any 
	                        of the "flowStart" timestamp Information

	                        Elements flowStartMilliseconds, 
	                        flowStartMicroseconds,
flowStartNanoseconds, and 
	                        flowStartDeltaMicroseconds can be used. 
	
	   time last flow dropped 
	                        The timestamp of the last IP packet that
was 
	                        ignored by the Metering Process.  For
this 
	                        timestamp, any of the "flowEnd"
timestamp 
	                        Information Elements
flowEndMilliseconds, 
	                        flowEndMicroseconds, flowEndNanoseconds,
and 
	                        flowEndDeltaMicroseconds can be used. 
	
	Firstly, these definitions are inconsistent since the names and
the first definition say "flow" while the second definition says "IP
packet". Obviously "IP packet" != "flow" :-o 
	
	Secondly, "The timestamp of the first Flow was dropped by the
Metering Process." is bad English: at least it's missing "that". 
	
	8.
	
	Section 4.3.  The Exporting Process Reliability Statistics
Option Template
	These statistics where defined before we introduced the concept
of Transport Session.
	 This full section 3.4 should be re-specified for "4.3 The
Transport Session Reliability Statistics Option Template"

	9.
	
	Section 4.2.  The Metering Process Reliability Statistics Option
Template

	
	Suppose the packet pertains to several monitors. If some or all
of 
	them cannot record the packet (eg, cache full), should they all
report 
	the loss? That might look like some random number of packets had
been 
	lost, when in fact it's only one.

	10.
	
	Do we want to have a new IE  for the template lifetime, to be
sent in an Options Template Record.

	Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Mon, 28 Mar 2011 16:34:03 +0200

Hi Dan,
>
> Hi,
>
> When progressing from Proposed to Draft only editorial 
> clarifications editorial changes are allowed. You can also point to 
> portion of the specification that you would like to obsolete because 
> the implementation experience show that they are no longer used. 
> However, you cannot make changes that are not 100% compatible with the 
> version of the protocol defined at Proposed. If such changes are 
> needed the document needs to recycle at Proposed.
>
> Are you sure that all the changes that you are suggesting are 
> editorial clarifications? For example change #3, change #8 and change 
> #10?
>
thanks for the clarification.
#8 and #10, I agree with you: not editorial
Regarding #3, there is a gap in the specifications. Should we address 
it. The proposed solution would not bring incompatibilities.

Regards, Benoit.

>
>
> Thanks and Regards,
>
> Dan
>
>
>     ------------------------------------------------------------------------
>     *From:* [email-masked] [mailto:[email-masked]] *On
>     Behalf Of *Benoit Claise
>     *Sent:* Sunday, March 27, 2011 4:27 PM
>     *To:* [email-masked]
>     *Subject:* [IPFIX] Draft Standard version of RFC5101
>
>     Dear all,
>
>     suggests to change the definition from "IP Traffic Flow or Flow"
>     to "Traffic Flow or Flow", and from "IP packets" to "packets" when
>     RFC5101 will go from Proposed Standard to Draft Standard.
>
>     Thinking some more about it, is this really a mistake if SCTP and
>     TCP would behave as UDP, i.e. accept the exact same (Options)
>     Template Record to be resent? Proposal with something such as:
>     It's NOT RECOMMENDED that _identical _Template Record are resent
>     in SCTP and TCP.... but it will not cause the SCTP association or
>     TCP association to be shut down...
>     Before applying this change, we have to double-check if there are
>     no complications for draft-ietf-ipfix-export-per-sctp-stream-08
>
>     4.
>
>     OLD:
>
>     Exporting Process ID
>     identify a specific exporter in the case  of multiple exporters at
>     the same address.
>
>
>
>     5.
>
>     lifetime on the collector should be 3,000 packets. Yet once we're
>     no longer using the template, we'll be exporting 0 packets...
>     A new sampling rate etc may not require a new template at all; it
>     may be expressed with the same template, just different data. The
>     template belongs to the transport layer and really has nothing to
>     do with the sampler change!
>
>     the first definition say "flow" while the second definition says
>     "IP packet". Obviously "IP packet" != "flow" :-o
>
>     Do we want to have a new IE  for the template lifetime, to be sent
>     in an Options Template Record.
>
>
>     Regards, Benoit.
>
>

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Mon, 28 Mar 2011 19:05:14 +0200

Hi,

So the question that the WG needs to decide is - are these issues
important enough to justify recycling at Proposed rather than
progression to Draft?

Thanks and Regards,

Dan

	From: Benoit Claise [mailto:[email-masked]] 
	Sent: Monday, March 28, 2011 4:34 PM
	To: Romascanu, Dan (Dan)
	Cc: [email-masked]
	Subject: Re: [IPFIX] Draft Standard version of RFC5101

	Hi Dan,

		Hi,

		When progressing from Proposed to Draft only editorial
clarifications editorial changes are allowed. You can also point to
portion of the specification that you would like to obsolete because the
implementation experience show that they are no longer used. However,
you cannot make changes that are not 100% compatible with the version of
the protocol defined at Proposed. If such changes are needed the
document needs to recycle at Proposed. 

		Are you sure that all the changes that you are
suggesting are editorial clarifications? For example change #3, change
#8 and change #10? 

	thanks for the clarification.
	#8 and #10, I agree with you: not editorial
	Regarding #3, there is a gap in the specifications. Should we
address it. The proposed solution would not bring incompatibilities.
	
	Regards, Benoit.

		
		Thanks and Regards,
		
		Dan

			From: [email-masked]
[mailto:[email-masked]] On Behalf Of Benoit Claise
			Sent: Sunday, March 27, 2011 4:27 PM
			To: [email-masked]
			Subject: [IPFIX] Draft Standard version of
RFC5101

			Dear all,
			
			Here is the list of things to be changed in the
Draft Standard version of RFC5101.
			Along the years, I've been collecting (from
different people a list of what could be improved in IPFIX. Many points
come from discussions with Paul Aitken.

			
			2. the email thread
			
			3. Different Template Record management between
the transport protocol (feedback from the IPFIX interop)
			
			SCTP transport
			
			   If the Collecting Process receives a Template
that has
			   already been received but that has not
previously been withdrawn
			   (i.e., a Template Record from the same
Exporter Observation Domain
			   with the same Template ID received on the
SCTP association), then the
			   Collecting Process MUST shut down the
association.
			
			So an identical Template Record is resent (like
in UDP), the Collecting Process MUST shut down the association
			First problem: we don't have the same statement
for TCP, which only says:
			
			   If the Collecting Process receives a
malformed IPFIX Message, it MUST
			   discard the IPFIX Message and SHOULD log the
error.
			So this is not consistent as we don't know what
"malformed" means.
			Thinking some more about it, is this really a
mistake if SCTP and TCP would behave as UDP, i.e. accept the exact same
(Options) Template Record to be resent? Proposal with something such as:
It's NOT RECOMMENDED that identical Template Record are resent in SCTP
and TCP.... but it will not cause the SCTP association or TCP
association to be shut down...
			Before applying this change, we have to
double-check if there are no complications for
draft-ietf-ipfix-export-per-sctp-stream-08
			
			4. 
			
			OLD:
			
			Exporting Process ID
			                        The identifier of the
Exporting Process for
			                        which lack of
reliability is reported.  There
			                        are three Information
Elements specified in
			                        [RFC5102
<http://tools.ietf.org/html/rfc5102> ] that can be used for this
purpose:
			                        exporterIPv4Address,
exporterIPv6Address, or
			                        exportingProcessId.
This Information Element
			                        MUST be defined as a
Scope Field.
			NEW:
			
			Exporting Process ID
			                        The identifier of the
Exporting Process for
			                        which lack of
reliability is reported.  There
			                        are three Information
Elements specified in
			                        [RFC5102
<http://tools.ietf.org/html/rfc5102> ] that can be used for this
purpose:
			                        exporterIPv4Address,
exporterIPv6Address, or
			                        exportingProcessId.
This Information Element, 
			                        or these Information
Elements if multiple are 
			                        needed, MUST be defined
as a Scope Field
			Justification: both the address and process ID
are needed to identify a specific exporter in the case  of multiple
exporters at the same address.

			5.

			   Following a configuration change that can
modify the interpretation 
			   of the Data Records (for example, a sampling
rate change) a new 
			   Template ID MUST be used, and the old
Template ID MUST NOT be reused 
			   until its lifetime (see Section 10.3.7) has
expired. 

			   The Template lifetime at the Collecting
Process MUST be at least 3 
			   times higher than the Template refresh
timeout configured on the 
			   Exporting Process. 

			In Flexible NetFlow, template refresh is in
units of packets, so that template export scales with the volume of
exported data. 
			If we're no longer exporting any packets then
the "3 times higher" threshold will never be reached... so the templates
will never expire. 
			eg, if we're exporting templates every 1,000
packets, then the lifetime on the collector should be 3,000 packets. Yet
once we're no longer using the template, we'll be exporting 0 packets...

			So the RFC should specify the units, or give an
alternative mechanism. 

			6. 
			
			Section 8: 
			   If the measurement parameters change such
that a new Template is 
			   required, the Template MUST be withdrawn
(using a Template Withdraw 
			   Message and a new Template definition) or an
unused Template ID MUST 
			   be used.  Examples of the measurement changes
are: a new sampling 
			   rate, a new Flow expiration process, a new
filtering definition, etc. 
			
			A new sampling rate etc may not require a new
template at all; it may be expressed with the same template, just
different data. The template belongs to the transport layer and really
has nothing to do with the sampler change! 

			7.
			
			Section 4.3.  The Exporting Process Reliability
Statistics Option Template
			   time first flow dropped 
			                        The timestamp of the
first Flow was dropped by 
			                        the Metering Process.
For this timestamp, any 
			                        of the "flowStart"
timestamp Information 
			                        Elements
flowStartMilliseconds, 
			                        flowStartMicroseconds,
flowStartNanoseconds, and 

			   time last flow dropped 
			                        The timestamp of the
last IP packet that was 
			                        ignored by the Metering
Process.  For this 
			                        timestamp, any of the
"flowEnd" timestamp 
			                        Information Elements
flowEndMilliseconds, 
			                        flowEndMicroseconds,
flowEndNanoseconds, and 
			                        flowEndDeltaMicroseconds
can be used. 
			
			Firstly, these definitions are inconsistent
since the names and the first definition say "flow" while the second
definition says "IP packet". Obviously "IP packet" != "flow" :-o 
			
			Secondly, "The timestamp of the first Flow was
dropped by the Metering Process." is bad English: at least it's missing
"that". 
			
			8.
			
			Section 4.3.  The Exporting Process Reliability
Statistics Option Template
			These statistics where defined before we
introduced the concept of Transport Session.
			 This full section 3.4 should be re-specified
for "4.3 The Transport Session Reliability Statistics Option Template"

			9.
			
			Section 4.2.  The Metering Process Reliability
Statistics Option Template
			
			Suppose a packet cannot be classified before
it's dropped. Which 
			Metering Process (meteringProcessId) should
report this?
			
			Suppose the packet pertains to several monitors.
If some or all of 
			them cannot record the packet (eg, cache full),
should they all report 
			the loss? That might look like some random
number of packets had been 
			lost, when in fact it's only one.

			10.
			
			Do we want to have a new IE  for the template
lifetime, to be sent in an Options Template Record.

			Regards, Benoit.