Subject: [IPFIX] WG last call on draft-ietf-ipfix-reducing-redundancy-01.txt until December 1st

From: Juergen Quittek <[email-masked]>
Date: Mon, 13 Nov 2006 20:17:02 +0100

Dear all,

As agreed at the IPFIX session in San Diego last week
this is the WG last call on the Internet draft on
Reducing redundancy in IPFIX and PSAMP reports,
draft-ietf-ipfix-reducing-redundancy-01.txt.

The WG last call starts today and will be closed on
December 1st.

Please review the draft and send your comments!  Please also
send a message to the mailing list if you reviewed it and
have no comment to make!

Thanks,

    Juergen

_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Brian Trammell <[email-masked]>
Date: Fri, 17 Nov 2006 00:49:27 -0500

All,

I reviewed the reducing redundancy draft. Looks good, other than a  
couple of minor general comments.

First, we should keep this text parallel with some SCTP-related  
changes in the IPFIX protocol. Reliability in SCTP is selected per- 
message, not per-stream, as Randall Stewart pointed out in San Diego.  
So:

4.1.1 OLD:

Common Properties MUST be exported via Data Records defined by an  
Option Template Record and MUST be sent only once with the reliable  
stream of PR-SCTP association or within the TCP connection.

4.1.1 NEW:

Common Properties MUST be exported via Data Records defined by an  
Option Template Record. Like Template Records, they MUST be sent only  
once per SCTP association or TCP connection, and MUST be sent  
reliably via SCTP if SCTP is the transport protocol.

5.1 OLD:

The Common Properties Flow Records MUST be sent on a reliable SCTP  
stream.

5.1 NEW:

The Common Properties Flow Records MUST be sent reliably.

Second, there appears to be some spurious mention of templates in  
section 7, presumably as the cpId management text evolved from  
template management text:

7 para 5 OLD:

(i.e. a commonPropertiesID from the same Exporter Observation Domain  
with the same Template ID received on the SCTP association)

7 para 5 NEW:

(i.e. a commonPropertiesID from the same Exporter Observation Domain  
received on the SCTP association)

7 para 7 OLD:

If a Collecting Process receives a Common Properties Withdrawal  
message, the Collecting Process MUST delete the corresponding Common  
Properties associated with the specific SCTP association and specific  
Observation Domain, and stop decoding IPFIX Messages that use the  
withdrawn Templates.

7 para 7 NEW:

If a Collecting Process receives a Common Properties Withdrawal  
message, the Collecting Process MUST delete the corresponding Common  
Properties associated with the specific SCTP association and specific  
Observation Domain.

7.1 para 3 OLD:

    The Collecting Process MUST NOT decode any further Data Records  
which
    are associated with the expired Common Properties.  If a Common
    Property is refreshed with a Template Record that differs from the
    previous received Template Record, the Collecting Process SHOULD log
    a warning and replace the previous received Common Property with the
    new one.  The Template lifetime at the Collecting Process MUST be at
    least 3 times higher than the Template refresh timeout configured on
    the Exporting Process.

7.1 para 3 NEW:

    The Collecting Process MUST NOT decode any further Data Records  
which
    are associated with the expired Common Properties.  If a Common
    Property is refreshed with a definition that differs from the
    previous definition, the Collecting Process SHOULD log
    a warning and replace the previous received Common Property with the
    new one.  The Common Property lifetime at the Collecting Process  
MUST be at
    least 3 times higher than the Common Property timeout configured on
    the Exporting Process.

7.1 para 4 OLD:

    The Collecting Process SHOULD accept Data Records without the
    associated Common Property(ies) required to decode the Data Record.
    If the Common Property(ies) have not been received at the time Data
    Records are received, the Collecting Process SHOULD store the Data
    Records for a short period of time and decode them after the  
Template
    Records (or other definitions) are received.  The short period of
    time MUST be lower than the lifetime of definitions associated with
    identifiers considered unique within the UDP session.

7.1 para 4 NEW:

    The Collecting Process SHOULD accept Data Records without the
    associated Common Properties required to decode the Data Record.
    If the Common Properties have not been received at the time Data
    Records are received, the Collecting Process SHOULD store the Data
    Records for a short period of time and decode them after the Common
    Properties definitions are received.  The short period of
    time MUST be lower than the lifetime of definitions associated with
    identifiers considered unique within the UDP session.

Regards,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Fri, 17 Nov 2006 00:49:27 -0500

All,

4.1.1 OLD:

4.1.1 NEW:

5.1 OLD:

5.1 NEW:

7 para 5 OLD:

7 para 5 NEW:

7 para 7 OLD:

7 para 7 NEW:

7.1 para 3 OLD:

7.1 para 3 NEW:

7.1 para 4 OLD:

7.1 para 4 NEW:

Regards,

Brian

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 17 Nov 2006 11:39:54 +0100

Dear all,

Some meccorrections for draft-ietf-ipfix-reducing-redundancy-01:

fig. 7:
Shouldn't "index of properties A" be in front of "scrAddrA"?

p. 18:
it's => it is
etc... => etc.

p. 21:
"In that case, for a flow of 1000 packets the amount of data can be
decreased _by_ more than 33 percent"
can't => cannot

p. 22:
Somehow mark the notion of time in the table, e.g. with horizontal lines
between distinct export instances? Or add timestamp fields.

p. 23:
Field Count = 3 ?

Some comments on Section 8:

   The main advantage of the method specified in this document is the
   reduction in the amount of measurement data that has to be
   transferred from the Exporter to the Collector.

I would like to add here "under some conditions" (or something similar)
since the proposed method does not necessarily lead to a reduction of
the amount of data.

   The transfer of the commonPropertiesIDs originates some overhead.
   Note that IPFIX allows reduced-size encoding of Information
   Elements.  In cases where the range of the commonPropertiesID can be
   restricted, reduced-size encoding can be applied also to the
   commonPropertiesID, and would result in a further bandwidth
   efficiency gain.

I would say: the overhead of the commonPropertiesID IE is very
significant if 64 bit are used. The length of the commonPropertiesID
field also gives you a necessary condition if you ever want to reach a
reduced amount of data: The length must be shorter than the length of
the replaced IEs, otherwise you even increase the amount monitoring data!
As a consequence, reduced-size encoding should be used whenever possible.

   The example in section 11.2 below uses IPFIX to export measurement
   data for each received packet.  In that case, for a flow of 1000
   packets the amount of data can be decreased more than 33 percent.

The example in 11.2 refers to OWD measurements, which is a very special
application where the proposed method is very beneficial. Maybe you
should mention "OWD measurements" here, since 33 percent may not be the
general case.

   While the goal of this specification is to reduce the bandwidth, the
   efficiency might be limited.

As I said before, you can even increase the amount of data with the
proposed method. So I suggest to explain under which conditions the
approach achieves its goal.
How many times do I need to refer to the common properties until the
amount of data is reduced for the first time?
This should be a simple calculation. Even if this number cannot be
predicted in advance in pratice, it might be useful to estimate when I
should use the approach.

Regards,
Gerhard

-- 
Dipl.-Ing. Gerhard Münz
Computer Networks and Internet
Wilhelm Schickard Institute for Computer Science
University of Tuebingen
Sand 13 (Room B309), D-72076 Tuebingen, Germany
Phone: +49 7071 29-70534 / Fax: +49 7071 29-5220
E-mail: [email-masked]
WWW:    http://net.informatik.uni-tuebingen.de/~muenz
_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 17 Nov 2006 11:39:54 +0100

Dear all,

fig. 7:

p. 18:
it's => it is
etc... => etc.

p. 21:

p. 22:

p. 23:
Field Count = 3 ?

Regards,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Sat, 18 Nov 2006 02:04:01 +0000

Hi all

I have reviewed the reducing redundancy document and I have the
following comments and questions.  I'm afraid I've got quite a
lot to say so please ask about anything that isn't clear or you
disagree with.

regards

Andrew

Questions
=========
a) As there are no enhancements or additions (after all, their
   just options) is this a proposed Informational RFC?
b) If so, can MUST, SHOULD etc, be used outside of directly
   (or perhaps indirectly) quoting the standard?

If there are references to PSAMP drafts, doesn't that
block this from becoming an RFC until the PSAMP ones
are RFCs?  Do we have to mention PSAMP so much anyway,
as it's just a specialised use case of IPFIX?

Since a flow is defined as a group of packets that share
a set of common properties, i.e. key fields, is there
any danger that the commonPropertiesID will be wrongly
assumed to refer to only the complete set of key fields?

Is there any need to define a new term for "Specific
Properties"?  These are just part of a normal Data Record.
For example:
     "...  The rows are then referenced by the Specific Properties
      by using the appropriate value for the Common Properties
      identifier."

could simply be:
     "...  The rows are then referenced by Data Records by
      using the appropriate value for the Common Properties
      identifier."

Are option withdrawal messages part of IPFIX?
Are they meaningful in IPFIX?  i.e. per the PROTO will an option
with scope but no data be deemed invalidated by the Collector.
Scope without data is the same as never having recieved the
mapping in the first place.  I think that it would be up to the
implementation but it seems like a good idea so if we can't add
it to the PROTO can we put it in the implementation guidelines?

How do you propose to withdraw common properties when they are
associated with a template rather than the Common Properties index?

General Comments
================
I think, sections 3.1.2, 8.2 and 8.3 would be better grouped
into a 3.2 Advanced Techniques section.

I think 3.2 Per Packet Data Reduction, should be removed and
any information that is not in section 3.1 should be added to
to the PSAMP example.

I think the the per-packet example is a good example of
per flow reducing redundancy and section 11.1.1 can be removed.

Labels for the figures would be better if they were centred.

There is all sorts of extra spacing at the end of the lines.

Specific Comments
=================
2. Terminology 

Tabbing is inconsistent in definition of "Selector":
            (i)   the Packet Content; 
                   
           (ii)  information derived from the packet's treatment at the 
                  Observation Point; 
                   
            (iii) any selection state that may be maintained by the  
                  Selection Process. 

3.1.2   Multiple Data Reduction

I don't think this section is explained very well.  From my original
email to Elisa and Lutz (18th Nov 2005):
Andrew Johnson wrote:
> Flows
> =====
> Flow A: flowId 1, flowId 11, 30 packets, 6000 bytes
> Flow B: flowID 1, flowId 12, 50 packets, 9500 bytes
> Flow C: flowId 2, flowId 11, 60 packets, 8000 bytes
> Flow D: flowID 2, flowId 12, 10 packets, 1500 bytes
> 
> Options
> =======
> flowId  1 - destinationIpv4address 10.0.0.1, destinationTcpPort 80
> flowId  1 - destinationIpv4address 10.0.0.5, destinationTcpPort 80
> 
> flowId 11 - sourceIpv4address 10.0.1.10, sourceTcpPort 1932
> flowId 12 - sourceIpv4address 10.0.1.15, sourceTcpPort 1842

This shows how 4 (2+2) options can be expanded to 4 combinations of
Data Record to reduce redundancy without the need to define 4 complete
sets of Common Properties and therefore not save any bandwidth.  The
more options you have the better this works.  20 options (10+10) would
give you up to 100 possible pairs common properties.

This isn't really made clear in the draft text, with only "flow1"
having two indices.   The example in 11.1.2 doesn't even have any!

While I agree that using a template that doesn't include the cpID
is a good idea to efficiently communicate flows with not so common
properties, however I would call this an "Advanced Technique" to be
listed in the section I proposed earlier.

3.2 Per Packet Data Reduction

The text reads:
   If filtering is applied to select a series of packets, using the 
   PSAMP specifications to export per-packet information might be 
   relatively inefficient if the filtered fields (the common 
   attributes) are exported in every single record.

However in section 6.5.2.5 (Property Match Filtering) of PSAMP-PROTO
it is explained how the filter information MUST be sent.  This is
designed so that the selector ID will match the one from the Data
Record (via the selectionSequenceId) and perform exactly the function
that is being proposed here for the commonPropertiesId.

4. Specifications for bandwidth saving information export

What is being specified here that isn't true for ALL options as
described in IPFIX-PROTO?

I think section 3 does a better job of explaining how all this
works.  I'm really not sure what section 4 adds.
(Aside: figure 9 refers to Common Sroperties).

6. commonPropertiesID Management

The text reads:
   The commonPropertiesID values MAY be assigned sequentially, but it�s 

It seems the "it's" is using an invalid character rather than an
apostrophe.

7. The Collecting Process Side

The text reads:
   Like TemplateIDs the commonPropertiesIDs are generated dynamically 
   by the Exporting Process.  Hence a restart of the Exporting Process 
   may imply a renumbering of commonProperiesIDs. 

The collector can't know if the Exporter Process restarted, this is
why we rescoped the commonPropertiesID to the Transport Session, so
the above text is now redundant.

The text reads:
   Common Properties IDs are unique per SCTP association and per 
   Observation Domain.  If the Collecting Process receives a common 
   Properties ID which has already been received but which has not 
   previously been withdrawn (i.e. a commonPropertiesID from the same 
   Exporter Observation Domain with the same Template ID received on the 
   SCTP association), then the Collecting Process MUST shutdown the 
   association.

and:
   If the Collecting Process receives a Common Properties Withdrawal 
   message for a Common Properties that it has not received before on 
   this SCTP assocation, it MUST reset the SCTP association, discard the 
   IPFIX Message, and SHOULD log the error as it does for malformed 
   IPFIX Messages.    

I think this might go against IPFIX-PROTO.

The text reads:
   If the Common Properties are not refreshed before that lifetime has 
   expired, the Collecting Process MUST discard the corresponding 
   definition of the commonPropertiesID and any current and future 
   associated Data Records.  In which case, an alarm MUST be logged. 

What happened to storing it until the option turned up.  Furthermore
you shouldn't dictate the the record is discarded, just because you
can't tell a piece of potentially trivial information about a flow
doesn't mean it's meaningless.  The Common Properties in question
could be anything.

Besides, we're not changing the protocol here, right?

8.1 Transport Protocol Choice 
    
The text reads:
   The proposed method is most effective using a reliable transport 
   protocol for the transfer of the Common Properties.  Therefore the 
   use of SCTP or TCP is recommended.

Note that SCTP isn't necessarily reliable.

8.3 commonPropertiesID vs. TemplateID scope 

Additional flow properties can be tied to almost any I.E.  For example
a particular interface could be associated with the one IP address it
serves.

8.4 Efficiency Gain 

The text reads:
  "...it can�t predict whether future..."

Weird apostrophe character again.

Examples:

1. There is no padding between records, it's against IPFIX-PROTO,
   unless you use the special padding I.E.
2. The templates define the counts as 4 octets but the records show
   them as 2.
3. Since 11.1.1 and 11.2 don't demonstrate anything different, can
   we have just one example?
4. The tabbing is sometimes a little of on the run though of the I.E.s,
   and I think we can loose the commas.  e.g.

OLD
     - the source IPv4 Address, sourceIPv4Address [IPFIX-INFO], with a 
        type of 8 and a length of 4 octets 

NEW
     -  the source IPv4 Address: sourceIPv4Address [IPFIX-INFO] with a 
        type of 8 and a length of 4 octets

5. What is a packet ID anyway?  In general PSAMP uses sampling to gather
   detailed information about a small proportion of the network traffic.
   In most cases there won't be many common factors between the reports.
   The real exception is hash-based filtering, which is what I think the
   example uses.

6. Do we have to use enterprise specific things here?  What about:
     observationTimeMilliseconds (323),
     digestHashValue (326),
     ipTotalLength (224)

---

From: Andrew Johnson <[email-masked]>
Date: Sat, 18 Nov 2006 02:04:01 +0000

Hi all

regards

Andrew

Questions
=========

could simply be:

General Comments
================

Specific Comments
=================
2. Terminology 

                   

The text reads:

The text reads:

The text reads:

The text reads:

and:

The text reads:

    
The text reads:

8.4 Efficiency Gain 

The text reads:

Examples:

OLD

NEW

---

From: Juergen Quittek <[email-masked]>
Date: Sat, 18 Nov 2006 17:07:13 +0100

Hi Andrew,

>    just options) is this a proposed Informational RFC?

Yes. That is what our charter says about this document.

>    (or perhaps indirectly) quoting the standard?

Yes, it can be used.  These keywords express what is a MUST/SHOULD/MAY
for a certain specification.  The use of these words does not imply
that the specification is a standards track document.

For example the Cisco NetFlow RFC 3954 is informational but uses
MUST, SHOULD, etc. for stating what is a MUST or SHOULD for NetFLow
implementations.

Thanks,

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Sat, 18 Nov 2006 17:07:13 +0100

Hi Andrew,

Thanks,

    Juergen

---

From: Paul Aitken <[email-masked]>
Date: Mon, 20 Nov 2006 15:40:53 +0000

Dear all,

> draft-ietf-ipfix-reducing-redundancy-01.txt.

The draft should discuss whether it's possible to define common property 
trees, where common properties are defined in terms of other common 
properties.

If so, it should also discuss what to do when one or more necessary 
common properties are missing, and how to avoid circular definitions.

-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.

---

From: Paul Aitken <[email-masked]>
Date: Mon, 20 Nov 2006 15:40:53 +0000

Dear all,