Subject: Re: [IPFIX] WG last call ondraft-ietf-ipfix-reducing-redundancy-01.txt until December 1st

From: "Boschi, Elisa" <[email-masked]>
Date: Wed, 6 Dec 2006 10:57:50 -0000

Andrew, Benoit, all,

Comments, as usual, inline...

Andrew Johnson wrote:

>>> Since a flow is defined as a group of packets that share
>>> a set of common properties, i.e. key fields, is there
>>> any danger that the commonPropertiesID will be wrongly
>>> assumed to refer to only the complete set of key fields?
>>
>>
>> By definition, I think this is impossible. The flow key 
>> definition says "Each of the fields which are used to define 
>> a Flow are termed Flow Keys.". So there is nothing in common 
>> between flows. However, I understand your concern for novice readers.
>>
>> OLD section 1 "Introduction"
>>
>> Regardless of the fields content, flow records with common  
>> properties export the same fields in every single flow record.  
>> These common properties may represent values common to a collection 
>> of flows or packets, or values that are invariant over time. 
>> The reduction of redundant data from the export stream can result 
>> in a significant reduction of the transferred data.
>>
>> NEW section 1 "Introduction"
>>
>> Regardless of the fields content, flow records with common properties 
>> export the same fields in every single flow record.  
>> These common properties may represent values common to a collection 
>> of flows or packets, or values that are invariant over time.  
>> Note that the common properties don't represent the list of flow keys, 
>> which are used to define a flow definition: however, the common 
>> properties may contain some of the flow keys. The reduction of redundant 
>> data from the export stream can result in a significant reduction of 
>> the transferred data.
>
>
> My concern wasn't really directed at the draft itself, but just the 
> name "common properties".  
> From IPFIX-PROTO:
>  "A Flow is defined as a set of IP packets passing an Observation   
> Point in the network during a certain time interval.  All packets   
> belonging to a particular Flow have a set of common properties."
>
> And then from IPFIX-INFO:
>   Description of commonPropertiesId:
>      An identifier of a set of common properties that is unique per
>      Observation Domain.  Typically, this Information Element is used
>      to link to information reported in separate records.
>
> The terminology is almost identical.  The commonPropertiesId refers
> to "a set of common properties" that is quite different from ones
> that determine a Flow.

so maybe the text proposed by Benoit helps clarifying things. I agree with adding it

>
>>> Is there any need to define a new term for "Specific
>>> Properties"? These are just part of a normal Data Record.
>>> For example:
>>> "... The rows are then referenced by the Specific Properties
>>> by using the appropriate value for the Common Properties
>>> identifier."
>>>
>>> could simply be:
>>> "... The rows are then referenced by Data Records by
>>> using the appropriate value for the Common Properties
>>> identifier."
>>
>>
>> In your example, the "Data Records" doesn't mean the full data 
>> records before applying the redundancy reduction mechanism but only a part of it.
>> So it might be confusing... That's the trick we found.
>

The Specific Properties are "what is left" of the Data Record when the Common Properties have been defined. In your example above using Data Records is wrong... Using the term Specific Properties seems clearer... (and it seemed intuitive after the term Common Properties had been used)

>>>
>>>
>>> How do you propose to withdraw common properties when they are
>>> associated with a template rather than the Common Properties index?
>>
>>
>> I don't ;) This is the reason why the use the Common Properties index is a cleaner design.
>
>
> I think withdrawal messages should work in general for options
> data, so why not send an empty record with the template as scope?

yes we should add a sentence on that. Maybe in 8.3 (CommonProertiesID vs. TemplatedID scope). Something like:

Note that, if templateIDs are used, and similarly to commonPropertiesID management, templateIDs that are not used anymore SHOULD be withdrawn. The withdrawal message is an Option Data Record consisting of only one scope field namely the templateID to be withdrawn
 
0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Set ID = 3            |      Length = 14 octets       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Template ID N           |       Field Count = 1         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Scope Field count = 1    |0|     templateID =  X         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Scope 1 Field Length = 8     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 
Figure 12: templateID Withdrawal Message

>>> General Comments
>>> ================
>>> I think, sections 3.1.2, 8.2 and 8.3 would be better grouped
>>> into a 3.2 Advanced Techniques section.
>

I am not sure I would call 8.2 and 8.3 advanced techniques... they really seem more an evaluation thing... Like recommending using reduced size encoding for commonPropertiesId whenever possible for increased data reduction(in 8.2)

>>>
>>> I think 3.2 Per Packet Data Reduction, should be removed and
>>> any information that is not in section 3.1 should be added to
>>> to the PSAMP example.
>>> I think the the per-packet example is a good example of
>>> per flow reducing redundancy and section 11.1.1 can be removed.
>>
>> See one of my previous remark about:
>>
>>  3.1    Per Flow Data Reduction...............................11
>>  3.1.1 Unique Data Reduction..................................11
>>  3.1.2 Multiple Data Reduction................................12
>>  3.2    Per Packet Data Reduction.............................13
>>
>> Again, it's because you know the boundary between IPFIX and PSAMP
>
>
> I just don't think the repetition is necessary.  Per-packet is
> just a good example of the technique, the Per Packet section
> doesn't seem to add any new information. 

see my proposal below

>
>>> 3.1.2 Multiple Data Reduction
>>>
>>> I don't think this section is explained very well. From my original
>>> email to Elisa and Lutz (18th Nov 2005):
>>> Andrew Johnson wrote:
>>>
>>>> Flows
>>>> =====
>>>> Flow A: flowId 1, flowId 11, 30 packets, 6000 bytes
>>>> Flow B: flowID 1, flowId 12, 50 packets, 9500 bytes
>>>> Flow C: flowId 2, flowId 11, 60 packets, 8000 bytes
>>>> Flow D: flowID 2, flowId 12, 10 packets, 1500 bytes
>>>>
>>>> Options
>>>> =======
>>>> flowId 1 - destinationIpv4address 10.0.0.1, destinationTcpPort 80
>>>> flowId 1 - destinationIpv4address 10.0.0.5, destinationTcpPort 80
>>>>
>>>> flowId 11 - sourceIpv4address 10.0.1.10, sourceTcpPort 1932
>>>> flowId 12 - sourceIpv4address 10.0.1.15, sourceTcpPort 1842
>>>
>>>
>>> This shows how 4 (2+2) options can be expanded to 4 combinations of
>>> Data Record to reduce redundancy without the need to define 4 complete
>>> sets of Common Properties and therefore not save any bandwidth. The
>>> more options you have the better this works. 20 options (10+10) would
>>> give you up to 100 possible pairs common properties.
>>>
>>> This isn't really made clear in the draft text, with only "flow1"
>>> having two indices.
>>
>> I agree the examples in figure 4/5 don't show this.
>> Maybe a new section in section 8 "Export and Evaluation Considerations"?
>

Agreed. But I think it should be addressed in section 3

>>> The example in 11.1.2 doesn't even have any!
>>
>> Agreed. This should be corrected.
>>
>>>
>>> While I agree that using a template that doesn't include the cpID
>>> is a good idea to efficiently communicate flows with not so common
>>> properties, however I would call this an "Advanced Technique" to be
>>> listed in the section I proposed earlier.
>

do you mean using the templateID instead of the CommonPopertiesID?

>>>
>>>
>>> 3.2 Per Packet Data Reduction
>>>
>>> The text reads:
>>> If filtering is applied to select a series of packets, using the PSAMP specifications to export per-packet information might be relatively inefficient if the filtered fields (the common attributes) are exported in every single record.
>>>
>>> However in section 6.5.2.5 (Property Match Filtering) of PSAMP-PROTO
>>> it is explained how the filter information MUST be sent. This is
>>> designed so that the selector ID will match the one from the Data
>>> Record (via the selectionSequenceId) and perform exactly the function
>>> that is being proposed here for the commonPropertiesId.
>>
>> Good point. I would remove this paragraph and move to this place the 
>> one starting with:
>>
>>  An example of the per packet data reduction is the measurement of  
>> One-Way Delay (OWD), where the exact same specific packet must be   
>> observed at the source and destination of the path to be measured.   
>> By subtracting ...
>

Remember that section 3 explains the general solution...It doesn't seem to be a good idea to have in the general specification just the description of one example application, such as OWD.
See my proposal below

>>>
>>> 4. Specifications for bandwidth saving information export
>>>
>>> What is being specified here that isn't true for ALL options as
>>> described in IPFIX-PROTO?
>>
>>
>> What do you suggest?
>
>
> Well it seems that section three really explains almost the same
> thing 3 times.  The section 4 explains most of it again (with a
> bit more info) another 3 times.
> I would prefer section 3 to simply explain the technique.  Walk
> through how the information from a Flow is split into two parts,
> and then add the bit from section 4 how these two parts are
> exported.
>
> Using more than one CPid in a record should be part of the
> advanced techniques once the basics are made clear.  Per packet
> should just be a detailed example at the end.

I propose the following change:

section three is split in just two sections: uique data reduction and multiple data reduction. In the introductory part of section three we write something like:

The PSAMP specifications are used for the export of per-packet information, exporting the specific observed packet in an IPFIX Flow Record.  This can be considered as a special Flow Record case, composed of a single packet. Therefore, the method described in this document is applicable to per packet data reduction, e.g. for export of One Way Delay (OWD) measurements (see Appendix), trajectory sampling, etc.

The text on OWD can be moved to the Appendix, where we describe in detail the OWD example.

Regards,
Elisa
**************************************************************************************************
 
 E-mail Confidentiality Notice and Disclaimer.
 
  This email and any files transmitted with it are confidential and are intended solely for 
the use of the individual or entity to which they are addressed. Access to this e-mail by 
anyone else is unauthorised. If you are not the intended recipient, any disclosure, 
copying, distribution or any action taken or omitted to be taken in reliance on it, is 
prohibited. 
  E-mail messages are not necessarily secure.  Hitachi does not accept responsibility 
for any changes made to this message after it was sent. 
  Please note that Hitachi checks outgoing e-mail messages for the presence of 
computer viruses.

**************************************************************************************************
_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 07 Dec 2006 00:04:13 +0000

> Comments, as usual, inline...

Ditto...

>  >> between flows. However, I understand your concern for novice readers.
[SNIP]
> so maybe the text proposed by Benoit helps clarifying things. I agree 

How does that help?  As I just said, "my concern wasn't really directed
at the draft itself..."  The only change to help clarify this would be
in IPFIX-INFO in the description of the commonPropertiesId.  Perhaps
to (I changed the unique within to what it should be too):
   An identifier of a set of properties that is common across more
   that one Flow.  Typically, this Information Element is used to
   link to information reported in separate records.  The identifier
   is unique per Transport Session and Observation Domain.

> The Specific Properties are "what is left" of the Data Record when the 
> Common Properties have been defined. In your example above using Data 
> Records is wrong...

Wrong?  It wouldn't be the first time, but perhaps you would like to
explain your reasoning for saying so.

A Flow is handed from the Metering Process to the Exporting Process.
The Exporting Process performs some magic, possibly including some
reducing redundancy type stuff and produces and IPFIX export stream.

>From IPFIX-PROTO:
 Data Record 
   A Data Record is a record that contains values of the parameters 
   corresponding to a Template Record...

The Data Records are something that the Exporting Process produces,
it is not right to talk about Data Records before something is
exported.

>                     Using the term Specific Properties seems clearer... 
> (and it seemed intuitive after the term Common Properties had been used)

By all means say that the Data Record that is sent contains properties
specific to the Flow.  I'm not complaining about calling them specific
properties, but I don't think there is any need to have a new definition
of "Specific Properties".

> Note that, if templateIDs are used, and similarly to commonPropertiesID 
> management, templateIDs that are not used anymore SHOULD be withdrawn. 

Erm, careful.  Properties associated with a template need to withdrawn,
not necessarily the templateID / template itself which is done with a
template withdrawal message.

> I am not sure I would call 8.2 and 8.3 advanced techniques... they 
> really seem more an evaluation thing... Like recommending using reduced 
> size encoding for commonPropertiesId whenever possible for increased 
> data reduction(in 8.2)

The idea is that you want one section that simply explains the system.
Once the reader understands the basics you can then add things like
using RLE for the IDs.  It's "advanced" because it's not essential,
it's not needed in the most basic of systems.

> Agreed. But I think it should be addressed in section 3

I think section 3 should concentrate on explaining the most basic system
from start to end.  Once the user is clear about that then you can add
complications, such as this.

> do you mean using the templateID instead of the CommonPopertiesID?

No.  I'm talking about sending Data Records that contain all the
properties from the original Flow when none of its properties are
shared between Flows.  Since none of this particular Flows properties
are common, then sending it in full is more efficient that sending it
with a cpID.

>  >>> If filtering is applied to select a series of packets, using the 
>  >>> PSAMP specifications to export per-packet information might be 
>  >>> relatively inefficient if the filtered fields (the common attributes) 
>  >>> are exported in every single record.
>  >>>
> Remember that section 3 explains the general solution...It doesn't seem 
> to be a good idea to have in the general specification just the 
> description of one example application, such as OWD.
> See my proposal below
> 
>  >>>

Also, I'd like to reduce repetition from the protocol draft.  You
shouldn't explain how options work here, just reference the relevant
section of [IPFIX-PROTO].

> section three is split in just two sections: uique data reduction and 
> multiple data reduction. In the introductory part of section three we 
> write something like:

You don't need to repeat how this works for "multiple data reduction".
It works in the same way, but you have 2 cpIDs.  Get the basics clear
first and then expand on it later.

> The PSAMP specifications are used for the export of per-packet 
> information, exporting the specific observed packet in an IPFIX Flow 
> Record.  This can be considered as a special Flow Record case, composed 
> of a single packet. Therefore, the method described in this document is 
> applicable to per packet data reduction, e.g. for export of One Way 
> Delay (OWD) measurements (see Appendix), trajectory sampling, etc.
> 

Moving the per-packet stuff to a detailed example in the Appendix
sounds like a good idea to me.

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 07 Dec 2006 00:04:13 +0000

Ditto...

[SNIP]

 Data Record 

Andrew