Subject: Re: [IPFIX] Problems with Flow Collection in large-scale networks: feedback

From: Haruhiko Nishida <[email-masked]>
Date: Mon, 10 Mar 2008 11:28:52 -0400

comments inline.

-- Haru

On 2008/03/10, at 10:42, Kobayashi Atsushi wrote:

>
> Forwarded by Kobayashi Atsushi <[email-masked]>
> ----------------------- Original Message -----------------------
> From:    Kobayashi Atsushi <[email-masked]>
> To:      "[email-masked]" <[email-masked]>
> Date:    Mon, 10 Mar 2008 18:09:39 +0900
> Subject: Re: [IPFIX] Problems with Flow Collection in large-scale  
> networks: feedback
> ----
>
>
> Hi Benoit,
>
> Thank you very much for your feedback.
> Please see in-line.
>
> On Sun, 09 Mar 2008 22:44:22 -0400
> Benoit Claise <[email-masked]> wrote:
>
>> Kobayashi-san,
>>
>> I think the draft improved compared to last time.
>>
>> Here is my feedback.
>>
>> - In the terminology section, you're missing a reference to the
>> terminology from RFC5101, since you capitalized some of the terms.
>
> Yes. I missed it. I will improve in next version.
>
>> -
>>
>>         The
>>         second is changing the Template and record structure by  
>> adding/
>>         deleting specified Information Elements.  Example devices for
>>         this type are the IPFIX Masquerading Proxy and the IPFIX
>>From [email-masked]  Mon Mar 10 09:18:20 2008
From: Haruhiko Nishida <[email-masked]>
To: [email-masked]
Date: Mon, 10 Mar 2008 11:28:52 -0400
Subject: Re: [IPFIX] Problems with Flow Collection in large-scale networks:
	feedback
Sender: [email-masked]

comments inline.

-- Haru

On 2008/03/10, at 10:42, Kobayashi Atsushi wrote:

>
> Forwarded by Kobayashi Atsushi <[email-masked]>
> ----------------------- Original Message -----------------------
> From:    Kobayashi Atsushi <[email-masked]>
> To:      "[email-masked]" <[email-masked]>
> Date:    Mon, 10 Mar 2008 18:09:39 +0900
> Subject: Re: [IPFIX] Problems with Flow Collection in large-scale  
> networks: feedback
> ----
>
>
> Hi Benoit,
>
> Thank you very much for your feedback.
> Please see in-line.
>
> On Sun, 09 Mar 2008 22:44:22 -0400
> Benoit Claise <[email-masked]> wrote:
>
>> Kobayashi-san,
>>
>> I think the draft improved compared to last time.
>>
>> Here is my feedback.
>>
>> - In the terminology section, you're missing a reference to the
>> terminology from RFC5101, since you capitalized some of the terms.
>
> Yes. I missed it. I will improve in next version.
>
>> -
>>
>>         The
>>         second is changing the Template and record structure by  
>> adding/
>>         deleting specified Information Elements.  Example devices for
>>         this type are the IPFIX Masquerading Proxy and the IPFIX
>>         Concentrator.
>> NEW
>> 	The
>>         second is changing the Template and record structure by  
>> adding/
>>         aggregating/deleting specified Information Elements.   
>> Example devices for
>>         this type are the IPFIX Masquerading Proxy and the IPFIX
>>         Concentrator.
>>
>
> Yes.
>
>> - Section 3.1. You mentioned sampling rate of 1/1000 to 1/1000. I've
>> seen 1/100
>
> Where is it?
> I mentioned that "typically sampling packets with rates ranging from
> 1/10,000 to 1/1,000".

I thought Benoit knows some use cases using 1/100. If so, it's good  
news. Benoit, will you let us know some more on the case?

(As a draft, we can modify 1/10,000-1/1,000 to 1/10,000-1/100 easily)

>> - Section 3.1. You're missing an important application -> data  
>> retention.
>
> You mean that data retention is storage of traffic data during  
> reasonable
> amount of time according to government regulation?
> I was not familiar with data retention using NetFlow/sFlow. Exactly,
> NetFlow/sFlow/IPFIX can become the tool for data retention.

neither of me nor Kobayashi-san knows about data retention by netflow/ 
IPFIX. Could you add some description on the part?

>> - Section 4. twice the same idea in the following sentences
>>
>>   It should be noted that the
>>   current sampling rate might become infeasible for Exporters within
>>   the next five years.  We expect to encounter this problem within  
>> the
>>   next five years in large-scale networks.
>>
>
> Yes. Latter sentence can be removed.
>
>> - I would develop further the section 4.2 to express that there is a
>> different approach than exporting complete flow records to a  
>> mediation
>> device, which in turn does some aggregation/filtering. I mean, the
>> router have several different caches, each optimized for a specific
>> application. So each cache has its own series of flow keys. And the
>> content of each cache is sent to a specific collector. For example, a
>> collector for security, another one for capacity planning, etc...
>
> I can image you mentioned.
> Each cache has its own Flow Keys. The output of each cache is send  
> to a
> specific collector.
>
>> - Section 4.4 "In general, load balancing allows work of Collectors  
>> to
>> be divided. Classifying Flow Records based on the value of specified
>> Information Elements can prove to be very useful to achive  
>> scalability.
>> " This is not load balancing, as the title says.
>
> Does it prefer "load sharing method" or "load distribution"?

I feel both "load sharing/balancing" wordings are wrong, since in this  
example collectors have different purposes... (but couldn't find good  
word yet)

>> - I would add a new section 4.5 "Flow based sampling"
>
> OK.
>
>> - section 5.1 "Implementing load balancing"
>>
>>   "There currently is no description of a classification function in
>>   IPFIX."
>>
>> This is normal as IPFIX is the export protocol. You speak about a
>> specific feature of the Exporting Process that would use RFC5101
>>
>>   "According to the current specification, Exporters send all
>>   Flow Records to multiple Collectors and Collectors drop  
>> uninteresting
>>   Flow Records to reduce their load."
>>
>> This is wrong.
>
> Wrong? If so, I can not image the current specification regarding
> classification.
>
> As another case, observed packets become input to multiple Metering
> Processes, and in each Metering Process, these interesting packets are
> selected. Finally, output of each Metering Process is sent to a  
> specific
> Collector. Is it correct?

Purpose of section 5.1 is to say "we need classification function". I  
believes that we all shares the purpose. Then, we need just rewrite  
the section.

>>   "The Collector, however, needs to recognize the
>>   precise Observation Point whether Flow Records go through an IPFIX
>>   Mediator or not."
>>
>> This is not always true. The answer is: it depends.
>
> Yes. Not alway. For example, in creating traffic matrix using  
> exporter IP
> address it is necessary. I will improve.
>
>
>
>> For example, for the core traffic         Concentrator.
>> NEW
>> 	The
>>         second is changing the Template and record structure by  
>> adding/
>>         aggregating/deleting specified Information Elements.   
>> Example devices for
>>         this type are the IPFIX Masquerading Proxy and the IPFIX
>>         Concentrator.
>>
>
> Yes.
>
>> - Section 3.1. You mentioned sampling rate of 1/1000 to 1/1000. I've
>> seen 1/100
>
> Where is it?
> I mentioned that "typically sampling packets with rates ranging from
> 1/10,000 to 1/1,000".

I thought Benoit knows some use cases using 1/100. If so, it's good  
news. Benoit, will you let us know some more on the case?

(As a draft, we can modify 1/10,000-1/1,000 to 1/10,000-1/100 easily)

>> - Section 3.1. You're missing an important application -> data  
>> retention.
>
> You mean that data retention is storage of traffic data during  
> reasonable
> amount of time according to government regulation?
> I was not familiar with data retention using NetFlow/sFlow. Exactly,
> NetFlow/sFlow/IPFIX can become the tool for data retention.

neither of me nor Kobayashi-san knows about data retention by netflow/ 
IPFIX. Could you add some description on the part?

>> - Section 4. twice the same idea in the following sentences
>>
>>   It should be noted that the
>>   current sampling rate might become infeasible for Exporters within
>>   the next five years.  We expect to encounter this problem within  
>> the
>>   next five years in large-scale networks.
>>
>
> Yes. Latter sentence can be removed.
>
>> - I would develop further the section 4.2 to express that there is a
>> different approach than exporting complete flow records to a  
>> mediation
>> device, which in turn does some aggregation/filtering. I mean, the
>> router have several different caches, each optimized for a specific
>> application. So each cache has its own series of flow keys. And the
>> content of each cache is sent to a specific collector. For example, a
>> collector for security, another one for capacity planning, etc...
>
> I can image you mentioned.
> Each cache has its own Flow Keys. The output of each cache is send  
> to a
> specific collector.
>
>> - Section 4.4 "In general, load balancing allows work of Collectors  
>> to
>> be divided. Classifying Flow Records based on the value of specified
>> Information Elements can prove to be very useful to achive  
>> scalability.
>> " This is not load balancing, as the title says.
>
> Does it prefer "load sharing method" or "load distribution"?

I feel both "load sharing/balancing" wordings are wrong, since in this  
example collectors have different purposes... (but couldn't find good  
word yet)

>> - I would add a new section 4.5 "Flow based sampling"
>
> OK.
>
>> - section 5.1 "Implementing load balancing"
>>
>>   "There currently is no description of a classification function in
>>   IPFIX."
>>
>> This is normal as IPFIX is the export protocol. You speak about a
>> specific feature of the Exporting Process that would use RFC5101
>>
>>   "According to the current specification, Exporters send all
>>   Flow Records to multiple Collectors and Collectors drop  
>> uninteresting
>>   Flow Records to reduce their load."
>>
>> This is wrong.
>
> Wrong? If so, I can not image the current specification regarding
> classification.
>
> As another case, observed packets become input to multiple Metering
> Processes, and in each Metering Process, these interesting packets are
> selected. Finally, output of each Metering Process is sent to a  
> specific
> Collector. Is it correct?

Purpose of section 5.1 is to say "we need classification function". I  
believes that we all shares the purpose. Then, we need just rewrite  
the section.

>>   "The Collector, however, needs to recognize the
>>   precise Observation Point whether Flow Records go through an IPFIX
>>   Mediator or not."
>>
>> This is not always true. The answer is: it depends.
>
> Yes. Not alway. For example, in creating traffic matrix using  
> exporter IP
> address it is necessary. I will improve.
>
>
>
>> For example, for the core traffic matrix in capacity planning, having
>> the router exporting the flow records is enough.
>
>> - section 5.6 "session management"
>> You want to talk about transport session, right? (to refer to  
>> RFC5101)
>
> Yes. It means "Transport Session".
>
>> This session is confusing. After reading several times, I understand
>> that you speak about the Collecting and Exporting Processes of the
>> mediation device.
>
> Yes. I will improve.
>
>> - section 5.6
>>
>>   "If the session
>>   between a Router and a Mediator fails and the IPFIX Mediator shuts
>>   down the session between the Collector and the Mediator, the
>>   Collector would futilely continue to try to establish the session."
>>
>>   "If
>>   one of these sessions between Routers and the Mediator fails and
>>   IPFIX Mediator shuts down the session between a Collector and the
>>   Mediator, other sessions between the Routers and the Mediator would
>>   not be relayed at all."
>>
>> Not necessarily. It depends how we write the specifications.
>
> You mean that removing all of section 5.6 is better?
>
>> - Section 5.6, last paragraph.
>>
>>   "Even if one session is reset,
>>   the status of the other sessions is kept.  However, if Templates  
>> that
>>   reset the session of Collecting Processes would not be withdrawn,
>>   there is a possibility that assigned Template IDs overlap.  The
>>   Collector might then shut down IPFIX session, because it identifies
>>   the overlapping Template ID as part of an anomalous message.  In  
>> the
>>   case of resetting a session of the Collecting Process, the  
>> operation
>>   processed by IPFIX Mediator should be defined."
>>
>> This is wrong IMHO. The mediation device should take care of the  
>> management of NEW templates. We can't rely on the template id  
>> received.
>
> I think that it is not necessary to rewrite Template ID in case of
> relaying from one Transport Session to one Transport Session.
>
> But, exactly, assigning new template id in case of all situation is
> safer for IPFIX protocol.
>
> You mean that IPFIX Mediator implemented in relation to RFC5101 does  
> not
> fall into the situation mentioned in section 5.6? After all, section  
> 5.6
> is not necessary, is it? It seems deletable.

Purpose of section 5.6 is to say "some kind of session management is  
needed". Since the draft is problem statements, I agree that current  
draft description is too detailed and should be categorized  
requirement or specification.

>
>
> Regards,
> Atsushi KOBAYASHI
>
>>
>> Regards, Benoit.
>>
>>
>>
>>
>>
>> _______________________________________________
>> IPFIX mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/ipfix
>
> --------------------- Original Message Ends --------------------

_______________________________________________
IPFIX mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ipfix

 matrix in capacity planning, having
>> the router exporting the flow records is enough.
>
>> - section 5.6 "session management"
>> You want to talk about transport session, right? (to refer to  
>> RFC5101)
>
> Yes. It means "Transport Session".
>
>> This session is confusing. After reading several times, I understand
>> that you speak about the Collecting and Exporting Processes of the
>> mediation device.
>
> Yes. I will improve.
>
>> - section 5.6
>>
>>   "If the session
>>   between a Router and a Mediator fails and the IPFIX Mediator shuts
>>   down the session between the Collector and the Mediator, the
>>   Collector would futilely continue to try to establish the session."
>>
>>   "If
>>   one of these sessions between Routers and the Mediator fails and
>>   IPFIX Mediator shuts down the session between a Collector and the
>>   Mediator, other sessions between the Routers and the Mediator would
>>   not be relayed at all."
>>
>> Not necessarily. It depends how we write the specifications.
>
> You mean that removing all of section 5.6 is better?
>
>> - Section 5.6, last paragraph.
>>
>>   "Even if one session is reset,
>>   the status of the other sessions is kept.  However, if Templates  
>> that
>>   reset the session of Collecting Processes would not be withdrawn,
>>   there is a possibility that assigned Template IDs overlap.  The
>>   Collector might then shut down IPFIX session, because it identifies
>>   the overlapping Template ID as part of an anomalous message.  In  
>> the
>>   case of resetting a session of the Collecting Process, the  
>> operation
>>   processed by IPFIX Mediator should be defined."
>>
>> This is wrong IMHO. The mediation device should take care of the  
>> management of NEW templates. We can't rely on the template id  
>> received.
>
> I think that it is not necessary to rewrite Template ID in case of
> relaying from one Transport Session to one Transport Session.
>
> But, exactly, assigning new template id in case of all situation is
> safer for IPFIX protocol.
>
> You mean that IPFIX Mediator implemented in relation to RFC5101 does  
> not
> fall into the situation mentioned in section 5.6? After all, section  
> 5.6
> is not necessary, is it? It seems deletable.

Purpose of section 5.6 is to say "some kind of session management is  
needed". Since the draft is problem statements, I agree that current  
draft description is too detailed and should be categorized  
requirement or specification.

>
>
> Regards,
> Atsushi KOBAYASHI
>
>>
>> Regards, Benoit.
>>
>>
>>
>>
>>
>> _______________________________________________
>> IPFIX mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/ipfix
>
> --------------------- Original Message Ends --------------------

_______________________________________________
IPFIX mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ipfix

---

From: Benoit Claise <[email-masked]>
Date: Tue, 11 Mar 2008 12:04:54 -0400

Haruhiko-san,
> IPFIX. Could you add some description on the part?
Ok, I'll write some text later.
> requirement or specification.
Exactly.

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Tue, 11 Mar 2008 12:04:54 -0400

Haruhiko-san,
Exactly.

Regards, Benoit.