Subject: [IPFIX] Problems with Flow Collection in large-scale networks: feedback

From: Benoit Claise <[email-masked]>
Date: Sun, 09 Mar 2008 22:44:22 -0400

Kobayashi-san,

I think the draft improved compared to last time.

Here is my feedback.

- In the terminology section, you're missing a reference to the 
terminology from RFC5101, since you capitalized some of the terms.
-

         The
         second is changing the Template and record structure by adding/
         deleting specified Information Elements.  Example devices for
         this type are the IPFIX Masquerading Proxy and the IPFIX
         Concentrator.
NEW
	The
         second is changing the Template and record structure by adding/
         aggregating/deleting specified Information Elements.  Example devices for
         this type are the IPFIX Masquerading Proxy and the IPFIX
         Concentrator.

- Section 3.1. You mentioned sampling rate of 1/1000 to 1/1000. I've 
seen 1/100
- Section 3.1. You're missing an important application -> data retention.
- Section 4. twice the same idea in the following sentences

   It should be noted that the
   current sampling rate might become infeasible for Exporters within
   the next five years.  We expect to encounter this problem within the
   next five years in large-scale networks.

- I would develop further the section 4.2 to express that there is a 
different approach than exporting complete flow records to a mediation 
device, which in turn does some aggregation/filtering. I mean, the 
router have several different caches, each optimized for a specific 
application. So each cache has its own series of flow keys. And the 
content of each cache is sent to a specific collector. For example, a 
collector for security, another one for capacity planning, etc...
- Section 4.4 "In general, load balancing allows work of Collectors to 
be divided. Classifying Flow Records based on the value of specified 
Information Elements can prove to be very useful to achive scalability. 
" This is not load balancing, as the title says.
- I would add a new section 4.5 "Flow based sampling"
- section 5.1 "Implementing load balancing"

   "There currently is no description of a classification function in
   IPFIX."

This is normal as IPFIX is the export protocol. You speak about a 
specific feature of the Exporting Process that would use RFC5101

   "According to the current specification, Exporters send all
   Flow Records to multiple Collectors and Collectors drop uninteresting
   Flow Records to reduce their load."

This is wrong.
-

   "The Collector, however, needs to recognize the
   precise Observation Point whether Flow Records go through an IPFIX
   Mediator or not."

This is not always true. The answer is: it depends.
For example, for the core traffic matrix in capacity planning, having 
the router exporting the flow records is enough.
- section 5.6 "session management"
You want to talk about transport session, right? (to refer to RFC5101)
This session is confusing. After reading several times, I understand 
that you speak about the Collecting and Exporting Processes of the 
mediation device.
- section 5.6

   "If the session
   between a Router and a Mediator fails and the IPFIX Mediator shuts
   down the session between the Collector and the Mediator, the
   Collector would futilely continue to try to establish the session."

   "If
   one of these sessions between Routers and the Mediator fails and
   IPFIX Mediator shuts down the session between a Collector and the
   Mediator, other sessions between the Routers and the Mediator would
   not be relayed at all."

Not necessarily. It depends how we write the specifications.
- Section 5.6, last paragraph.

   "Even if one session is reset,
   the status of the other sessions is kept.  However, if Templates that
   reset the session of Collecting Processes would not be withdrawn,
   there is a possibility that assigned Template IDs overlap.  The
   Collector might then shut down IPFIX session, because it identifies
   the overlapping Template ID as part of an anomalous message.  In the
   case of resetting a session of the Collecting Process, the operation
   processed by IPFIX Mediator should be defined."

This is wrong IMHO. The mediation device should take care of the management of NEW templates. We can't rely on the template id received.

Regards, Benoit.

_______________________________________________
IPFIX mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ipfix

---

From: Kobayashi Atsushi <[email-masked]>
Date: Mon, 10 Mar 2008 18:09:39 +0900

Hi Benoit,

Thank you very much for your feedback.
Please see in-line.

On Sun, 09 Mar 2008 22:44:22 -0400

> terminology from RFC5101, since you capitalized some of the terms.

Yes. I missed it. I will improve in next version.

>          this type are the IPFIX Masquerading Proxy and the IPFIX

Yes.

> - Section 3.1. You mentioned sampling rate of 1/1000 to 1/1000. I've 

Where is it?
I mentFrom [email-masked]  Mon Mar 10 08:02:16 2008
Date: Mon, 10 Mar 2008 18:09:39 +0900
From: Kobayashi Atsushi <[email-masked]>
To: "[email-masked]" <[email-masked]>
Subject: Re: [IPFIX] Problems with Flow Collection in large-scale networks:
	feedback
Sender: [email-masked]

Hi Benoit,

Thank you very much for your feedback.
Please see in-line.

On Sun, 09 Mar 2008 22:44:22 -0400

> terminology from RFC5101, since you capitalized some of the terms.

Yes. I missed it. I will improve in next version.

>          this type are the IPFIX Masquerading Proxy and the IPFIX

Yes.

> - Section 3.1. You mentioned sampling rate of 1/1000 to 1/1000. I've 

Where is it?
I mentioned that "typically sampling packets with rates ranging from
1/10,000 to 1/1,000". 

> - Section 3.1. You're missing an important application -> data retention.

You mean that data retention is storage of traffic data during reasonable
amount of time according to government regulation?
I was not familiar with data retention using NetFlow/sFlow. Exactly,
NetFlow/sFlow/IPFIX can become the tool for data retention.

>    next five years in large-scale networks.

Yes. Latter sentence can be removed.

> collector for security, another one for capacity planning, etc...

I can image you mentioned.
Each cache has its own Flow Keys. The output of each cache is send to a
specific collector.

> " This is not load balancing, as the title says.

Does it prefer "load sharing method" or "load distribution"?

> - I would add a new section 4.5 "Flow based sampling"

OK.

>    Flow Records to reduce their load."

Wrong? If so, I can not image the current specification regarding
classification.

As another case, observed packets become input to multiple Metering
Processes, and in each Metering Process, these interesting packets are
selected. Finally, output of each Metering Process is sent to a specific
Collector. Is it correct?

> This is not always true. The answer is: it depends.

Yes. Not alway. For example, in creating traffic matrix using exporter IP
address it is necessary. I will improve.

> You want to talk about transport session, right? (to refer to RFC5101)

Yes. It means "Transport Session".

> that you speak about the Collecting and Exporting Processes of the 

Yes. I will improve.

> Not necessarily. It depends how we write the specifications.

You mean that removing all of section 5.6 is better?

>    the status of the other sessions is kioned that "typically sampling packets with rates ranging from
1/10,000 to 1/1,000". 

> - Section 3.1. You're missing an important application -> data retention.

You mean that data retention is storage of traffic data during reasonable
amount of time according to government regulation?
I was not familiar with data retention using NetFlow/sFlow. Exactly,
NetFlow/sFlow/IPFIX can become the tool for data retention.

>    next five years in large-scale networks.

Yes. Latter sentence can be removed.

> collector for security, another one for capacity planning, etc...

I can image you mentioned.
Each cache has its own Flow Keys. The output of each cache is send to a
specific collector.

> " This is not load balancing, as the title says.

Does it prefer "load sharing method" or "load distribution"?

> - I would add a new section 4.5 "Flow based sampling"

OK.

>    Flow Records to reduce their load."

Wrong? If so, I can not image the current specification regarding
classification.

As another case, observed packets become input to multiple Metering
Processes, and in each Metering Process, these interesting packets are
selected. Finally, output of each Metering Process is sent to a specific
Collector. Is it correct?

> This is not always true. The answer is: it depends.

Yes. Not alway. For example, in creating traffic matrix using exporter IP
address it is necessary. I will improve.

> You want to talk about transport session, right? (to refer to RFC5101)

Yes. It means "Transport Session".

> that you speak about the Collecting and Exporting Processes of the 

Yes. I will improve.

> Not necessarily. It depends how we write the specifications.

You mean that removing all of section 5.6 is better?

> This is wrong IMHO. The mediation device should take care of the management of NEW templates. We can't rely on the template id received.

I think that it is not necessary to rewrite Template ID in case of
relaying from one Transport Session to one Transport Session.

But, exactly, assigning new template id in case of all situation is
safer for IPFIX protocol.

You mean that IPFIX Mediator implemented in relation to RFC5101 does not
fall into the situation mentioned in section 5.6? After all, section 5.6
is not necessary, is it? It seems deletable.

Regards,
Atsushi KOBAYASHI

ept.  However, if Templates that
> This is wrong IMHO. The mediation device should take care of the management of NEW templates. We can't rely on the template id received.

I think that it is not necessary to rewrite Template ID in case of
relaying from one Transport Session to one Transport Session.

But, exactly, assigning new template id in case of all situation is
safer for IPFIX protocol.

You mean that IPFIX Mediator implemented in relation to RFC5101 does not
fall into the situation mentioned in section 5.6? After all, section 5.6
is not necessary, is it? It seems deletable.

Regards,
Atsushi KOBAYASHI

---

From: Benoit Claise <[email-masked]>
Date: Tue, 11 Mar 2008 12:03:11 -0400

Kobayashi-san,

> I mentioned that "typically sampling packets with rates ranging from
I mean that I have seen customers with 1/100.
> NetFlow/sFlow/IPFIX can become the tool for data retention.
Data retention is now a law at the European level.
And obviously, NetFlow/IPFIX is a master piece in the architecture.
> Each cache has its own Flow Keys. The output of each cache is send to a
Exactly.
> Does it prefer "load sharing method" or "load distribution"?
Maybe export filtering?

> Wrong? If so, I can not image the current specification regarding
I mean that IPFIX, being an export protocol, doesn't impose any 
constraints on the metering process and on what you export to collectors.
It doesn't impose that you export ALL flow records to collector(s), and 
then do filtering.
Hence, my example in this email thread: multiple caches, each with its 
own set of flow keys, exported to different collectors. Even if you add 
filtering before exporting, you can still use IPFIX.
> You mean that removing all of section 5.6 is better?
no, I mean that your statements assume certain 
specifications/implementations at the mediation device.
> relaying from one Transport Session to one Transport Session.
Correct, but that's the only case.
And if you have a single Transport Session matching to another Transport 
Session, I'm questioning the need for a mediation device ;-)
> is not necessary, is it? It seems deletable.
I think that you should leave it in there, but that you have to explain 
the potential problems, without assuming a specific implementation.

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Tue, 11 Mar 2008 12:03:11 -0400

Kobayashi-san,

Exactly.
Maybe export filtering?

Regards, Benoit.

---

From: Kobayashi Atsushi <[email-masked]>
Date: Wed, 12 Mar 2008 03:01:03 +0900

Hi Benoit,

Please see in-line.

On Tue, 11 Mar 2008 12:03:11 -0400

> I mean that I have seen customers with 1/100.

I focus on large-scale networks. I have not seen tier1 level provider
with 1/100. Is it true? It must be huge Flow Records. My simple question.

> And obviously, NetFlow/IPFIFrom [email-masked]  Tue Mar 11 11:03:37 2008
Date: Wed, 12 Mar 2008 03:01:03 +0900
From: Kobayashi Atsushi <[email-masked]>
To: Benoit Claise <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [IPFIX] Problems with Flow Collection in large-scale networks:
	feedback
Sender: [email-masked]

Hi Benoit,

Please see in-line.

On Tue, 11 Mar 2008 12:03:11 -0400

> I mean that I have seen customers with 1/100.

I focus on large-scale networks. I have not seen tier1 level provider
with 1/100. Is it true? It must be huge Flow Records. My simple question.

> > Does it prefer "load sharing method" or "load distribution"?

Maybe it is so.

> It doesn't impose that you export ALL flow records to collector(s), and 

Yes. It is one solution of using feature of IPFIX. I don't mean to
impose it. I will improve the description of this section.

> filtering before exporting, you can still use IPFIX.

Yes. Your idea using multiple caches can be applied to Metering Process
in Mediator, too.

> specifications/implementations at the mediation device.

OK. I understand you mean. I should not assume some specification.

> >>    case of resetting a session of the CollecX is a master piece in the architecture.
> >   

Maybe it is so.

> It doesn't impose that you export ALL flow records to collector(s), and 

Yes. It is one solution of using feature of IPFIX. I don't mean to
impose it. I will improve the description of this section.

> filtering before exporting, you can still use IPFIX.

Yes. Your idea using multiple caches can be applied to Metering Process
in Mediator, too.

> specifications/implementations at the mediation device.

OK. I understand you mean. I should not assume some specification.

> Session, I'm questioning the need for a mediation device ;-)

In following case of using Masquerading Proxy , we can image it.

       .--------------.
IPFIX  |              | IPFIX
-----> | -anonymize-> | ------->
IPFIX  |              | IPFIX
-----> | -anonymize-> | ------->
IPFIX  |              | IPFIX
-----> | -anonymize-> | ------->
       |              |
       '--------------'

Regards,
Atsushi KOABYASHI

ting Process, the operation
> Session, I'm questioning the need for a mediation device ;-)

In following case of using Masquerading Proxy , we can image it.

       .--------------.
IPFIX  |              | IPFIX
-----> | -anonymize-> | ------->
IPFIX  |              | IPFIX
-----> | -anonymize-> | ------->
IPFIX  |              | IPFIX
-----> | -anonymize-> | ------->
       |              |
       '--------------'

Regards,
Atsushi KOABYASHI