Subject: [IPFIX] *Proposed* new work items, take two

From: Nevil Brownlee <[email-masked]>
Date: Thu, 26 Jul 2007 10:08:16 +1200

Hi again all:

Here's a slightly improved, slightly clearer, version of my earlier posting.

Cheers, Nevil

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

IPFIX: Proposal for new work
       Based on discussion at IETF 69, Chicago

For IPFIX mailing list, we suggest that items 1-6
(above the --- line) be adopted as new WG work items.
Please comment by 6 Aug 07 <<<<
Also, we need reviewers; please fill in and email back the
questionnaire form below!!

0. Remove SCTP stream restrictions in Protocol Document.
   Agreed: WG needs to do this, as soon as possible.
   Procedure:
    - Call the protocol draft out of the RFC EDitor Queue
    - Make changes (only those needed for SCTP Stream use),
      issue a revised draft
    - Run a short (1 week) WG last Call
    - Run a short (2 weeks, it's a Standards Track draft) IETF Last Call
    - Send draft back to RFC for evalkuatio
    - IESG sends it back to the RFC Editor Queue
   Assuming that all goes well, we estimate all that will take
   6~8 weeks.  By then we expect the PSAMP Info draft to have
   progressed so that it will be in the RFC Editor Queue too!
    - Corresponding changes to Implementation Guidelines and
      Testing drafts will also be needed, they will be made
      in the next revisions of both these drafts

1. Configuration Data Model (Gerhard Muenz)
   Agreed: Important to WG, not clear what protocol would be needed.
   Proposal: (a) Develop Configuration Info Model as a WG item,
       in English (and also an XML version) - Standards Track RFC.
       (b) Consider what protocol would be best for IPFIX Configuration.
           Requirements RFC?      Milestones: WG LC before IETF 71

2. SCTP Per-Stream Draft (Benoit Claise)
   Agreed: Needs to be explored, as a possible protocol option.
   Procedure: Make this a WG item, as Experimental RFC.
   Milestones:  WG LC after IETF 70, to IESG before IETF  71.

3. File Format (Brian Trammel)
   Agreed: WG item, as soon as possible.
   Procedure: Make this a WG item, as an Informational RFC.
   Milestones:  WG LC after IETF 70, to IESG before IETF  71.

4. Extended Types (Elisa Boschi)
   Needed for IPFIX files, most useful for Enterprise IEs; need to
   specify what operations make sense for an IE.
   Support shown for this as a WG item, as for File Format.
   Milestones:  WG LC after IETF 70, to IESG before IETF  71.

For items 0 to 4 above, we'd like to make a list of several committed
reviewers for each. Please help me with this by filling in and 
returning the form below

IPFIX work item Reviewers
-------------------------

Name: _____________________________

I will review documents for the following items:
(put a 'y' in the boxes on the next line)

0 [ ],  1 [ ],  2 [ ],  3 [ ],  4 [ ]

Email this form to [email-masked] -- thankyou !

-----------------------------------------------------------------------

5. IPFIX Mediation (Atayushi Kobayashi)
   Needed by (at least) two large operators.
   Could be integrated with Falko Dressler's draft on IPFIX Aggregation.
   Possible future WG item, continue discussion on mailing list.

6. Order of Information Elements (Irino)
   Interesting, need more people to measure possible efficiency gain.
   Would this be better as an individual draft?

7. Flow Sampling (Tanja Zseby)
   Early stage so far, comments/participation encouraged.

=======================================================================

-----------------------------------------------------------------------
  Nevil Brownlee                    Computer Science Department | ITS
  Phone: +64 9 373 7599 x88941             The University of Auckland
  FAX: +64 9 373 7453   Private Bag 92019, Auckland 1142, New Zealand

-----------------------------------------------------------------------
This mail sent through University of Auckland http://www.auckland.ac.nz

_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: "STEPHAN Emile RD-CORE-LAN" <[email-masked]>
Date: Thu, 26 Jul 2007 01:26:06 +0200

Hi Nevil,

Your choice might consider that items 3 and 4 don't received support any operators and that item 5 did.

Items 3, 4 and 5 provide good examples of mediator functions. So, in my opinion, the first step consists in writing a doc giving the requirements and the arch of a mediator.

Regards
Emile   

> -----Message d'origine-----
> De : Nevil Brownlee [mailto:[email-masked]]
> Envoyé : jeudi 26 juillet 2007 00:08
> À : [email-masked]
> Objet : [IPFIX] *Proposed* new work items, take two
> 
> 
> Hi again all:
>

---

From: "STEPHAN Emile RD-CORE-LAN" <[email-masked]>
Date: Thu, 26 Jul 2007 01:26:06 +0200

Hi Nevil,

Regards
Emile

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 26 Jul 2007 07:56:01 +0200

Dear all,

I got questions from several WG members on the status of this list
of candidate work items for an updated IPFIX charter.

According to the discussion we had in our session on Tuesday,
the list of work items below is not final, but a suggestion that
should be discussed on this mailing list.

For item
0. Remove SCTP stream restrictions in Protocol Document
we agreed to approve the consensus we had in the session on the
mailing list.

For items
1. Configuration Data Model (Gerhard Muenz)
2. SCTP Per-Stream Draft (Benoit Claise)
3. File Format (Brian Trammel)
there was no doubt expressed that these would be
very good work items for IPFIX.

For items
4. Extended Types (Elisa Boschi)
5. IPFIX Mediation (Atayushi Kobayashi)
6. Order of Information Elements (Irino)
discussions were controversial and time was limited
So, we agreed to discuss them further on the mailing list.
I will send an individual mail for triggering discussion on
each of them in separate emails.

For any of these items: Please have a look at the related drafts
and tell us what you think about them.

Thanks,

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 26 Jul 2007 07:56:01 +0200

Dear all,

For item

For items

For items

Thanks,

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 26 Jul 2007 08:27:47 +0200

Dear all,

Candidates #4 to become a new IPFIX work item is described by
draft draft-boschi-ipfix-extended-type-00.txt.

>    Milestones:  WG LC after IETF 70, to IESG before IETF  71.

It is a method for assigning data types to unknown IPFIX information
elements by using option records that map data types to element identifiers.

It can be applied in cases where an IPFIX collector does not know the
data type of received information elements. This may in particular happen
if enterprise specific elements are used. Without this information the
collector would treat these elements as if they were of type octet array.
With this information it may be able to treat them more appropriately.

The current version of the IPFIX file format draft recommends using this
method when IPFIX records are stored.

Do you think this is a useful extension of the IPFIX protocol?
Do you think we should accept this as an IPFIX work item?
Please have a look at the draft and send your comments.

Thanks,

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 26 Jul 2007 08:27:47 +0200

Dear all,

Thanks,

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 26 Jul 2007 08:38:36 +0200

Dear all,

Candidates #6 to become a new IPFIX work item is described by
draft draft-irino-ipfix-ie-order-02.txt.

>    Would this be better as an individual draft?

This draft suggests a special ordering of information elements in a
flow record.  At the very brief presentation at our session, the author
explained that he can improve the performance of IPFIX exporter and
collector significantly. To prove this, he compared a random order
of information elements with an order according to his proposal
and achieved significant performance improvements.

Speaking as technical contributor, I think that if these results would
be the same also for other implementations of IPFIX, then this looks like
a very good IPFIX work item.

Speaking as co-chair, I would like to ask everybody who has an IPFIX
implementation to conduct a short experiment in order to find out if
the improvement is the same for his/her implementations.

Would any implementor be willing to do so?
If yes, until when would you have results?

Thanks,

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 26 Jul 2007 08:38:36 +0200

Dear all,

Thanks,

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 26 Jul 2007 08:53:05 +0200

Dear all,

Candidate #5 to become a new IPFIX work item is described by
draft-kobayashi-ipfix-mediator-model-00.txt.

>    Possible future WG item, continue discussion on mailing list.

This draft describes how IPFIX flow records from a large number of exporters
can be aggregated by one or more a mediator devices. This is a contribution
from an operator and during the discussion Emile coming from another
operator confirmed that this is a highly relevant use case.

At the IETF we are usually short of feedback from operators. If now two
operators tell us that the WG should work on this issue, we should
consider the item seriously.

Please have a look at the draft and tell us what you think about the idea
and if this should or should not become and IPFIX work item.

Thanks,

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 26 Jul 2007 08:53:05 +0200

Dear all,

Thanks,

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 26 Jul 2007 09:01:26 +0200

Dear all,

Candidate #7 to become a new IPFIX work item is described by
draft-peluso-flowselection-tech-00.txt.

>    Early stage so far, comments/participation encouraged.

This draft discusses an issue, we have been discussing already two or three
years ago: the sampling of flows by a metering/exporting process. Still,
this issue is rather new to the IPFIX WG and has not yet been discussed a
lot. Research work might be needed to fully understand the issue.

Speaking as technical contributor, I think that this work needs to become
better understood and more mature before becoming a WG work item.

Please have a look at the draft and tell us your opinion.

Thanks,

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 26 Jul 2007 09:01:26 +0200

Dear all,

Thanks,

    Juergen

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 26 Jul 2007 13:17:36 +0200

Dear all,

I'm undecided about this draft.

On the one hand, it's a nice idea to have some IEs which allow
describing the type and semantic of other IEs. On the other hand, my
impression is that the usefulness is very restricted and not as broad as
claimed in the introduction of the draft:

   ... Having to do some sort of tool-specific
   configuration (or code modification) on every tool just to tell which
   type each used Information Element has is a huge amount of work for
   users and implementors of enterprise-specific Information Elements.
   Many tools in fact only need to know the field's data type to work on
   them. ...

In my opinion, any deeper analysis and processing of metering data
requires knowledge about the semantic. Just knowing the data type may be
enough to display values correctly. Considering
informationElementSemanticType, you might also know if the calculation
of statistical properties (max, min, mean, variance etc.) makes sense or
not. But any further analysis would probably require more semantical
knowledge. That's why I assume that, in practice, analyzing and
processing enterprise-specific data will mostly be performed with
specialized analysis tools that have been implemented for this specific
purpose. And for these tools, there is no need to export the IE information.

By the way: Have you considered an additional IE like
informationElementDescription holding an octet string describing the
semantic of the IE "in human-readable English"?

Regards,
Gerhard

-- 
Dipl.-Ing. Gerhard Münz
Computer Networks and Internet
Wilhelm Schickard Institute for Computer Science
University of Tuebingen
Sand 13 (Room B309), D-72076 Tuebingen, Germany
Phone:  +49 7071 29-70534       Fax: +49 7071 29-5220
E-mail: [email-masked]
WWW:    http://net.informatik.uni-tuebingen.de/~muenz

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 26 Jul 2007 13:17:36 +0200

Dear all,

Regards,
Gerhard

---

From: Paul Aitken <[email-masked]>
Date: Thu, 26 Jul 2007 15:14:57 +0100

Gerhard,

> semantic of the IE "in human-readable English"?

Perhaps we should just export an XML definition with all the fields 
which we find in IPFIX-INFO? :-)

-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.

---

From: Paul Aitken <[email-masked]>
Date: Thu, 26 Jul 2007 15:14:57 +0100

Gerhard,

---

From: Hitoshi Irino <[email-masked]>
Date: Thu, 26 Jul 2007 23:42:51 +0900

Dear all,

> If yes, until when would you have results?

A day before yesterday, I had a presentation about a performance result
when exporters and collectors use same order, however I didn't have any 
time to talk about the reason of difference of performance.

In the slide,
( http://www3.ietf.org/proceedings/07jul/slides/ipfix-10.pdf )
I use very primitive collector which I implemented. This collector
receive data records and write the data to file.
This collector has internal proprietary format to store to data to file.
The format consists of some Information Elements whose size are full
size. (For example, bgpSourceAsNumber is 4 octets.)

This collector process following steps:
  1) read the data record.
  2) convert received data records to the collector's format on memory.
  3) write to file from memory.

I tested each combination of 3 kind of orders for template of exporter
and format of collector.
  1) An order same as NetFlow version 5
  2) Suggested Order in my draft (separating constant fixed length IEs
and reduced size encoding applicable IEs)
  3) Random order
In this test, exporter's template includes set of Information Elements
same as NetFlow version 5 fields.
These templates and formats are shown in page 9 in the slide.

I measured 10 times this test.
The matrix table in page 4 is the result of this test. Upper one shows
total time, lower one shows ratio.

My collector program can process bulk of sequential Information
Elements. For example in the case reading from date records ordered by
suggested order to writing format formed by suggested order, my
collector program copies first 20 octets (from sourceIPv4Address to
ipClassOfService) at once.

I described counts for processing in from page 10 to 12 in the slide.
For example, in the best case that a exporter uses suggested order and a
collector uses suggested order, the counts for processing (processing
means coping data in this case) is 8 (on middle table in page 11). In
the worst case a exporter uses random order and collector uses random
order, the counts is 17 (on right table in page 11). Difference of the
counts for processing effects their performance.
my presentation shows one of the best case.

Therefore, there are effective case and ineffective case by ordering.

Example of effective case
  writing to files
  real time flow filtering (comparing bulk data)

Example of ineffective case
  when process each Information Element

By the way, if all exporter supports draft-muenz-ipfix-configuration and
operators can set configuration about template completely freely, my
draft is not important, because operator can adjust configuration for
collector.
However currently NetFlow v9 implementations of some router makers are
not configurable at all. Operators can choice preset template on these 
implementations. If this state continues in future, reference order is 
valuable for these exporters' implementations and collectors' 
implementations. Therefore I've thought a reference order should be
defined as Informational. I suggested an order adjusted for collector's
performance, however perhaps it is not suitable for exporter. So I hope
to discuss on WG.

thanks,
Hitoshi Irino

-- 
Hitoshi Irino
NTT Network Service Systems Laboratories
9-11 Midori-cho 3-Chome, Musashino-shi, Tokyo 180-8585 Japan
Tel: +81-422-59-4403  Fax: +81-422-59-4549

---

From: Hitoshi Irino <[email-masked]>
Date: Thu, 26 Jul 2007 23:42:51 +0900

Dear all,

In the slide,

thanks,
Hitoshi Irino

---

From: Paul Aitken <[email-masked]>
Date: Thu, 26 Jul 2007 16:01:53 +0100

Hitoshi-san,

> implementations.

Cisco's new Flexible NetFlow (http://cisco.com/go/fnf) allows users to 
configure which fields are monitored for each individual cache.

Obviously this affects the export templates - but since the user 
configures which fields are in the cache (and not the export templates), 
the router is able to store and export the fields in the most optimal way.

Actually, I think that changing what is monitored by (re)configuring the 
export template(s) is exactly the wrong way to go.

Rather, the user should configure what they want to monitor and the 
export template(s) should be changed by the Exporting Process to support 
the fields that are being monitored.

If your research proves a way to optimise the export, then this model 
also allows the Exporting Process to optimise the exported data 
independently of the user's configuration. So it's definitely a big win.

> Therefore I've thought a reference order should be defined as Informational.

If other groups are able to show similar improvements, then I think we 
will all want to adopt your ideas.

So I hope that everyone will follow Jeurgen's request:

     I would like to ask everybody who has an IPFIX implementation to
     conduct a short experiment in order to find out if the improvement
     is the same for his/her implementations.

---

From: Paul Aitken <[email-masked]>
Date: Thu, 26 Jul 2007 16:01:53 +0100

Hitoshi-san,

---

From: Benoit Claise <[email-masked]>
Date: Thu, 26 Jul 2007 10:04:23 -0500

Dear all,

I have exactly the same concern.

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Thu, 26 Jul 2007 10:04:23 -0500

Dear all,

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Thu, 26 Jul 2007 10:17:47 -0500

Dear all,

During the last IETF, the draft was presented and my initial thought 
was: it makes sense to put the variable length I.E. at the end of 
record. So we could add a few sentences in the implementation guideline 
about it.
Now, when I look at the slide 4 of 
http://www3.ietf.org/proceedings/07jul/slides/ipfix-10.pdf, I see a gain 
in time that varies from 22 to 86%. Pretty impressive.
So if this gain is confirmed, then this is a very interesting draft for 
the working group, which could be experimental to start with...

I'm also happy that Irino-san clarifies the scope the draft below

So the big gain is when writing into a file, while the gain is lower 
(actually no sure if there is one) when writing into a DB

Regards, Benoit.
> when exporters and collectors use same order, however I didn't have 
> any time to talk about the reason of difference of performance.
>
> In the slide,

---

From: Benoit Claise <[email-masked]>
Date: Thu, 26 Jul 2007 10:17:47 -0500

Dear all,

Regards, Benoit.

---

From: "STEPHAN Emile RD-CORE-LAN" <[email-masked]>
Date: Thu, 26 Jul 2007 18:37:13 +0200

HI Juergen and Nevil,
 
Work item #3
Storing IPFIX records in files requires interoperability only when exchanging these files with some one else or some other application. So it's a function of mediation.
 
Work item #4 points indirectly to a major point concern for telcos: how to map a private IE on a standard one? 
 
Work item #5 defines mediator arch & functions. Then it gives many use cases and requirements.
 
Work items #3, #4 and #5 give many functions a mediator may implement.
At this step, the functions required for interoperability between mediators are not identified. So please consider the necessity to define the interoperability requirements prior to re charter.
 
Regards
Emile

> De : Juergen Quittek [mailto:[email-masked]]
> Envoyé : jeudi 26 juillet 2007 08:28
> À : [email-masked]
> Objet : [IPFIX] new work item candidate #4: extended types
> 
> Dear all,
>

---

From: "STEPHAN Emile RD-CORE-LAN" <[email-masked]>
Date: Thu, 26 Jul 2007 18:37:13 +0200

HI Juergen and Nevil,
 
Work item #3

 
 
Regards
Emile

---

From: Benoit Claise <[email-masked]>
Date: Thu, 26 Jul 2007 12:41:54 -0500

Hi Emile,
>
> HI Juergen and Nevil,
>
>  
>
> Work item #3
>
> Storing IPFIX records in files requires interoperability only when 
> exchanging these files with some one else or some other application. 
> So it's a function of mediation.
>
>  
>
> Work item #4 points indirectly to a major point concern for telcos: 
> how to map a private IE on a standard one?
>
I'm not sure this draft answers this question.

Regards, Benoit.
> At this step, the functions required for interoperability between 
> mediators are not identified. So please consider the necessity to 
> define the interoperability requirements prior to re charter.
>
>  
>
> Regards
>
> Emile
>
>  
>
>  
>
> > elements by using option records that map data types to element
>
> > identifiers.
>
> >
>
> ------------------------------------------------------------------------
>

---

From: Benoit Claise <[email-masked]>
Date: Thu, 26 Jul 2007 12:41:54 -0500

Hi Emile,

Regards, Benoit.

---

From: Paul Aitken <[email-masked]>
Date: Thu, 26 Jul 2007 19:19:43 +0100

Emile,

> exchanging these files with some one else or some other application. So 
> it's a function of mediation.

If I consider a slight change of focus and name to "IP Flow Information 
Exchange", then I think an interoperable file format is crucial.

It's possible that simple collectors will want to use the file format to 
store received flows, and such files may be exchanged by researchers for 
analysing data or testing purposes.

So while the file format MAY be a function of mediation, I certainly 
don't think it's exclusively a mediation function.

Cheers.

---

From: Paul Aitken <[email-masked]>
Date: Thu, 26 Jul 2007 19:19:43 +0100

Emile,

Cheers.

---

From: Brian Trammell <[email-masked]>
Date: Thu, 26 Jul 2007 13:34:12 -0500

Benoit, Emile,

>> Work item #4 points indirectly to a major point concern for  
>> telcos: how to map a private IE on a standard one?
>>
> I'm not sure this draft answers this question

Speaking as a co-author of the draft in question, indeed, it does  
not. It aims to provide a generalized method for representing the  
properties of Information Elements as defined in the IPFIX  
Information Model inline within an IPFIX Message. We have a bit more  
work to do on a -01 revision which will address those Information  
Element properties (e.g., units, names) not presently in the draft;  
such a revision is forthcoming.

Information Element equivalence, while potentially interesting, is  
out of scope and would be a subject for future work in another draft.  
I'm not sure how useful it would be beyond two particular cases (1. a  
private IE which is submitted to IANA and becomes an IANA IE; 2. two  
vendors exporting _exactly_ the same data with different IEs within  
their own respective PEN number spaces), but I'll admit I've not  
thought that deeply about it...
>> Work items #3, #4 and #5 give many functions a mediator may  
While the file draft and extended type draft define functions that  
may be implemented within a mediator, they are not specifically  
scoped to the mediator case. So I don't think it's necessary to  
define a mediator architecture in order to specify a file format and  
extended type mechanism.

(i.e., what Paul just said).

Regards,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Thu, 26 Jul 2007 13:34:12 -0500

Benoit, Emile,

Regards,

Brian

---

From: "STEPHAN Emile RD-CORE-LAN" <[email-masked]>
Date: Thu, 26 Jul 2007 20:47:57 +0200

Hi Benoit,
 
The aim of this draft is to standardize the transcoding of private IE to standard datatype. It is a subset of one telco need: mapping private IE to a standard IE. 
 
The approach I propose is really open because new datatypes IE may be added in the future in the IANA IPFIX info model registry. 
 
Regards
Emile
 
________________________________

De : Benoit Claise [mailto:[email-masked]] 
Envoyé : jeudi 26 juillet 2007 19:42
À : STEPHAN Emile RD-CORE-LAN
Cc : Juergen Quittek; Nevil Brownlee; [email-masked]
Objet : Re: [IPFIX] requirements: Extraction, Distribution, Duplication,Aggregation , Storage, Self Description, Anonymization, Obfuscation ...
 
Hi Emile,

HI Juergen and Nevil,
 
Work item #3

Regards, Benoit.

 
 
Regards
Emile

> Please have a look at the draft and send your comments.

________________________________

---

From: "STEPHAN Emile RD-CORE-LAN" <[email-masked]>
Date: Thu, 26 Jul 2007 20:47:57 +0200

Hi Benoit,

 
Regards
Emile

 
Hi Emile,

HI Juergen and Nevil,
 
Work item #3

Regards, Benoit.

 
 
Regards
Emile

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 26 Jul 2007 20:54:03 +0200

Hi Emile,

Am 26.07.2007 18:37 Uhr schrieb "STEPHAN Emile RD-CORE-LAN" unter
<[email-masked]>:

> HI Juergen and Nevil,
>  
> Work item #3
> Storing IPFIX records in files requires interoperability only when exchanging
> these files with some one else or some other application. So it's a function
> of mediation.

I agree that one may see it this way and it is a very good point to consider
when working on a file format.  However, for the discussions we are having
I would prefer to separate the discussion on the file format work item from
the discussion on the mediator work item.

> a private IE on a standard one?

Now I understand your requirement.
However, as it is currently written, the proposed work does not suggest to
map enterprise-specific IEs to standard ones, but to just assign data types
to unknown (enterprise-specific) IEs. This is a use case different to yours.

> Work item #5 defines mediator arch & functions. Then it gives many use cases

I fully agree.
 
> At this step, the functions required for interoperability between mediators
> are not identified. So please consider the necessity to define the
> interoperability requirements prior to re charter.

Thank you  for this hint.
We will consider it if we charter a work item on mediation.

Thanks,

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 26 Jul 2007 20:54:03 +0200

Hi Emile,

I fully agree.

Thank you  for this hint.

Thanks,

    Juergen

---

From: "STEPHAN Emile RD-CORE-LAN" <[email-masked]>
Date: Thu, 26 Jul 2007 21:47:45 +0200

Hi Paul,

I focus on telco current needs.

There are multiple ways to carry a 'file'. You can pipe it through some filtering or aggregation process. In this context Brian draft has great value because it is defining I/O for IPFIX middlewares. 

Regards
Emile

> De : Paul Aitken [mailto:[email-masked]]
> Envoyé : jeudi 26 juillet 2007 20:20
> Objet : Re: [IPFIX] requirements: Extraction, Distribution, Duplication,
> Aggregation , Storage, Self Description, Anonymization, Obfuscation ...
> 
> Emile,
> 
> 
> > Work item #3
> >

---

From: "STEPHAN Emile RD-CORE-LAN" <[email-masked]>
Date: Thu, 26 Jul 2007 21:47:45 +0200

Hi Paul,

Regards
Emile

---

From: "STEPHAN Emile RD-CORE-LAN" <[email-masked]>
Date: Thu, 26 Jul 2007 22:27:52 +0200

Hi Juergen and Nevil,

It does not make sense to separate item 3 and 5 because they compliment together regarding telco needs of standard I/O for IPFIX middlewares. 

I don't support item 4 because it introduces a second level of datamodeling and because the datatype lists are not extensible. 

Regards
Emile

> Envoyé : jeudi 26 juillet 2007 20:54
> À : STEPHAN Emile RD-CORE-LAN; Nevil Brownlee; [email-masked]
> Cc : Romascanu, Dan (Dan)
> Objet : Re: requirements: Extraction, Distribution,
> Duplication,Aggregation , Storage, Self Description, Anonymization,
> Obfuscation ...
> 
> Hi Emile,
> 
> 
> exchanging
> > Work item #4 points indirectly to a major point concern for telcos: how
> to map
> > a private IE on a standard one?
> 
> > Work item #5 defines mediator arch & functions. Then it gives many use
> cases
> > and requirements.
> 
> I fully agree.
> 
> mediators

---

From: "STEPHAN Emile RD-CORE-LAN" <[email-masked]>
Date: Thu, 26 Jul 2007 22:27:52 +0200

Hi Juergen and Nevil,

Regards
Emile

---

From: Brian Trammell <[email-masked]>
Date: Thu, 26 Jul 2007 16:48:55 -0500

Nevil and all,

Comments inline on the proposed work items below...

Cheers,

Brian

> Here's a slightly improved, slightly clearer, version of my earlier  

It's unclear to me that, after the stream restriction change in point  
0. above, this should be Experimental as opposed to Informational...  
Per 2026, Experimental RFCs are intended as an archival record of a  
research work. Unless the primary intention is to change this draft  
to be a description and evaluation of the performance benefits of  
this (and potentially other) stream selection methods, it appears  
very much to be the description of an export method (which will be in  
line with the protocol after the stream restriction change in point  
0. above) that has potential performance and other benefits within a  
defined set of use cases, very much like the recently-approved  
(Informational) Reducing Redundancy draft.

>   Milestones:  WG LC after IETF 70, to IESG before IETF  71.

As per Dan Romascanu's comments on the File Format during the IPFIX  
meeting in Dallas or Montreal (I do not recall which at the moment),  
and, indeed, per the Intended Status on the present revision of the  
document, I believe the File Format draft should be considered as a  
Standards Track RFC. The primary motivation of the draft is to  
promote the interoperability of files used in flow measurement  
infrastructures, to provide in a way a "fourth transport  
protocol" (filesystems and anything that can access filesystems) for  
IPFIX Messages.

>   Milestones:  WG LC after IETF 70, to IESG before IETF  71.

Another point to reiterate here - the File Format draft includes a  
normative reference to this draft, as it uses the Options Template  
defined there to meet the self-description requirement when a file  
contains Enterprise-Specific Information Elements. The two should be  
considered for adoption together. The primary reason the extended  
type draft was created (from text originally in the file draft) was  
that we (the file draft authors) thought that the method might be  
useful on the wire, and we received feedback during and after the  
meeting in Prague that we were not the only ones who thought so.

Regards,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Thu, 26 Jul 2007 16:48:55 -0500

Nevil and all,

Cheers,

Brian

Regards,

Brian

---

From: kobayashi atsushi <[email-masked]>
Date: Fri, 27 Jul 2007 08:16:16 +0900

Hi Nevil and all,

>    Could be integrated with Falko Dressler's draft on IPFIX Aggregation.

I think IPFIX Aggregation-draft focuses on the aggregation method and
reporting methods of aggregation rule.
But, now Mediator-draft describes all-around topics of flow and protocol
mediation. It is different scope.

They seems to be compossible. Do you think integration is necessary ?
Do you mean that we consider more the part of aggregation, as first step?

Best Regards,
Atsushi KOBAYASHI

On Thu, 26 Jul 2007 10:08:16 +1200

>    Possible future WG item, continue discussion on mailing list.

--- 
Atsushi KOBAYASHI  <[email-masked]>
NTT Information Sharing Platform Lab.
tel:+81-(0)422-59-3978 fax:+81-(0)422-59-5637

---

From: kobayashi atsushi <[email-masked]>
Date: Fri, 27 Jul 2007 08:16:16 +0900

Hi Nevil and all,

Best Regards,
Atsushi KOBAYASHI

---

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 27 Jul 2007 14:49:08 +0200

Kobayashi, all,

> Do you mean that we consider more the part of aggregation, as first step?

The aggregation draft is very specific, while your mediator draft is
very broad.
A short summary of the aspects covered by the aggregation draft to
determine potential overlaps:

1) architecture of a concentrator
2) description language for configuring metering processes
   (essentially input filters and flow keys)
3) a couple of new IEs useful to report aggregates and applied input
   filters (e.g. port range IE)
4) a new template type to efficiently report information about applied
   input filters

I assume that item 1) is/will be covered in your draft.

Item 2) is related to configuration and covered by the configuration draft.

The remaining items 3) and 4) might be interesting extensions to
efficiently report how a mediator modifies/filters the original data.
Yet, you would probably try to export this kind of information with
exiting protocol mechanisms at first.

Regards,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 27 Jul 2007 14:49:08 +0200

Kobayashi, all,

Regards,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 27 Jul 2007 14:57:25 +0200

Hi Emile, all,

> It does not make sense to separate item 3 and 5 because they compliment together regarding telco needs of standard I/O for IPFIX middlewares. 

I do not understand your concerns about separating the file format from
the mediator draft.

Similar to the separation of the IPFIX protocol and the IPFIX
architecture, the IPFIX file format describes an interface while the
IPFIX mediator is about an architecture and specific application of
protocol mechanisms. I assume separating these two aspects is a common
approach in the IETF.

Regards,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 27 Jul 2007 14:57:25 +0200

Hi Emile, all,

Regards,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 27 Jul 2007 15:22:47 +0200

Brian,

Brian Trammell wrote:
> considered for adoption together. The primary reason the extended type
> draft was created (from text originally in the file draft) was that we
> (the file draft authors) thought that the method might be useful on the
> wire, and we received feedback during and after the meeting in Prague
> that we were not the only ones who thought so.

The file format should allow to store all the information which is
*available* to the collector when it receives the data. Information
which is not available will just not be stored.

The self-description issue can be considered as an orthogonal problem.
You might recommend in your draft that the exporter should export
self-descriptive data. How this is done does not have to be specified by
the file format, i.e. the reference to the extended types draft is not
necessary.

Regards,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 27 Jul 2007 15:22:47 +0200

Brian,

Regards,
Gerhard

---

From: Paul Aitken <[email-masked]>
Date: Fri, 27 Jul 2007 15:27:13 +0100

Kobayashi-san,

> The aggregation draft is very specific, while your mediator draft is

This is a good point. I wonder whether it would be better to divide the 
mediator into several separate and more general drafts which vendors 
could implement in a single product?

---

From: Paul Aitken <[email-masked]>
Date: Fri, 27 Jul 2007 15:27:13 +0100

Kobayashi-san,

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 27 Jul 2007 16:45:17 +0200

Dear Hitoshi, all,

I had a look at your slides. Here are some thoughts about it.

With respect to the Template structure, I see three aspects which affect
the performance of the collector:

- Order of IEs.
- Reduced size encoding.
- Position of variable length IEs in the record.

Let's assume the collector has to copy the record data into an internal
data structure positioned at a certain address in memory.
(Only) if the following two conditions are fulfilled, a performance gain
can be achieved by copying blocks of consecutive IEs:

1. Order of IEs (within the block) is the same in the Template and in
   the internal data structure
2. No reduced size encoding is applied.

As you wrote in your email, your collector makes use of this block
copying feature. And this feature is the main reason why your collector
is so fast when using the "suggested order" at both exporter and collector.

Jürgen: Common collectors not supporting this block copying feature
won't show the same performance gain. So, just repeating the experiments
with other collector implementations does not seem to be very promising.

Here are some further thoughts:
- Having the same order of IEs while using reduced size encoding at
  the same time does not enable to copy multiple IEs at once.
  => no performance gain
- Not using reduced size encoding at all while allowing random IE
  orders, we can omit type casts in the collector, but casts usually
  do not actually require computational resources.
  => no performance gain
- The treatment of variable length IEs always results in performance
  drawbacks because the offsets of the different IEs cannot be
  calculated in advance. So, you would probably not use variable length
  IEs at all if performance is an issue. Also, I do not see how the
  performance can be increased significantly if variable length IEs are
  positioned at the end of the record. You still need to calculate all
  offsets to determine record and set boundaries.

A final comment: You often need to convert values from network-byte
order to host-byte order. If you do this, you have to process the
corresponding IEs separately and cannot use the block copy.

Conclusion: Only under the specific conditions described above, using a
specialized collector, the performance can be improved.

Regards,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 27 Jul 2007 16:45:17 +0200

Dear Hitoshi, all,

- Order of IEs.
- Reduced size encoding.

Regards,
Gerhard

---

From: kobayashi atsushi <[email-masked]>
Date: Tue, 31 Jul 2007 00:35:54 +0900

Hi Gerhard,

> exiting protocol mechanisms at first.

Thank you for your suggestion.
Surely, the mechanism of exporting method might be one of my action
items.

Does the overview of final documents become as follows ?

1) Architecture of IPFIX Concentrator/Mediator
Maybe, the internal component part of IPFIX Mediator draft covers the
architecture of IPFIX Concentrator.

2) Configuration Data Model for IPFIX and PSAMP
Your draft will covers the cofiguration data model for IPFIX
Concentrator/Mediator devices? I will check your draft, again.

3) New information Elements for IPFIX Concentrator/Mediator
My draft proposes a couple of new IEs. But, they are different from what
you think. After discussing item#1 and item#4, I think it is better that
we consider what information elements are needed.

4) Reporting Method for IPFIX Concentrator/Mediator
This draft will cover the reporting method of aggregation draft.
But, I am not sure clearly that the necessity of the reporting method is
for only IPFIX Concentrator/Mediator or whole of Exporter. I think that
we will discuss this item after our agreement for definition of
mediator/concentrator.

At first, I would like to achieve the consensus of IPFIX members about
item#1 in order to progress the approach.

> Dipl.-Ing. Gerhard M〓z

---

---

From: kobayashi atsushi <[email-masked]>
Date: Tue, 31 Jul 2007 00:35:54 +0900

Hi Gerhard,

---

---

From: kobayashi atsushi <[email-masked]>
Date: Tue, 31 Jul 2007 01:36:33 +0900

Hi Paul,

On Fri, 27 Jul 2007 15:27:13 +0100

> could implement in a single product?

Do you mean my draft should be separated into each device, as follows?

- IPFIX concentrator
- IPFIX proxy
- IPFIX firewall
- IPFIX distributor 
It distributes Flow Records to several collectors. My draft has not
described enough, yet.
...

I can't name the flow mediation device that modifies the Flow Records.

Surely, making up the drafts of each device is one of approaches.

But, IPFIX protocol considerations and internal component might become
similar description. Other things could be common parts.
I think that making up the document of similar devices might be unhappy.
Even if becoming the separated drafts, the overview draft of IPFIX
Mediators might be necessary for common parts.

Roughly, it could be separated into the IPFIX protocol Mediation draft
and the Flow Mediation draft.

And also, the Flow Mediation could be separated into changing the
granularity of Flow, such as filtering and aggregation, and changing the
content of Flow Records, such as modification.

But yet, should my draft be separated into some devices?

---

---

From: kobayashi atsushi <[email-masked]>
Date: Tue, 31 Jul 2007 01:36:33 +0900

Hi Paul,

- IPFIX concentrator
- IPFIX proxy
- IPFIX firewall
- IPFIX distributor 

---

---

From: Paul Aitken <[email-masked]>
Date: Mon, 30 Jul 2007 21:26:44 +0100

Kobayashi-san,

> Surely, making up the drafts of each device is one of approaches.

Exactly, instead of covering such a broad scope in one draft. However, 
it's only one possible approach.

> Mediators might be necessary for common parts.

True.

> But yet, should my draft be separated into some devices?

I had hoped some other IPFIX-ers would comment.

---

From: Paul Aitken <[email-masked]>
Date: Mon, 30 Jul 2007 21:26:44 +0100

Kobayashi-san,

True.

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 31 Jul 2007 11:09:15 +0200

Kobayashi, Paul, all,

Paul Aitken wrote:
>>> This is a good point. I wonder whether it would be better to divide
>>> the mediator into several separate and more general drafts which
>>> vendors could implement in a single product?
>>
>> - IPFIX distributor It distributes Flow Records to several collectors.
>> My draft has not
>> described enough, yet.
>> ...
>>

IPFIX concentrator, IPFIX proxy, IPFIX firewall, IPFIX distributor etc.,
do we really need to distinguish between this whole bunch of new devices?

I think these expressions reflect more the role or function of a
mediator within a given network.

The question is also, if operators really need all of these new
functionalities, or if we talk about potentially useful things that will
never be used in practice.

> I had hoped some other IPFIX-ers would comment.

I see the following two aspects:

1) *Relaying and multiplexing of IPFIX transport sessions* without
changing or filtering the data. This is currently called "protocol
mediation". If this function is really requested by operators, there are
some issues to be solved, like:
- how to translate knowledge about the original observation domain into
option data?
- how to convert relative timestamps?
- how to do the template management, especially in case of multiplexing?

2) *Filtering, anonymization and aggregation* of measurement data.
This is currently called "flow mediation", yet I think that it should
not be restricted to flow data, and also not be restricted to mediators.
The issues here are mostly related to the reporting of what is going on
in the Metering Process, i.e. what kind of filtering, what kind of
anonymization, what kind of aggregation etc. takes place.
I think that all these functions are not specific to the Mediator, i.e.
they can also be implemented in a usual exporter. So I would not cover
them in a mediator draft, but in a more generic draft (or several
problem-specific ones) describing advanced functions within the Metering
Process and how to report them.

Regards,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 31 Jul 2007 11:09:15 +0200

Kobayashi, Paul, all,

Regards,
Gerhard

---

From: kobayashi atsushi <[email-masked]>
Date: Wed, 01 Aug 2007 02:24:54 +0900

Hi Gerhard,

I understood what you thought. 
The description of each functions, aggregation, anonymization and filtering
might be better to move to other drafts. I would like to know other
IPFIX members comments about this approach.

To prevent from being misunderstood, I do not intend to restrict these
functions in mediator. Even if the IPFIX mediator draft describes the
several functions, I think that it is hard to say that they are
restricted in only mediator functions.

First of all, the concept of flow includes the aggregated Flow and fine
grained Flow. It implies the metering process of the original exporter
covers the aggregation function.
And also, the filtering function of Flow Records seems to be covered by
the process of the original Exporter according to IPFIX architecture
draft figure 6. 

Regards,
Atsushi KOBAYASHI

On Tue, 31 Jul 2007 11:09:15 +0200

---

---

From: kobayashi atsushi <[email-masked]>
Date: Wed, 01 Aug 2007 02:24:54 +0900

Hi Gerhard,

Regards,
Atsushi KOBAYASHI

---