Subject: [IPFIX] Mediator Problem Statement: Review

From: Gerhard Muenz <[email-masked]>
Date: Mon, 17 Nov 2008 12:09:01 +0100

Dear Atsushi, all,

I reviewed the mediator problem statement draft.

For me, the purpose of the draft is not quite clear. Is it to justify
the need of IPFIX Mediators? Then, the focus should be more on
monitoring situations that cannot be adequately handled without this
intermediate box between exporters and collectors.

If functionalities like configurable flow keys, flow selection,
anonymization are realized on an IPFIX exporter directly, they can be
considered as part of the Metering Process. So, there is no mediation
involved.

Therefore, I would restrict the usage of the term "mediation" to the
IPFIX mediator case.

Regards,
Gerhard

> Abstract
> 
>    Flow-based measurement is currently a popular method for various

Remove "currently". We should assume that this is also valid in future.

>    network monitoring usages.  The sharing of flow-based information
>    among orthogonal monitoring applications raises open issues in terms

not sure what you mean by "orthogonal"

>    of scalability, reliability and flexibility that IPFIX Mediation may
>    help resolve.  IPFIX Mediation reroutes, replicates, filters,
>    aggregates, correlates or modifies Flow Records or Packet Reports, or
>    changes a transport protocol.  This document describes the

You could introduce more clearly two classes of mediation:
- mediation of content which changes the information, e.g. sampling,
filtering, aggregation, anonymization,...
- mediation of IPFIX transport which does not affect the information,
e.g. rerouting, replication, changing the transport protocol etc.
(storage would also fall into this category)

>    applicability of IPFIX Mediation and the problems that IPFIX
>    Mediation might encounter.
> 
> 
> Table of Contents
> 
>    1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
>    2.  Terminology and Definition . . . . . . . . . . . . . . . . . .  4
>    3.  Flow-Based Mediation: Applicability Examples . . . . . . . . .  9
>      3.1.  IPFIX Export Across Domains  . . . . . . . . . . . . . . .  9
>      3.2.  Data Retention . . . . . . . . . . . . . . . . . . . . . .  9
>      3.3.  Interoperability between Legacy Protocols and IPFIX  . . .  9
>      3.4.  Rerouting Flow Records/Packet Reports  . . . . . . . . . . 10
>      3.5.  IPFIX Export from Branch Office  . . . . . . . . . . . . . 10
>      3.6.  Correlation of Flow Records/Packet Reports Information . . 11
>    4.  Approaches to Scalability  . . . . . . . . . . . . . . . . . . 12
>      4.1.  Adjusting Sampling Rates . . . . . . . . . . . . . . . . . 12
>      4.2.  Flow Aggregation . . . . . . . . . . . . . . . . . . . . . 12
>        4.2.1.  Flow Aggregation on Original Exporters . . . . . . . . 13
>        4.2.2.  Flow Aggregation on IPFIX Concentrators  . . . . . . . 13
>      4.3.  Time Composition . . . . . . . . . . . . . . . . . . . . . 13
>      4.4.  Space Composition  . . . . . . . . . . . . . . . . . . . . 14
>      4.5.  Distributing Load among Collectors . . . . . . . . . . . . 14
>      4.6.  Flow Selection Sampling  . . . . . . . . . . . . . . . . . 14
>    5.  Problems with using IPFIX Mediators  . . . . . . . . . . . . . 15
>      5.1.  Loss of Observation Point Information  . . . . . . . . . . 15
>      5.2.  Loss of Base Time Information  . . . . . . . . . . . . . . 16
>      5.3.  Loss of Option Template Information  . . . . . . . . . . . 16
>      5.4.  Observation Domain ID and Template ID Management . . . . . 16
>      5.5.  Transport Sessions Management  . . . . . . . . . . . . . . 16
>    6.  Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 18
>    7.  Security Considerations  . . . . . . . . . . . . . . . . . . . 20
>    8.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 21
>    9.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 22
>      9.1.  Normative References . . . . . . . . . . . . . . . . . . . 22
>      9.2.  Informative References . . . . . . . . . . . . . . . . . . 22
>    Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 23
>    Intellectual Property and Copyright Statements . . . . . . . . . . 25
> 
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                 [Page 2]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
> 1. Introduction
> 
> 
>    While the IPFIX requirements defined in [RFC3917] mention an
>    intermediate function, such as an IPFIX Proxy or an Concentrator,

"...mention intermediate functions, such as IPFIX proxies and IPFIX
concentrators [not sure if these terms should start with capital
letters],..."

>    there is no document to define the function called IPFIX Mediation.
>    IPFIX Mediation is an additional function to suit the needs of some
>    measurement system.  In this document, we describe several applicable

Isn't IPFIX Mediation supposed to be an umbrella term which includes
IPFIX concentrators and IPFIX proxies?

>    examples of IPFIX Mediation.  Furthermore, we describe the problems
>    of IPFIX Mediation considering implementations.  These problems can
>    be solved by additional specifications without influencing the
>    present IPFIX specification defined in [RFC5101].

"IPFIX protocol specification"

>    Section 2 describes the terminology used in this document.  Section 3
>    describes applicable examples of IPFIX Mediation.  As more effective
>    cases, section 4 describes some usages of the IPFIX Mediation in
>    large-scale networks.  Finally, section 5 describes the problems an
>    implementation of an IPFIX Mediation device might face.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                 [Page 3]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
> 2. Terminology and Definition
> 
> 
>    The terms in this section are in line with those in the IPFIX
>    specification document [RFC5101] and the PSAMP specification document
>    [I-D.ietf-psamp-protocol].  Additional terms required for the IPFIX

What do you mean by "inline"?
If you reuse terms from RFC5101, then you do not have to repeat them
here. Only additional terms should appear in this section. Otherwise it
is not clear which terms are new.

>    Mediation are also defined here.  All these terms are capitalized in
>    this document.
> 
>    Observation Point
> 
>       An Observation Point is a location in the network where IP packets
>       can be observed.  Examples include: a line to which a probe is
>       attached, a shared medium, such as an Ethernet-based LAN, a single
>       port of a router, or a set of interfaces (physical or logical) of
>       a router.
> 
>       Note that every Observation Point is associated with an
>       Observation Domain (defined below), and that one Observation Point
>       may be a superset of several other Observation Points.  For
>       example, one Observation Point can be an entire line card.  That
>       would be the superset of the individual Observation Points at the
>       line card's interfaces.
> 
>    Observation Domain
> 
>       An Observation Domain is the largest set of Observation Points for
>       which Flow information can be aggregated by a Metering Process.
>       For example, a router line card may be an Observation Domain if it
>       is composed of several interfaces, each of which is an Observation
>       Point.  In the IPFIX Message it generates, the Observation Domain
>       includes its Observation Domain ID, which is unique per Exporting
>       Process.  That way, the Collecting Process can identify the
>       specific Observation Domain from the Exporter that sends the IPFIX
>       Messages.  Every Observation Point is associated with an
>       Observation Domain.  It is RECOMMENDED that Observation Domain IDs
>       also be unique per IPFIX Device.
> 
>    Flow Key
> 
>       Each of the fields that:
> 
>       1. belong to the packet header (e.g., destination IP address),
> 
>       2. are a property of the packet itself (e.g., packet length),
> 
>       3. are derived from packet treatment (e.g., Autonomous System (AS)
>       number),
> 
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                 [Page 4]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
>       and that are used to define a Flow are termed Flow Keys.
> 
>    Flow Record
> 
>       A Flow Record contains information about a specific Flow that was
>       observed at an Observation Point.  A Flow Record contains measured
>       properties of the Flow (e.g., the total number of bytes for all
>       the Flow's packets) and usually characteristic properties of the
>       Flow (e.g., source IP address).
> 
>    Packet Reports
> 
>       Packet Reports comprise a configurable subset of a packet's input
>       to the Selection Process, including the Packet Content,
>       information relating to its treatment (for example, the output
>       interface), and its associated selection state (for example, a
>       hash of the Packet Content).
> 
>    Exporting Process
> 
>       The Exporting Process sends Flow Records to one or more Collecting
>       Processes.  The Flow Records are generated by one or more Metering
>       Processes.
> 
>    Exporter
> 
>       A device that hosts one or more Exporting Processes is termed an
>       Exporter.
> 
>    IPFIX Device
> 
>       An IPFIX Device hosts at least one Exporting Process.  It may host
>       further Exporting Processes and arbitrary numbers of Observation
>       Points and Metering Processes.
> 
>    Collecting Process
> 
>       A Collecting Process receives Flow Records from one or more
>       Exporting Processes.  The Collecting Process might process or
>       store received Flow Records, but such actions are out of scope for
>       this document.
> 
>    Collector
> 
>       A device that hosts one or more Collecting Processes is termed a
>       Collector.
> 
> 
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                 [Page 5]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
>    IPFIX Message
> 
>       An IPFIX Message is a message originating at the Exporting Process
>       that carries the IPFIX records of this Exporting Process and whose
>       destination is a Collecting Process.  An IPFIX Message is
>       encapsulated at the transport layer.
> 
>    Information Element
> 
>       An Information Element is a protocol and encoding-independent
>       description of an attribute that may appear in an IPFIX Record.
>       The IPFIX information model [RFC5102] defines the base set of
>       Information Elements for IPFIX.  The type associated with an
>       Information Element indicates constraints on what it may contain
>       and also determines the valid encoding mechanisms for use in
>       IPFIX.
> 
>    IPFIX Mediation
> 
>       IPFIX Mediation is a function located between Exporting Processes
>       and Collecting Processes.  The IPFIX Mediation can be included in

Isn't it located between a Collecting Process and an Exporting Process?

CP => Mediation => EP

>       any IPFIX Devices.  The IPFIX Mediation consists of a set of some
>       of functions:

"IPFIX Mediation consists of one or multiple of the following functions:"

> 
>       *  rerouting input Flow Records/Packet Reports to a appropriate
>          Collecting Process

BTW: this is a proxy function, so mediation is an umbrella term (see above)

>       *  replicating input Flow Records/Packet Reports

Not clear for what purpose Records should be replicated. Maybe add "in
order to ....".

>       *  filtering and selecting input Flow Records/Packet Reports

remove "input"?

>       *  aggregating input Flow Records/Packet Reports based on new Flow
>          Keys

"based on a new set of Flow Key fields"?

>       *  correlating a set of Flow Records/Packet Reports for creating
>          new Flow Records/Packet Reports with new metrics
> 
>       *  modifying input Flow Records/Packet Reports

remove "input"?

>       *  changing a transport protocol which carries IPFIX Messages

"changing the transport protocol"

>       The modification of Flow Records/Packet Reports includes these
>       functions:
> 
>       *  changing the value of specified Information Elements
> 
>       *  adding new Information Elements by deriving further Flow or
>          packet properties from existing fields or calculating new
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                 [Page 6]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
>          metrics
> 
>       *  deleting specified Information Elements.

I would move these bullet points below "modifying Flow Records/Packet
Reports" in the preceding list because at that time the reader already
wonders what "modify" could mean.
> 
>       IPFIX Mediation can be included in any devices, such as routers,
>       switches, NMS(Network Management Systems), or be deployed in
>       stand-alone devices.

I'm not sure. I would rather restrict IPFIX Mediation functions to IPFIX
Mediators. If performed on the original exporter, we already do have the
Metering Process which can do all these things.

> 
>    Flow-Based Collector Selection
> 
>       The Flow-Based Collector Selection evaluates an input Flow Record/
>       Packet Report based on the value of the specified Information
>       Element and then selects Collector for each input Flow Record/
>       Packet Report.

"Flow-Based Collector Selection is an IPFIX Mediation function which
selects a Collector for every Flow Record/Packet Report. The selection
of a Collector is based on match filters applied to record fields
identified by Information Elements."

>    IPFIX Mediator
> 
>       An IPFIX Mediator contains one or more functions defined in IPFIX

"one or more IPFIX Mediation functions"

>       Mediation.  The IPFIX Mediator can be a stand-alone or a virtual
>       device.  It also contains one or more Collecting Processes and one

What do you mean by "virtual"?

>       or more Exporting Processes.
> 
>    Original Exporter
> 
>       An Original Exporter is an IPFIX Device which hosts Observation
>       Points where IP packets can be directly observed.

"where the metered IP packets are observed"

>    IPFIX Proxy
> 
>       An IPFIX Proxy is an IPFIX Mediator that receives IPFIX Messages
>       from Original Exporter, and sends IPFIX Messages to one or more

It is possible that an IPFIX Proxy receives IPFIX Messages from an IPFIX
Mediator?

Sending to multiple Collectors requires "replication" of IPFIX Messages,
which implies that the included records are replicated as well.
You mention "replication" in the case of IPFIX Distributor below. Maybe
you want to mention it here as well?

>       Collectors.  It may alter a part of IPFIX Message in order to

"a part of the"

>       comply with IPFIX Protocol specifications.  It may also change
>       type of transport protocol, such as UDP, TCP, SCTP and PR-SCTP,

"change the transport protocol"

>       and convert a legacy protocol message to an IPFIX Message, if
>       necessary.
> 
>    IPFIX Concentrator
> 
>       An IPFIX Concentrator is an IPFIX Mediator that receives Flow
>       Records/Packet Reports, aggregates them, then exports the
>       aggregated Flow Records.
> 
>    IPFIX Distributor
> 
>       An IPFIX Distributor is an IPFIX Mediator that reroutes input Flow
>       Records/Packet Reports based on the result of Flow-Based Collector
>       Selection.  It may filter or replicate input Flow Records/Packet
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                 [Page 7]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
>       Reports, if necessary.

What is the difference between IPFIX Proxy and IPFIX Distributor?
Is it that an IPFIX Proxy handles entire IPFIX Messages while an IPFIX
Distributor handles individual records?

>    IPFIX Masquerading Proxy
> 
>       An IPFIX Masquerading Proxy is an IPFIX Mediator that screens out
>       a part of data of input Flow Records/Packet Reports according to
>       configured policies.  It can thus, for example, hide the network
>       topology information or customers' IP addresses.

Note: If you call this a "proxy", then my above distinction between
IPFIX Proxy and IPFIX Distributor does not work.

> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                 [Page 8]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
> 3. Flow-Based Mediation: Applicability Examples
> 
> 
> 3.1. IPFIX Export Across Domains
> 
> 
>    IPFIX export across administrative domains can be used to measure
>    traffic for wide-area traffic engineering, or to analyze the trend of
>    Internet traffic.  In such cases, operators need to adhere to privacy
>    policies and prevent the transmission of confidential information.

"prevent the transmission of confidential information" is unclear.
Rather: "prevent the access to confidential traffic measurements by
other operators"

>    Using an IPFIX Masquerading Proxy allows them to operate on Flow
>    Records safely by anonymizing and filtering them.  IP Flow

Who operates on the Flow Records?
Anonymization rather makes it possible to give the Flow Records away to
other operators without violating privacy policies.

>    anonymisation is described in [I-D.boschi-ipfix-anon] in detail.
> 
> 3.2. Data Retention
> 
> 
>    Data retention refers to the storage of traffic data by service
>    providers and commercial organizations.  According to European
>    Commission directives, operators are required to retain both IP and
>    voice traffic data, in wired and wireless networks, generated by end
>    users while using a service provider's services.  The goal of data
>    retention is to ensure that call detail records and Flow Records are
>    available for the purpose of detection, investigation, and
>    prosecution of serious crimes, if necessary.  The European Commission
>    directives define the following data retention services:
> 
>    o  Fixed telephony (includes fixed voice, voicemail, and conference
>       and data calls)
> 
>    o  Mobile telephony (includes mobile voice, voicemail, conference and
>       data calls, SMS, and MMS)
> 
>    o  Internet telephony (includes every multimedia session associated
>       with IP multimedia services)
> 
>    o  Internet e-mail
> 
>    o  Internet access
> 
>    By monitoring Flow Records, IPFIX can fulfill these requirements of
>    Internet access services.

Ok, but where is the link to IPFIX Mediation?

> 3.3. Interoperability between Legacy Protocols and IPFIX
> 
> 
>    During the migration process from a legacy protocol such as NetFlow
>    [RFC3954] to IPFIX, both NetFlow and IPFIX Exporters will need to co-

"are likely to coexist"

>    exist in the same network.  An IPFIX Proxy which converts a legacy
>    protocol to IPFIX will allow operators to continue measuring Flows
>    from legacy Exporters, even after introducing IPFIX Collectors.
> 
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                 [Page 9]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
> 3.4. Rerouting Flow Records/Packet Reports
> 
> 
>    Recently, several networks seem to have shifted towards integrated

remove "seem to"

>    networks, such as the Internet and MPLS, which includes IPv4, IPv6,
>    and VPN traffic.  Flow Records/Packet Reports of these types need to

"...of different networks need to..."

>    be analyzed separately and from different perspectives.  However,
>    handling them separately without improving the capability of the
>    Collector is difficult.  An IPFIX Distributor rerouting Flow Records/
>    Packet Reports based on the result of Flow-based Collector Selection,
>    would be necessary.  Thus, it allows individual Collectors related to

"An IPFIX Distributor allows separating and rerouting Flow
Records/Packet Reports from different networks to different Collectors
based on Flow-based Collector Selection."

>    each network to analyze traffic data for their own specific purposes.
> 
>    As another example, in case of rerouting specific customer's Flow
>    Records, an IPFIX Distributor needs to identify each customer.  As

it rather needs to identify the customer networks

>    identification data, the RD (Route Distinguisher), ingress IF,

What is "IF"?

>    peering AS number, or BGP next hop are listed.  As shown in the

Where are they listed?

Proposed formulation:
"RD (..), ingress interface, ....., may be evaluated to distinguish
different customer networks."

>    following figure, the IPFIX Distributor reroutes Flow Records based

remove "As shown"

>    on the RD value.  This system allows each customer's traffic to be
>    inspected independently.
> 
>                                                .---------.
>                                                |Traffic  |
>                                          .---->|Collector|<==>Customer#A
>                                          |     |#1       |
>                                          |     '---------'
>                                       RD=100:1
>                         .-----------.    |
>       .--------.        |IPFIX      |----'     .---------.
>       |IPFIX   |        |Distributor| RD=100:2 |Traffic  |
>       |router#1|------->|           |--------->|Collector|<==>Customer#B
>       |        |        |           |          |#2       |
>       '--------'        |           |----.     '---------'
>                         '-----------'    |
>                                       RD=100:3
>                                          |     .---------.
>                                          |     |Traffic  |
>                                          '---->|Collector|<==>Customer#C
>                                                |#3       |
>                                                '---------'
> 
>    Figure A: Rerouting Flow Records to Collectors using IPFIX
>    Distributor
> 
> 3.5. IPFIX Export from Branch Office
> 
> 
>    Generally, in big enterprise networks, traffic data from branch
>    offices is gathered in a central office.  But, in the long distance
>    branch office case, the bandwidth for transport IPFIX is not enough.

"sufficient"

> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                [Page 10]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
>    Therefore, it is beneficial that an IPFIX Concentrator located in a
>    branch office exports aggregated Flow Records to cope with the
>    limitation of bandwidth.
> 
> 3.6. Correlation of Flow Records/Packet Reports Information
> 
> 
>    The correlation of Flow Records/Packet Reports information offers
>    some new metrics.  There are some examples as follows:
> 
>    o  One way delay follows from correlating Packet Reports exported
>       from different Exporters on the path.
> 
>    o  The result of a queueing or rate-limiting function applied to

"queuing"

>       ingress or egress interface follows from correlating Flow Records
>       with the same Flow Key observed at both interfaces.
> 
>    o  Average/maximum/minimum values follow from correlating each in a
>       set of Flow Records.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                [Page 11]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
> 4. Approaches to Scalability
> 
> 
>    Usually, operators measure traffic at several Observation Points for
>    a specific purpose, typically sampling packets with rates ranging
>    from 1/10,000 to 1/100.  This value depends on several factors, such
>    as the capacity of the management network, the available storage and
>    speed of the Collector, and the load on the routers/switches.
> 
>    On the one hand, the number of Observation Points in the networks can
>    even be increased to improve the effectiveness of these methods.  In

"is expected to increase in order to"

>    the near future, we anticipate that the advanced features of IPFIX,
>    such as the monitoring of wide-area traffic matrices and QoS
>    performance, will accelerate IPFIX utilization.
> 
>    On the other hand, the increasing amount of traffic brought about by
>    broadband users might have an impact on measurement parameters, such
>    as the sampling rate or granularity of Flows.  Generally, large-scale
>    networks already have multiple 10 Gb/s links, their total traffic
>    exceeding 100 Gb/s.  In the near future, broadband users' traffic
>    will increase by approximately 50% per year according to [TRAFGRW].
>    When operators monitor traffic of 500 Gb/s with a sampling rate of
>    1/1000, the amount of exported Flow Records from Exporters could
>    exceed 50 kFlows/s.  This value is beyond the ability of a single
>    Collector.
> 
>    This section explains how operators can cope with such a huge amount
>    of Flow Records using available IPFIX solutions.  Generally, the
>    solutions encompass two approaches: reducing the amount of exported
>    Flow Records or increasing the capacity of the Collecting Process.
>    The following sub-sections show each solution.
> 
> 4.1. Adjusting Sampling Rates
> 
> 
>    Adjusting the sampling rate can reduce the amount of Flow Records,
>    and a flow-based measurement system can thus easily adapt to the
>    ability of the Collecting and Exporting Processes.  However, in that
>    case, Flows with small traffic volumes could easily get lost.  If
>    traffic incidents happened, operators would no longer be able to
>    investigate traffic change.  While traffic volumes on networks

"If traffic volumes ..."

>    continue to increase, operators will not be able to maintain the
>    sampling rates currently used.  In the near future, flow-based
>    measurement systems possibly will not be able to detect traffic
>    anomalies which can currently be detected.
> 
> 4.2. Flow Aggregation
> 
> 
>    The simplest types of Flows are those comprised of all packets having
>    a fixed five tuple of protocol, source and destination IP addresses,
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                [Page 12]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
>    and source and destination port numbers.  On the other hand, choosing
>    a shorter Flow Key, such as a three tuple or two tuple, or a single
>    Flow Key, such as a network prefix, peering AS number, or BGP Next-
>    Hop, creates more aggregated Flow Records.  This solution is
>    especially useful for measurements of traffic exchange in an entire
>    network domain and for easy adjustments to the performance of a
>    Collector.
> 
> 4.2.1. Flow Aggregation on Original Exporters
> 
> 
>    Original Exporters can aggregate Flow Records to reduce the amount of
>    them.  But, in-depth traffic monitoring might not be possible, as it
>    is with the five tuple.  One way to this is to be able to specify the
>    Template Records for specific needs.  This extra flexibility in the

If you want to adopt the terminology from IPFIX configuration, it's the
Cache Layout that is configured (not the Template Record). In the vision
of IPFIX configuration, the Template Record results from the Cache
Layout, not the other way round.

>    Metering Process allows operators to specify their own set of Flow
>    Keys and extra Information Elements in the Template Record.
>    Specifically, Original Exporters classify the Flow Records by their
>    contents, and then aggregate them with appropriate Flow keys based on

You assume one specific implementation where the Metering Process first
produces IP-quintuple Flows which are then aggregated. However, it is
also possible to directly generate Flows with Flow Key fields differing
from the usual IP-quintuple. In this case, it is not an IPFIX Mediation
function but just a Metering Process. Also, I would not call this case
flow "aggregation".

>    a specific application.  There is an application for security,

The Exporter is not aware of any "application", it only functions
according to its configuration. So, I would remove "based on a specific
application". You might replace it by "based on a given configuration".

>    another for capacity planning, and so on.  The content and
>    granularity of the Flow can satisfy the requirements of each
>    Collector with a specific application.
> 
>    On one hand, this optimizes the Metering Process, because only Flows
>    of interest are looked at.  On the other hand, it optimizes the
>    Exporting Process, because only the information of interest is
>    exported.  Finally, this reduces load of the Collecting Process as
>    less Flow Records are handled, and Flow Record filtering and
>    aggregating are required.

Ok. So this subsection proposed a solution that works without IPFIX
Mediation.

> 4.2.2. Flow Aggregation on IPFIX Concentrators
> 
> 
>    Another approach involves a hierarchical measurement system using
>    IPFIX Concentrators.  Aggregation and storage for input Flow Records

RFC 3917:
   A concentrator receives flow records via the IPFIX protocol, merges
   them into more aggregated flow records, and exports them again using
   the IPFIX protocol.

So, storing records is not a primary function of an IPFIX Concentrator
as you write here.

>    on IPFIX Concentrators makes a most useful distributed-collection
>    system.  It allows other devices to retrieve the stored Flow Record.
>    This method increases the capacity of Collecting Process of whole
>    system.  Flow aggregation method is described in
>    [I-D.dressler-ipfix-aggregation] in detail.
> 
> 4.3. Time Composition
> 
> 
>    Time composition is defined as aggregation for consecutive Flow
>    Records with same Flow Keys.  It leads to the same output as setting

rather: "with identical Flow Key values"

>    a longer active interval timer on Original Exporters.  However, an
>    IPFIX Mediation can calculate average, maximum and minimum values of
>    each counter from Flow Records received with shorter interval time.
>    The output allows operators to keep track of changes that might have
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                [Page 13]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
>    happened during the time interval.
> 
> 4.4. Space Composition
> 
> 
>    Space composition is defined as aggregation for one or more Flow
>    Records involved in a larger Observation Domain or a set of
>    Observation Points.  It is divided into two types:
> 
>    o  Space Composition within one Exporter
> 
>       In that case, the spatial range is within one Exporters.  For

"Exporter"

>       example, the Flow Records observed at physical interfaces which
>       belong to virtual interface by link aggregation can be composed to
>       one Flow Records.

I do not understand.

>    o  Space Composition within some Exporters
> 
>       In that case, the spatial range consists of some different
>       Exporters.  For example, the Flow Records observed at same domain,

"in the same domain"

>       such as west area and east area of an ISP network, can be composed
>       to one Flow Records.
> 
> 4.5. Distributing Load among Collectors
> 
> 
>    As described in the previous section, an IPFIX Distributor reroutes
>    Flow Records/Packet Reports to appropriate Collectors based on the
>    result of the Flow-based Collector Selection.  It can thus distribute
>    the load among multiple Collectors according to a specific
>    application, each area, or each customer.
> 
> 4.6. Flow Selection Sampling
> 
> 
>    A Flow selection sampling method is described in
>    [I-D.peluso-flowselection] in detail.  Generally, the distribution of
>    the number of packets per Flow seems to be heavy-tailed.  Most types
>    of Flow Records are likely to be small Flows consisting of a small
>    number of packets.  The flow-based measurement system, in particular
>    the Collecting Process and Exporting Process, is burdened with a huge
>    amount of these small Flows.  If statistics information of small
>    Flows is exported as merging data by applying a policy or threshold,

"merged data"?

>    the burden on measurement system is reduced.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                [Page 14]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
> 5. Problems with using IPFIX Mediators
> 
> 
>    In this section, we focus on the problems related to the use of IPFIX
>    Mediators in consideration of implementation.
> 
> 5.1. Loss of Observation Point Information
> 
> 
>    Both the Exporter IP address indicated by the source IP address of
>    the IPFIX session as well as the Observation Domain ID included in
>    the IPFIX header are likely to be lost in the mediation process

"IPFIX Message header"

>    performed by an IPFIX Mediator.  This IP address and Observation
>    Domain ID indicate the Observation Point information from the
>    viewpoint of the entire network domain.  Such information is
>    necessary for guaranteeing the continuity of the work of the top
>    level Collector.  Even if an IPFIX Mediator could, with some new
>    mechanism, notify Collectors of this Observation Point information,
>    older Collectors might not accept it.  These Collectors would then
>    wrongly assume that the IP address of the IPFIX Mediator is that of
>    the Original Exporter.  The Collector, however, needs to recognize
>    the precise Observation Point whether Flow Records go through an
>    IPFIX Mediator or not.
> 
>    In the following figure, a Collector could identify 2 Exporters with
>    IP addresses of 10.1.1.3 and 10.1.1.2, respectively.  The Collector,
>    however, needs to somehow recognize Router#1 and Router#2, which are
>    the Original Exporters.  Defined notification methods that can be
>    interpreted by Collectors and Mediators are thus necessary.
> 
>    .--------.          .--------.
>    |IPFIX   |          |IPFIX   |
>    |Router#1|--------->|Mediator|---+
>    |        |          |        |   |
>    '--------'          '--------'   |      .---------.
>    IP:10.1.1.1         IP:10.1.1.3  '----->|         |
>    ODID:10             ODID:0              |Collector|
>                                     +----->|         |
>    .--------.                       |      '---------'
>    |IPFIX   |                       |
>    |Router#2|-----------------------'
>    |        |
>    '--------'
>    IP:10.1.1.2
>    ODID:20
> 
>    Figure B: Loss of Observation Point Information.

In some cases, e.g. masquerading, it might be desired to drop OD
information ;)

> 
> 
> 
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                [Page 15]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
> 5.2. Loss of Base Time Information
> 
> 
>    The Export Time field included in the IPFIX header indicates the base
>    time for Flow Records.  In IPFIX Information Elements, described in
>    [RFC5102], there are delta time fields that indicate the time
>    difference from the value of the Export Time field.  If the Flow
>    Records include any delta time fields and the IPFIX Mediator
>    overwrites the Export Time field when sending IPFIX messages, the
>    delta time fields become meaningless and, because Collectors can not
>    recognize this situation, wrong time values are propagated.

In the described case, the delta time fields contain wrong values
regardless of whether the Collector recognizes this situation or not!

> 
> 5.3. Loss of Option Template Information
> 
> 
>    In some cases, depending on the implementation of the IPFIX
>    Mediators, the information that is reported by the Option Templates
>    could also be lost.  If, for example, the sampling rate is not
>    communicated to the Collectors, a Collector would miscalculate the
>    traffic volume.  This might bring crucial problems.  Even if an IPFIX
>    Mediator were to simply relay received Option Template Information,

"was to"

>    the value of its scope fields would become meaningless in the context

"could become"

>    of a different session.  It should be noted that the minimal
>    information to be communicated by an IPFIX Mediator needs to be
>    defined.

remove "it should be noted that"

> 
> 5.4. Observation Domain ID and Template ID Management
> 
> 
>    The Observation Domain ID is locally unique to the Exporting Process
>    in an IPFIX Mediator, just like the Template ID is unique on the
>    basis of the Observation Domain ID.  These renewed identifiers should

Be careful with repeating incomplete RFC5101 specs. Template IDs are
unique per Transport Session!

>    be managed using the Transport Session Information of the Collecting
>    Process.  If IPFIX Mediators could not manage the relations among

"were not able to"

>    these identifiers and the received Transport Session Information, the
>    Mediators would, for example, relay wrong values for the scope fields
>    of the Option Template and for a "Template Withdraw Message".  In

You assume that the Template ID is in the Option's scope. That does not
have to be the case.

>    most cases, a Collector would not be able to interpret the Template
>    ID of a "Template Withdraw Message" and the scope fields of an Option
>    Template.  The Collector would then shut down the IPFIX Session.
> 
> 5.5. Transport Sessions Management
> 
> 
>    How an IPFIX Mediator maintains relationships between the Transport
>    Sessions of Collecting Processes and of Exporting Processes depends
>    on its implementation.  If multiple Transport Sessions of the
>    Collecting Process are relayed to single Transport Session of the
>    Exporting Process and the IPFIX Mediators shuts down the Transport
>    Session of the Exporting Process, Flow Records on other Transport
>    Sessions of the Collecting Processes would not be relayed at all.  In
>    the case of resetting a session of the Collecting Process, the

I do not see a problem here. Outgoing transport session and incoming
transport sessions should be independent and configurable. There is no
automatism to shut down an outgoing session because an incoming session
closes.

> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                [Page 16]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
>    behavior of the IPFIX Mediator needs to be defined.
> 
> 
>    .--------.
>    |IPFIX   |
>    |Router#1|----+
>    |        |    |
>    '--------'    X
>    .--------.    |     .--------.          .---------.
>    |IPFIX   |    '---->|IPFIX   |          |         |
>    |Router#2|--------->|Mediator|----X---->|Collector|
>    |        |    +---->|        |          |         |
>    '--------'    |     '--------'          '---------'
>    .--------.    |
>    |IPFIX   |    |
>    |Router#3|----'
>    |        |
>    '--------'
> 
>    Figure C: Relaying from Multiple Transport Sessions to Single
>    Transport Session.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                [Page 17]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
> 6. Conclusion
> 
> 
>    This document has covered the applicability of IPFIX Mediation and a
>    multitude of problems related to the implementation of IPFIX
>    Mediators.  To assist the ability of the Exporters and Collectors, it
>    should be noted that there are various IPFIX Mediation functions for
>    the operators to select from.  Examples of the applicability of IPFIX
>    Mediation are as follows.
> 
>    o  Regarding IPFIX Exporting across domains, IPFIX Masquerading
>       Proxies help operators to anonymize or filter Flow Records/Packet
>       Reports, preventing privacy violations.
> 
>    o  Regarding data retention, IPFIX Mediators enhance the storage of
>       the measurement system.
> 
>    o  Regarding interoperability, IPFIX Proxies provide interoperability
>       between legacy protocols and IPFIX, even during the migration
>       period to IPFIX.
> 
>    o  Regarding the Flow-based Collector Selection function, in
>       integrated networks, which mix MPLS VPN and IPv4/IPv6, this could
>       be utilized more frequently.  More sophisticated implementation
>       methods would enhance the effectiveness.
> 
>    o  Regarding scalability in large-scale networks, IPFIX Concentrators
>       or IPFIX Distributors help to achieve high sample rates and fine-
>       grained Flow analysis even as networks grow.  As IPFIX Mediation
>       functions, Flow selection sampling, aggregation and composition
>       are beneficial.
> 
>    As a result, the benefits of IPFIX Mediation become apparent.
>    However, there are still some open issues.
> 
>    o  With the use of IPFIX Mediators, both Observation Point and IPFIX
>       header information, such as the Exporter IP address, Observation
>       Domain ID, and Export Time field, might be lost.  This data should
>       therefore be communicated between the Original Exporter and
>       Collector via the IPFIX Mediator.
> 
>    o  With the use of IPFIX Mediators, data advertised by Option
>       Templates from the Original Exporter, such as the sampling rate
>       and sampling algorithm used, might be lost.  If a Collector is not
>       informed of current sampling rates, traffic information might
>       become worthless.
> 
>    o  IPFIX Mediators are required to manage Transport Sessions,
>       Template IDs, and Observation Domain IDs.  Otherwise, anomalous
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                [Page 18]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
>       IPFIX messages could be created.
> 
>    These problems stem from the fact that no standards regarding IPFIX
>    Mediation have been set.  In particular, the minimum set of
>    information which should be communicated between Original Exporters
>    and Collectors, interworking between different IPFIX Transport
>    Sessions, and the internal components of IPFIX Mediators should be
>    standardized.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Kobayashi, et al.        Expires March 30, 2009                [Page 19]
>  
> Internet-Draft         Mediation Problem Statement        September 2008
> 
> 
> 7. Security Considerations
> 
> 
>    A flow-based measurement system might lead to privacy violations,
>    such as the export of Flow Records to an outside address, if the
>    system is not confined to the large-scale network under observation.
>    General security issues of the IPFIX protocol are covered by the
>    security considerations section in [RFC5101].  Security MUST be
>    considered if different networks exchange Flow information.  As the
>    security of the exchange relies mostly on the protocol used, UDP does
>    not seem appropriate for the exchange of information between
>    networks.
> 

_______________________________________________
IPFIX mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ipfix