Subject: Re: [IPFIX] Review (was Re: WG last call on draft-ietf-ipfix-configuration-model-04)

From: Gerhard Muenz <[email-masked]>
Date: Mon, 01 Mar 2010 15:38:48 +0100

Hi Andrew,

Thanks a lot for your thorough review.

Please find answers and comments inline.

Regards,
Gerhard

Andrew Johnson wrote:
> One general comment:
> 
>   Why do only some of the UML fields have settings given>
>   e.g. "direction = both"

The meaning of "= X" is explained in Section 3.1:

    A default value is indicated after the attribute behind the equality
    symbol ("=").  It means that this default value MUST be used by the
    Monitoring Device if the parameter is not configured by the user.

However, nothing is written about what happens if "= X" is not present.

So, I have rewritten the first paragraphs of the section to clarify the 
meaning of default values, multiplicity indicators, and attribute 
properties, as well as their mapping to YANG

Please let me know if the text is ok and answers your question.

NEW:

3.1.  UML Representation

    We use UML class diagrams [UML] to explain the structure of the
    configuration data model.  The attributes of the classes are the
    configuration or state parameters of the Monitoring Device.

     +------------------------------------------------+
     | Destination                                    |
     +------------------------------------------------+
     | name                                           |<>-----+
     | exportMemberType = parallel                    |       | 0..1
     | ipfixVersion = 10                              |       |
     | transportProtocol                              |   +------------+
     | sourceIpAddress[0..*] {[0..1] for UDP and TCP} |   | Transport- |
     | destinationIpAddress                           |   | Layer-     |
     | destinationPort = 4739|4740                    |   | Security   |
     | ifName/ifIndex[0..1]                           |   +------------+
     | sendBufferSize {opt.}                          |
     | rateLimit[0..1]                                |
     | timedReliability = 0 {SCTP only}               |
     | numberOfStreams {opt.} {SCTP only}             |
     | templateRefreshTimeout = 600 {UDP only}        |
     | optionsTemplateRefreshTimeout = 600 {UDP only} |
     | templateRefreshPacket[0..1] {UDP only}         |
     | optionsTemplateRefreshPacket[0..1] {UDP only}  |
     +------------------------------------------------+

                  Figure 2: UML example: Destination class

    As an example, Figure 2 shows the UML diagram of the Destination
    class, which is specified in more detail in Section 4.4.1.  The upper
    box contains the name of the class.  The lower box lists the
    attributes of the class.  Each attribute corresponds to a parameter
    of the configuration data model.

    Behind an attribute's name, there may appear a multiplicity indicator
    in brackets (i.e., between "[" and "]").  An attribute with a
    multiplicity indicator of "[0..1]" represents an OPTIONAL
    configuration parameter which is only included in the configuration
    data if the user configures it.  Typically, the absence of an
    OPTIONAL parameter has a specific meaning.  For example, not
    configuring rateLimit in the Destination class means that no rate
    limiting will be applied to the exported data.  In YANG, an OPTIONAL
    parameter is specified as a "leaf" without "mandatory true"
    substatement.  The "description" substatement specifies the behavior
    for the case that the parameter is not configured.

    A multiplicity indicator of "[0..*]" means that this parameter MAY be
    configured multiple times with different values.  In the example,
    multiple source IP addresses may be configured for a multi-homed
    Exporting Process if SCTP is transport protocol.  In YANG, an
    attribute with multiplicity indicator "[0..*]" corresponds to a
    "leaf-list".

    Attributes without multiplicity indicator may be endued with a
    default value which is indicated behind the equality symbol ("=").
    If a default value exists, the parameter does not have to be
    explicitly configured by the user.  If the parameter is not
    configured by the user, the Monitoring Device MUST use the given
    default value for the given parameter.  In the example, IPFIX version
    10 must be used unless a different value is configured for
    ipfixVersion.  In YANG, an attribute with default value corresponds
    to a "leaf" with "default" substatement.

    In the example, there exist two default values for the destination
    port, namely the registered ports for IPFIX with and without
    transport layer security (i.e., DTLS or TLS), which are 4740 and
    4739, respectively.  In the UML diagram, the two default values are
    separated by a vertical bar ("|").  In YANG, such alternative default
    values cannot be specified formally.  Instead, they are defined in
    the "description" substatement of the "leaf".

    Further attribute properties are denoted in braces (i.e., between "{"
    and "}").  An attribute with property "{opt.}", such as
    sendBufferSize in the Destination class, represents a parameter that
    MAY be configured by the user.  If not configured by the user, the
    Monitoring Device MUST assign an appropriate value for this
    parameter.  As a result, the parameter will always exist in the
    configuration data, yet it is not mandatory for the user to configure
    it.  In YANG, this behavior cannot be specified formally.  Therefore,
    it is specified in the "description" substatement of the "leaf".

    The availability of a parameter may depend on another parameter
    value.  In the UML diagram, such restrictions are indicated as
    attribute properties.  In the example, the parameter timedReliability
    can only be configured if the transport protocol is SCTP ("{SCTP
    only}").  In YANG, such restrictions are formalized in the "when"
    substatement of the "leaf".

    If the multiplicity of a parameter changes depending on another
    parameter value, this is also denoted as an attribute property.  In
    the example, the parameter sourceIpAddress may be configured only
    once if the transport protocol is UDP or TCP ("{[0..1] for UDP and
    TCP}").  In YANG, this is formalized in the "must" substatement of
    the "leaf".

    Another attribute property not shown in the example is "{readOnly}"
    specifying state parameters which cannot be configured.  In YANG,
    this corresponds to the "config false" substatement.

    Attributes without multiplicity indicator, without default value, and
    without "{readOnly}" property are mandatory configuration parameters.
    These parameters MUST be configured by the user unless an attribute
    property determines that the parameter is not available.  In YANG, a
    mandatory parameter corresponds to a "leaf" with "mandatory true"
    substatement.  In the example, the user MUST configure the parameters
    name, transportProtocol, and destinationIpAddress.  The remaining
    parameters have either a default value, the "{opt.}" property, or a
    multiplicity indicator starting with zero.

> ========================================================================
> Section 3:
> ========================================================================
> If a Selection Process is part of a Metering Process and a single
> Selection Process can feed into more than one Cache, then it is
> implied that a single Metering Process can have more than once cache.

Note: The text has slightly changed after Atsushi's review. Now, it 
explicitly says that a Metering Process may contain multiple Selection 
Processes and multiple Caches.

Furthermore, it says that only packets observed at a single Observation 
Point may enter a Selection Process. IMO, this results from the PSAMP 
definitions of Observed Packet Stream and Selection Process. Please tell 
me if this conclusion is wrong.

This is how the text reads now:

                +-----------------------------------+
                | Metering Process                  |
                | +-----------+ Selected            |
       Observed | | Selection | Packet    +-------+ |  Stream of
       Packet  -->| Process   |---------->| Cache |--> Flow Records or
       Stream   | +-----------+ Stream    +-------+ |  Packet Reports
                +-----------------------------------+

      Figure 1: Selection Process and Cache forming a Metering Process

    The configuration data model adopts the separation of Selection
    Processes and Caches in order to support the flexible configuration
    and combination of these functional blocks.  As defined in [RFC5476],
    the Selection Process takes the Observed Packet Stream as its input
    and selects a subset of that stream as its output (Selected Packet
    Stream).  The Observed Packet Stream at the input of a Selection
    Process is composed of packets from a single Observation Point.  The
    action of the Selection Process on a single packet of its input is
    defined by a Primitive Selector or a Composite Selector.  The Cache
    generates Flow Records or Packet Reports from the Selected Packet
    Stream, depending on the configured Cache Mode.

    Beyond the basic Metering Process shown in Figure 1, the
    configuration data model enables the specification of more complex
    configurations.  A Metering Process may be composed of multiple
    Selection Processes and multiple Caches.  A single Cache may process
    the output of multiple Selection Processes.  The Selected Packet
    Stream may be copied to enter several Caches, for example one Cache
    that generates Flow Records and another Cache that generates Packet
    Reports.  Selection Processes can also be cascaded, such that the
    Selected Packet Stream at the output of one Selection Process is
    passed to another Selection Process.  Cascading Selection Processes
    can be useful in specific contexts such as the example in
    Section 7.3.

> I can see why it would be convenient to configure multiple caches to
> be populated from a single Selection Process, and it could even be
> implemented that way (to some extent), but that is not the PSAMP model.
> The configuration model doesn't have to exactly match this model, but
> you should ensure that the differences are made clear.

The PSAMP model has no notion of Caches.

Could you please show me the paragraphs in the PSAMP documents which 
would indirectly prevent the existence of more than one Cache per 
Metering Process?

> ========================================================================
> From section 3.1:
> ========================================================================
> Why have so many SCTP or UDP only options?  Can't you use UML
> inheritance to do something like:
> 
>     +------------------------------------------------+
>     | Destination                                    |
>     +------------------------------------------------+
>     | name                                           |
>     | exportMemberType = parallel                    |
>     | ipfixVersion = 10                              |
>     | transportProtocol                              |
>     | destinationIpAddress                           |
>     | destinationPort = 4739|4740                    |
>     | sendBufferSize {opt.}                          |
>     | rateLimit[0..1]                                |
>     +------------------------------------------------+
>           ^
>           | 1
>      +----+-------------------+
>      |                        |
>      |    +------------------------------------------------+
>      |    | SCTP Destination                               |
>      |    +------------------------------------------------+
>      |    | localIpAddress[0..*] {SCTP only}               |<>-----+	       
>      |    | timedReliability = 0 {SCTP only}               |       | 0..1    
>      |    | numberOfStreams {opt.} {SCTP only}             |       |	       
>      |    | sourceIpAddress[0..1] {UDP only}               |   +------------+
>      |    | templateRefreshTimeout = 600 {UDP only}        |   | Transport- |
>      |    | optionsTemplateRefreshTimeout = 600 {UDP only} |   | Layer-     |
>      |    | templateRefreshPacket[0..1] {UDP only}         |   | Security   |
>      |    | optionsTemplateRefreshPacket[0..1] {UDP only}  |   +------------+
>      |    +------------------------------------------------+
>      |
>      +---------------------+
>                            |
>     +------------------------------------------------+
>     | UDP Destination                                |
>     +------------------------------------------------+
>     | sourceIpAddress[0..1] {UDP only}               |<>-----+	       
>     | templateRefreshTimeout = 600 {UDP only}        |       | 0..1    
>     | optionsTemplateRefreshTimeout = 600 {UDP only} |       |	       
>     | templateRefreshPacket[0..1] {UDP only}         |   +------------+
>     | optionsTemplateRefreshPacket[0..1] {UDP only}  |   | Transport- |
>     +------------------------------------------------+   | Layer-     |
> 						           | Security   |
>                                                          +------------+

This looks nice. Nevertheless, there are some drawbacks:

Since YANG does not support inheritance, it cannot be mapped to YANG 
directly.

Furthermore, I would have to explain even more UML symbols and stuff in 
the document.

Note that the Destination class might be changed as proposed in
http://www.ietf.org/mail-archive/web/ipfix/current/msg05316.html

> Are there no options that are used with TCP?

You mean parameters which are available for TCP only?
I do not know any. If you have some, please let me know.

> Are the security parameters the same across DTLS and TLS?

There was some discussion on the mailing list whether we should add any 
(D)TLS parameters to the IPFIX configuration data model.

The conclusion was that only parameters regarding the mutual 
authentication of Exporters and Collectors should be added. Further 
parameters are too specific.

Regarding authentication, the parameters are the same for DTLS and TLS.

> ========================================================================
> From section 4.1:
> ========================================================================
> Doesn't the direction belong in the Observation Point class, not the
> Interface or LC class.

If the current position of the direction parameter bothers you, we can 
move it.

However, my reasoning is the following:
- If it appears in the ObservationPoint class, it looks like the 
Observation Point is doing some filtering.
- If it appears in the Interface/Linecard class, it looks like the 
filtering is already done at the interface or linecard.

I thought that the interface/linecard is doing the filtering.

There is another small advantage: The current position of the parameter 
makes it easier to add vendor or implementation-specific packet sources 
for which the direction parameter has no meaning, for example:

+------------------+
| PcapFile         |
+------------------+
| filename         |
+------------------+

> ========================================================================
> From Section 4.2.1
> ========================================================================
> Why packetsDropped and not packetsSelected, when PSAMP talks about
> packetsSelected in the stats it needs to export?

All state parameters are adopted from the IPFIX-MIB. There, we have:

+--ipfixSelectionProcessStatsTable(4)
    |
    +--ipfixSelectionProcessStatsEntry(1) 
[ipfixMeteringProcessCacheId,ipfixSelectionProcessIndex,ipfixSelectionProcessSelectorIndex]
       |
       +-- r-n Counter64 ipfixSelectionProcessStatsPacketsObserved(1)
       +-- r-n Counter64 ipfixSelectionProcessStatsPacketsDropped(2)
       +-- r-n TimeStamp ipfixSelectionProcessStatsDiscontinuityTime(3)

Probably, ipfixSelectionProcessStatsPacketsSelected would make more 
sense in the MIB.

I assume that the MIB standardization cannot be stopped any more, right?

So, what do you suggest?

> ========================================================================
> From Section 4.3.1
> ========================================================================
> Isn't is possible to generate a cache without any keys?
> This would generate a cache with a single flow?  Do we allow this?

A the moment, there is no restriction which says that a flow key field 
must exist.

Do you want to add such a restriction?

> Would it be useful to have two lists, one for keys and one for
> non-keys?

There is an argument for having a single list:

A Cache implementation may generate records using the specified field 
order. So, the fields occur in the same order as specified in the 
configuration.

With separate lists for flow key fields and non-key fields, we cannot 
arbitrarily mix these two types of fields.

Is there any advantage of having separate lists?

> For a permanent cache, update counters should be used and for a
> timeout cache the delta counters should be used.  Where is this
> sort of thing enforced?

This is not enforced. I would not strictly enforce it because strange 
configurations might make sense in specific use cases.

However, we could add a note that totalCounters are typically used for 
"permanent" and deltaCounters for "timeout".

Could you provide a sentence or two?

> ========================================================================
> From Section 4.4.1
> ========================================================================
>    exportMemberType:  Configures the export member type that corresponds
>       to the ipfixTransportSessionGroupMemberType object in
>       [I-D.ietf-ipfix-mib].  The following parameter values are
>       specified by the configuration data model:
>       *  primary: primary target of the Exporting Process
>       *  secondary: secondary target of the Exporting Process used when
>          the primary target is not reachable
>       *  parallel: parallel exporting to all destinations and files of
>          the Exporting Process
>       *  loadBalancing: load-balancing between different destinations
>          and files of the Exporting Process
>       "parallel" is the default value if this parameter is not
>       configured.  If one destination or file is configured as "primary"
> 
> ------------------------------------------------------------------------
> Comments
> ------------------------------------------------------------------------
> It's not clear to me that this parameter belongs to the Destination
> class.  The number of restrictions on when you can use the various
> types suggests that we could simplify things.
> 
> A Cache can pass flow records to multiple Exporting Processes, so we
> don't really need a parallel option.  This would mean that each
> Exporting Process is responsible for sending one complete set of
> records.  i.e. one record in, one record out.  It could do that, either
> using a redundancy model or with a load-balancing one.
> 
> In other words, the exportType would be an Exporting Process attribute
> which would indicate whether the list of Destinations should be load-
> balanced across or treated as Primary, Seondary, Tertiary, etc.

At the moment, the parameter is inline with the IPFIX-MIB:

ipfixExportMemberType OBJECT-TYPE
     SYNTAX      INTEGER {
                     unknown(0),
                     primary(1),
                     secondary(2),
                     parallel(3),
                     loadBalancing(4)
                 }
     MAX-ACCESS  read-only
     STATUS      current
     DESCRIPTION
         "The type of a member Transport Session in a Transport
         Session group (identified by the value of ipfixExportIndex,
         ipfixObservationDomainId and ipfixMeteringProcessCacheId).
         The following values are valid:

         unknown(0)
             This value MUST be used if the status of the group
             membership cannot be detected by the equipment. This
             value should be avoided as far as possible.

         primary(1)
             This value is used for a group member that is used as
             the primary target of an Exporter. Other group members
             (with the same ipfixExportIndex and
             ipfixMeteringProcessCacheId) MUST NOT have the value
             primary(1) but MUST have the value secondary(2).
             This value MUST also be specified if the Exporter does
             not support Transport Session grouping. In this case
             the group contains only one Transport Session.

         secondary(2)
             This value is used for a group member that is used as a
             secondary target of an Exporter. The Exporter will use
             one of the targets specified as secondary(2) within the
             same Transport Session group when the primary target is
             not reachable.

         parallel(3)
             This value is used for a group member that is used for
             duplicate exporting i.e., all group members identified
             by the ipfixExportIndex are exporting the same Records
             in parallel. This implies that all group members MUST
             have the same membertype parallel(3).

         loadBalancing(4)
             This value is used for a group member that is used
             as one target for load-balancing. This means that a
             Record is sent to one of the group members in this
             group identified by ipfixExportIndex.
             This implies that all group members MUST have the same
             membertype loadBalancing(4)."
     ::= { ipfixExportEntry 2 }

I'm open to better proposals.

"parallel" and "loadBalancing" can be configured once with a single 
ExportingProcess class parameter.

How would you configure "primary" and "secondary with a single Exporting 
Process parameter?

> ========================================================================
> From Section 4.7: TransportSession class
> ========================================================================
> I think you want to add a timestamp for when the session came up
> and when it went down (unless it's still active).

Currently, there is no such parameter in ipfixTransportSessionTable.

Hm, it seems that we should have asked you to review the IPFIX-MIB :)

Shall we go beyond IPFIX-MIB and add such state parameters?

> There's no association with the relevant destination.  It seems to me
> that your really want this as part of each Destination or FileReader.
> Specifically, I think you want to separate the Transport Session from
> the File Reader stats.  Potentially there could be a one to many
> relationship of Destination and Transport Sessions, each with their own
> timestamp for coming up and going down.

Currently, TransportSession is a mere copy of ipfixTransportSessionTable 
which exists for both, Exporters and Collectors.

In the case of the _Exporting Process_ , we could move the state 
parameters to the _Destination_ and _FileWriter_ class.

I think that there can be at most one Transport Session per Destination 
and per FileWriter object. At least at the same point in time.
However, it may happen that no Transport Session exists, for example in 
the case of a secondary export destination.

So, we could move a single set of Transport Session state parameters to 
the Destination and FileWriter classes. The parameters would appear or 
not, depending on whether the Transport Session exists.

In the case of the _Collecting Process_ , we can also move the Transport 
Session state parameters to the _Receiver_ and _FileReader_ classes. As 
you say, there can be multiple Transport Sessions per Receiver.

Probably, it makes sense to have a container 
<transportSessionStatistics> which groups the state parameters of a 
Transport Session. This container may appear multiple times in the case 
of the Receiver class, and only once in all other classes.

Would you agree with these changes?

-- 
Dipl.-Ing. Gerhard Münz
Chair for Network Architectures and Services (I8)
Technische Universität München - Department of Informatics
Boltzmannstr. 3, 85748 Garching bei München, Germany
Phone:  +49 89 289-18008       Fax: +49 89 289-18033
E-mail: [email-masked]    WWW: http://www.net.in.tum.de/~muenz

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 01 Mar 2010 17:58:43 +0100

Hi Lothar,

Thanks a lot for your review.

Please find answers inline.

Regards,
Gerhard

Lothar Braun wrote:

>> o  Definitions adopted from [RFC5101]: *  Collection Process *
>> Collector *  Data Record *  Exporter *  Flow Record *  Information
>> Element *  IPFIX Device *  IPFIX Message *  Observation Domain *
>> Observation Point *  (Options) Template
>> 
>> o  Definitions adopted from [I-D.ietf-ipfix-file]: *  File Reader *
>> File Writer
>> 
>> o  Definitions adopted from [RFC5476]: *  Filtering *  Observed
>> Packet Stream *  Packet Report *  PSAMP Device *  Sampling *
>> Selection Process *  Selection Sequence *  Selection Sequence
>> Report Interpretation *  Selector, Primitive Selector, Composite
>> Selector
> 
> You might want to add the term "Flow Keys" somewhere to this lists.

Right, done.

>> Cache:  The Cache is a functional block in a Metering Process which
>>  maintains IPFIX Flow Records or PSAMP Packet Reports.  According 
>> to [RFC5101], the maintenance of Flow Records may include creating 
>> new records, updating existing ones, computing Flow statistics, 
>> deriving further Flow properties, detecting Flow expiration, 
>> passing Flow Records to the Exporting Process, and deleting Flow 
>> Records.  The maintenance of Packet Reports covers the same set of 
>> functions.
>> 
>> Cache Layout:  The Cache Layout defines the superset of fields that
>>  are included in the Packet Reports or Flow Records maintained by 
>> the Cache.  The fields are specified by the corresponding 
>> Information Elements.  In general, the largest possible subset of 
>> the specified fields is derived for every Packet Report or Flow 
>> Record.  More specific rules about which fields must be included 
>> are given in Section 4.3.1.
>> 
>> Cache Mode:  The Cache Mode specifies whether Packet Reports or
>> Flow Records are generated by the Cache.  In the case of Flow
>> Records, it also specifies the Flow expiration policy.
>> 
> 
> I had some difficulties to understand the definition of the Cache. I
> could not understand what the packet cache does when generating PSAMP
> Packet Reports. It was clear after reading Section 4.3 when the Cache
> Modes are explained. Maybe there could be some text that explains
> that maintaining Packet Reports and Flow Records is the same task
> except that Packet Records expire after the first packet.

After Atsushi's review, the text of the Cache definition has been 
changed to:

    Cache:  The Cache is a functional block in a Metering Process which
       generates IPFIX Flow Records or PSAMP Packet Reports from a
       Selected Packet Stream, in accordance with its configuration.  If
       Flow Records are generated, the Cache performs tasks like creating
       new records, updating existing ones, computing Flow statistics,
       deriving further Flow properties, detecting Flow expiration,
       passing Flow Records to the Exporting Process, and deleting Flow
       Records.  If Packet Reports are generated, the Cache performs
       tasks like extracting packet contents and derived packet
       properties from the Selected Packet Stream, creating new records,
       and passing them as Packet Reports to the Exporting Process.

Does this text make things clearer?

 > Could there be an example for a Cache?

Not sure what you mean. What kind of example? Where should it appear in 
the text?

>> +-----------------------------------+ | Metering Process
>> | | +-----------+ Selected            | Observed | | Selection |
>> Packet    +-------+ |  Stream of Packet  -->| Process
>> |---------->| Cache |--> Flow Records or Stream   | +-----------+
>> Stream    +-------+ |  Packet Reports 
>> and combination of these functional blocks.  As defined in
>> [RFC5476], the Selection Process takes the Observed Packet Stream
>> as its input and selects a subset of that stream as its output
>> (Selected Packet Stream).  The action of the Selection Process on a
>> single packet of its input is defined by a Primitive Selector or a
>> Composite Selector. The Cache generates Flow Records or Packet
>> Reports from the Selected Packet Stream, depending on the
>> configured Cache Mode.
>> 
>> The Observed Packet Stream at the input of a Selection Process MUST
>> only contain packets originating from a single Observation Domain.
>> Similarly, the Selected Packet Stream at the input of a Cache MUST
>> only contain packets originating from a single Observation Domain.
>> Packets from Observation Points belonging to different Observation
>> Domains MUST NOT enter the same Selection Process or the same
>> Cache.
> 
> When I read this, I had the feeling that this text does poses some
> restriction on how I am able to configure my monitor. But it seems
> like this corresponds to the definition of an Observation Domain:
> 
> Observation Domain
> 
> An Observation Domain is the largest set of Observation Points for 
> which Flow information can be aggregated by a Metering Process. ...
> 
> Is there some further restriction that I miss?

The text has been changed because of some late changes in IPFIX-MIB and 
because of Atsushi's review.

PSAMP defines:

    * Observed Packet Stream

       The Observed Packet Stream is the set of all packets observed at
       the Observation Point.

    * Selection Process

       A Selection Process takes the Observed Packet Stream as its input
       and selects a subset of that stream as its output.

According to my interpretation of these definitions, a Selection Process 
may only processes packets from one Observation Point. Therefore, the 
text needed to be more restrictive.

On the other hand, there is no IPFIX/PSAMP document saying that a 
Metering Process cannot process packets of multiple ODs. Therefore, this 
restriction was removed.

I propose to add another paragraph in response to your next comment. So 
please find the complete new text below.

>> 3.2.  Exporter Configuration
>> 
>> Figure 4 below shows the main classes of the configuration data
>> model which are involved in the configuration of an IPFIX or PSAMP 
>> Exporter.  The role of the classes can be briefly summarized as 
>> follows:
>> 
>> o  The ObservationPoint class specifies an Observation Point (i.e.,
>>  an interface or linecard) of the Monitoring Device at which 
>> packets are captured for traffic measurements.  An object of the 
>> ObservationPoint class may be associated with one or more objects 
>> of the SelectionProcess class configuring Selection Processes that 
>> process the observed packets in parallel.  As long as an 
>> ObservationPoint object is specified without any references to 
>> SelectionProcess objects, the Observation Point is not deployed for
>> traffic measurements.
>> 
>> o  The SelectionProcess class contains the configuration parameters
>>  of a Selection Process.  The Selection Process may be composed of 
>> a single Selector or a sequence of Selectors, defining a Primitive 
>> or Composite Selector, respectively.
>> 
>> The Selection Process selects packets from an Observed Packet 
>> Stream originating from one or multiple Observation Points.
> 
> Could you please introduce the selectAll policy of the
> SelectionProcess at this point?
> 
> When I read this, I was wondering why I cannot connect an Observation
> Point directly to a Cache, if I do not want to sample or filter
> packets.

I propose to mention this in Section 3 already:

    Every Metering Process contains at least one Selection Process and at
    least one Cache.  If all packets in the Observed Packet Stream are
    selected without any Filtering or Sampling, this corresponds to a
    Selection Process implementing a trivial Selector which selects all
    packets.

Is this ok?

>> 4.1.  ObservationPoint Class
>> 
>> +-------------------------------+ | ObservationPoint              |
>>  +-------------------------------+          1
>> +--------------------+ | name
>> |<>----------| Interface/Linecard | | observationPointId {readOnly}
>> |            +--------------------+ | observationDomainId
>> | |                               | 0..*  0..*
>> +--------------------+ |
>> |----------->| SelectionProcess   | 
>> +-------------------------------+            +--------------------+
>> 
>> 
>> +------------------+   +----------------------------------+ |
>> Interface        |   | Linecard                         | 
>> +------------------+   +----------------------------------+ |
>> ifIndex/ifName   |   | entPhysicalIndex/entPhysicalName | |
>> direction = both |   | direction = both                 | 
>> +------------------+   +----------------------------------+
>> 
>> Figure 6: ObservationPoint class
>> 
> 
> Both, the Interface and the Linecard, contain the "direction". Is
> there a reason why it is not a part of the Observation Point if it
> has to be specified in both cases?

Good question which has also been mentioned by Andrew.

There is a small advantage: The current position of the parameter 
"direction" makes it easier to add vendor or implementation-specific 
packet sources for which the direction parameter has no meaning, for 
example:

+------------------+
| PcapFile         |
+------------------+
| filename         |
+------------------+

>> 
>> 4.3.1.  CacheLayout Class
>> 
>> +--------------+ | CacheLayout  | +--------------+   1..*
>> +--------------------------+ |              |<>------| CacheField
>> | |              |        +--------------------------+ |
>> |        | name                     | |              |        |
>> ieId/ieName              | |              |        | ieLength
>> {opt.}          | |              |        |
>> ieEnterpriseNumber[0..1] | |              |        |
>> isFlowKey[0..1]          | +--------------+
>> +--------------------------+
>> 
>> Figure 12: CacheLayout class
>> 
>> A Cache generates and maintains Packet Reports or Flow Records 
>> containing information that has been extracted from the incoming 
>> stream of packets.  Using the CacheField class, the CacheLayout
>> class specifies the superset of fields that are included in the
>> Packet Reports or Flow Records (see Figure 12).
>> 
>> If Packet Reports are generated (i.e., if Cache Mode is
>> "immediate"), every field specified by the Cache Layout MUST be
>> included in the resulting Packet Report unless the corresponding
>> Information Element is not applicable or cannot be derived from the
>> content or treatment of the incoming packet.
> 
> Could you give an example for "not applicable or cannot be derived
> from"? Maybe something like when a source port needs to be extracted
> from an ICMP packet.

Ok. What about the following new paragraph:

    For example, if a field is configured to contain the TCP source port
    (Information Element tcpSourcePort [RFC5102]), the field MUST be
    included in all Packet Reports which are related to TCP packets.
    Obviously, the field value cannot be determined for non-TCP packets.

At the moment, we leave it open how to handle the UDP case. There may be 
very simple Exporters supporting only a single Template per Cache.
In this case, the source port field can be simply set to zero.

>> If Flow Records are generated (i.e., if Cache Mode is "timeout" or 
>> "permanent"), every Flow Key field specified by the Cache Layout
>> MUST be included as Flow Key in the resulting Flow Record unless
>> the corresponding Information Element is not applicable or cannot
>> be derived from the content or treatment of the incoming packet.
>> Two packets MUST be accounted by different Flow Records if
>> different subsets of the Flow Key fields are applicable or
>> derivable.
> 
> I do not understand this. What do you mean with "different subsets of
> the Flow Key fields".
> 
> Is it maybe: "Two packets MUST be accounted by different Flow Records
> if they differ in at least one Flow Key field."

I like this sentence. And it is important that this behavior is well 
understood.

Therefore, I propose the following changes:

    If Flow Records are generated (i.e., if Cache Mode is "timeout" or
    "permanent"), every Flow Key field specified by the Cache Layout MUST
    be included as Flow Key in the resulting Flow Record unless the
    corresponding Information Element is not applicable or cannot be
    derived from the content or treatment of the incoming packet.  Two
    packets MUST be accounted by different Flow Records if they differ in
    at least one Flow Key field.  Hence, if a Flow Key field can be
    determined for one packet but not for the other, the two packets are
    accounted in different Flow Records.

    Every non-key field specified by the Cache Layout MUST be included in
    the resulting Flow Record unless the corresponding Information
    Element is not applicable or cannot be derived for the given Flow.
    Two packets are accounted by the same Flow Record if they only differ
    in non-key fields but not in any of the Flow Key fields.  The same
    applies if one or more non-key fields can be determined for one
    packet but not for the other.

    For example, if a non-key field specifies an Information Element
    whose value is determined by the first packet observed within a Flow
    (which is the default rule according to [RFC5102]), this field MUST
    be included in the resulting Flow Record if it can be determined from
    the first packet of the Flow.

Is this ok?

>> ieId, ieName, ieEnterpriseNumber:  These parameters specify a field
>> by the combination of the Information Element identifier or name, 
>> and the Information Element enterprise number.  Either ieId or 
>> ieName MUST be specified. ieEnterpriseNumber MUST be used for 
>> enterprise-specific Information Elements.  If ieEnterpriseNumber is
>> omitted or zero, this is Information Element is not enterprise- 
>> specific but registered at IANA.
> 
> If I want to configure a custom non-IANA field using the Enterprise
> ID 1. The field should have the ieID 1. What do I have to configure.
> Do I need to configure ieID 1, ieEnterpriseNumber 1 

yes.

> or do I need to give ieID 32768 (the first bit if the id field) and
> ieEnterpriseNumber 1?

no.

> So my question is: Is the enterprise Bit set by the user or is it
> done by the monitor if he sees the ieEnterpriseNumber field?

RFC5102 says:

    The value of these identifiers is in the range of 1-32767.  Within
    this range, Information Element identifier values in the sub-range of
    1-127 are compatible with field types used by NetFlow version 9
    [RFC3954].

    Enterprise-specific Information Element identifiers have the same
    range of 1-32767, but they are coupled with an additional enterprise
    identifier.  For enterprise-specific Information Elements,
    Information Element identifier 0 is also reserved.

So, 32768 is not a valid ID.

I will restrict the value range of all ieId parameters in YANG.

>> ieLength:  This parameter specifies the length of the field in 
>> octets.  A value of 65535 means that the field is encoded as 
>> variable-length Information Element.  For Information Elements of 
>> integer and float type, the field length MAY be set to a smaller 
>> value than the standard length of the abstract data type if the 
>> rules of reduced size encoding are fulfilled (see [RFC5101], 
>> Section 6.2).  If not configured by the user, the field length is 
>> set by the Monitoring Device.
> 
> The monitoring device can set the length field if it nows the ID
> (either IANA or known custom ID). But what if the monitor does not
> know a specific ID? Should there be a default value (which one?) for
> unknown IDs or should the device reject such a configuration?

If the Monitoring Device does not know the ID, the Cache cannot extract 
or derive the field value from the packets, anyway.

Since the Monitoring Device is obliged to export these fields unless 
they are not available, the device needs some knowledge about the IEs. 
So, I think that the device should be able to set a default length.

Do you think that we should change the text?

> I think the device should reject the configuration.

>> 7.3.  Export of Flow Records and Packet Reports
>> 
>> This configuration example demonstrates the combined export of Flow
>>  Records and Packet Reports for a single Observation Point.  A 
>> Selection Process (Selection Sequence ID = 1) applies random
>> Sampling to the stream of observed packets.  The output is passed
>> to a Cache generating Flow Records.  In parallel, the output is
>> passed to a second Selection Process (Selection Sequence ID = 2)
>> which discards all non-ICMP packets.  A second Cache generates
>> Packet Reports of the retained ICMP packets.  The output of both
>> caches is exported to a single Collector using SCTP.
>> 
>> <ipfix xmlns="urn:ietf:params:xml:ns:ietf-ipfix-psamp">
>> 
>> <observationPoint> <name>OP at linecard 3</name> 
>> <observationDomainId>9876</observationDomainId> <interface> 
>> <ifIndex>4</ifIndex> <direction>ingress</direction> </interface> 
>> <selectionProcess>Sampling</selectionProcess> </observationPoint>
>> 
>> <selectionProcess> <name>Sampling</name> 
>> <selectionSequenceId>1</selectionSequenceId> <selector> 
>> <name>Random sampler</name> <selectorId>1</selectorId> 
>> <sampUniProb> <probability>0.01</probability> </sampUniProb> 
>> </selector> <selectionProcess>ICMP</selectionProcess> <cache>Flow
>> cache</cache> </selectionProcess>
>> 
>> <selectionProcess> <name>ICMP</name> 
>> <selectionSequenceId>2</selectionSequenceId> <selector> <name>ICMP
>> filter</name>
>> <selectorId>2</selectorId> <filterMatch> <ieId>4</ieId> 
>> <value>1</value> </filterMatch> </selector> <cache>Packet
>> cache</cache> </selectionProcess>
>> 
>> <cache> <name>Flow cache</name> <cacheMode>timeout</cacheMode> 
>> <maxRecords>4096</maxRecords> <activeTimeout>5</activeTimeout> 
>> <inactiveTimeout>10</inactiveTimeout> <cacheLayout> <cacheField> 
>> <name>Field 1</name> <ieName>sourceIPv4Address</ieName> 
>> <isFlowKey/> </cacheField> <cacheField> <name>Field 2</name> 
>> <ieName>destinationIPv4Address</ieName> <isFlowKey/> </cacheField> 
>> <cacheField> <name>Field 6</name> 
>> <ieName>flowStartMilliSeconds</ieName> </cacheField> <cacheField> 
>> <name>Field 7</name> <ieName>flowEndSeconds</ieName> </cacheField> 
>> <cacheField> <name>Field 8</name> <ieName>octetDeltaCount</ieName> 
>> </cacheField> <cacheField> <name>Field 9</name> 
>> <ieName>packetDeltaCount</ieName> </cacheField> </cacheLayout> 
>> <exportingProcess>Export</exportingProcess> </cache>
>> 
>> <cache> <name>Packet cache</name>
>> <cacheMode>immediate</cacheMode> <maxRecords>512</maxRecords> 
>> <cacheLayout> <cacheField> <name>Field 1</name> <ieId>313</ieId> 
>> <ieLength>64</ieLength> </cacheField> <cacheField> <name>Field
>> 2</name> <ieId>154</ieId> </cacheField> </cacheLayout> 
>> <exportingProcess>Export</exportingProcess> </cache>
>> 
>> <exportingProcess> <name>Export</name> <destination> <name>SCTP
>> collector</name> <transportProtocol>sctp</transportProtocol> 
>> <destinationIpAddress>192.0.2.1</destinationIpAddress> 
>> <timedReliability>0</timedReliability> 
>> <numberOfStreams>2</numberOfStreams> </destination> 
>> </exportingProcess>
>> 
>> </ipfix>
>> 
>> The Observed Packet Stream at the input of the Selection Process 
>> "ICMP" originates from the Selection Process "Sampling", which thus
>>  constitutes a pseudo Observation Point.  In order to inform the 
>> Collector about the cascaded Selection Processes, the Exporting 
>> Process exports two Selection Sequence Report Interpretations as 
>> defined in [RFC5476], section 6.5.1, including the following
>> fields:
> 
> Shouldn't there be something like
> 
> <option> <selectionSequence/> </option>
> 
> in the exportingProcess configuration?

Yes, that would be useful to configure a PSAMP compliant Monitoring Device.

I added

    <options>
       <name>Options 1</name>
       <optionsType>selectionSequence</optionsType>
       <optionsTimeout>0</optionsTimeout>
     </options>

to examples 7.1 and 7.3.

---

From: Lothar Braun <[email-masked]>
Date: Wed, 3 Mar 2010 09:51:00 +0100

Hi,

please find nmy answers inline.

Best regards,
  Lothar

On Mar 1, 2010, at 5:58 PM, Gerhard Muenz wrote:
>>> maintains IPFIX Flow Records or PSAMP Packet Reports.  According to [RFC5101], the maintenance of Flow Records may include creating new records, updating existing ones, computing Flow statistics, deriving further Flow properties, detecting Flow expiration, passing Flow Records to the Exporting Process, and deleting Flow Records.  The maintenance of Packet Reports covers the same set of functions.
>>> are included in the Packet Reports or Flow Records maintained by the Cache.  The fields are specified by the corresponding Information Elements.  In general, the largest possible subset of the specified fields is derived for every Packet Report or Flow Record.  More specific rules about which fields must be included are given in Section 4.3.1.
> After Atsushi's review, the text of the Cache definition has been changed to:
> 

Yes, this definition is clearer.

> Not sure what you mean. What kind of example? Where should it appear in the text?

I think the new definition is good enough in explaining the functions of the cache, so that it should be clear without an example.

>>> Stream    +-------+ |  Packet Reports +-----------------------------------+
>>> The configuration data model adopts the separation of Selection Processes and Caches in order to support the flexible configuration
>> An Observation Domain is the largest set of Observation Points for which Flow information can be aggregated by a Metering Process. ...
> The text has been changed because of some late changes in IPFIX-MIB and because of Atsushi's review.
> 
> PSAMP defines:
> 
>   * Observed Packet Stream
> 
> According to my interpretation of these definitions, a Selection Process may only processes packets from one Observation Point. Therefore, the text needed to be more restrictive.
> 
> On the other hand, there is no IPFIX/PSAMP document saying that a Metering Process cannot process packets of multiple ODs. Therefore, this restriction was removed.
> 
> I propose to add another paragraph in response to your next comment. So please find the complete new text below.
> 
> 
>>> model which are involved in the configuration of an IPFIX or PSAMP Exporter.  The role of the classes can be briefly summarized as follows:
>>> an interface or linecard) of the Monitoring Device at which packets are captured for traffic measurements.  An object of the ObservationPoint class may be associated with one or more objects of the SelectionProcess class configuring Selection Processes that process the observed packets in parallel.  As long as an ObservationPoint object is specified without any references to SelectionProcess objects, the Observation Point is not deployed for
>>> traffic measurements.
>>> of a Selection Process.  The Selection Process may be composed of a single Selector or a sequence of Selectors, defining a Primitive or Composite Selector, respectively.
>>> The Selection Process selects packets from an Observed Packet Stream originating from one or multiple Observation Points.

Yes, I'm fine with this text.

>>> |----------->| SelectionProcess   | +-------------------------------+            +--------------------+
>>> Interface        |   | Linecard                         | +------------------+   +----------------------------------+ |
>>> direction = both |   | direction = both                 | +------------------+   +----------------------------------+
> There is a small advantage: The current position of the parameter "direction" makes it easier to add vendor or implementation-specific packet sources for which the direction parameter has no meaning, for example:
> 
> +------------------+
> | PcapFile         |
> +------------------+
> | filename         |
> +------------------+

Ok, I can see your point (and agree that it is a good idea to have the direction within the linecard/interface class).

>>> A Cache generates and maintains Packet Reports or Flow Records containing information that has been extracted from the incoming stream of packets.  Using the CacheField class, the CacheLayout

Yes, this is what I had in mind.

> At the moment, we leave it open how to handle the UDP case. There may be very simple Exporters supporting only a single Template per Cache.

Could it be a good idea to add an optional "default if key does not apply to packet" field to the CacheLayout class? 
This default value could also be taken if the device does not know how to retrieve a given IE (e.g. if it is a custom vendor specific ID).

The user who configures the device can then decide which value he want's to receive in this case.

Thinking about this a little more I would also add something like this:
	
	If the Cache does not know how to derive a field and does not have 
	a default value for this field, it MUST reject the configuration.

What do you think?

>>> If Flow Records are generated (i.e., if Cache Mode is "timeout" or "permanent"), every Flow Key field specified by the Cache Layout
> I like this sentence. And it is important that this behavior is well understood.
> 

Fine.

>>> by the combination of the Information Element identifier or name, and the Information Element enterprise number.  Either ieId or ieName MUST be specified. ieEnterpriseNumber MUST be used for enterprise-specific Information Elements.  If ieEnterpriseNumber is
>>> omitted or zero, this is Information Element is not enterprise- specific but registered at IANA.
> I will restrict the value range of all ieId parameters in YANG

Ok.

>>> ieLength:  This parameter specifies the length of the field in octets.  A value of 65535 means that the field is encoded as variable-length Information Element.  For Information Elements of integer and float type, the field length MAY be set to a smaller value than the standard length of the abstract data type if the rules of reduced size encoding are fulfilled (see [RFC5101], Section 6.2).  If not configured by the user, the field length is set by the Monitoring Device.
> If the Monitoring Device does not know the ID, the Cache cannot extract or derive the field value from the packets, anyway.

If it cannot derive the field value, it may set the value to 0 or some completely random value. This is the same situation as if you try to extract a source port from ICMP packets.

> Since the Monitoring Device is obliged to export these fields unless they are not available, the device needs some knowledge about the IEs. So, I think that the device should be able to set a default length.

It can always set a default length to an arbitrary value (e.g. every unknown field is 4 bytes). This could result in the device sending arbitrary sized fields with some random value for fields that are unknown to the device.

As a user, I would like to have the device reject such a configuration.

> Do you think that we should change the text?

I think there should be something like: 

	If the IE is unknown to the device and has now length field given, it MUST
	reject the configuration.

>>> Records and Packet Reports for a single Observation Point.  A Selection Process (Selection Sequence ID = 1) applies random
>>> <observationPoint> <name>OP at linecard 3</name> <observationDomainId>9876</observationDomainId> <interface> <ifIndex>4</ifIndex> <direction>ingress</direction> </interface> <selectionProcess>Sampling</selectionProcess> </observationPoint>
>>> <selectionProcess> <name>Sampling</name> <selectionSequenceId>1</selectionSequenceId> <selector> <name>Random sampler</name> <selectorId>1</selectorId> <sampUniProb> <probability>0.01</probability> </sampUniProb> </selector> <selectionProcess>ICMP</selectionProcess> <cache>Flow
>>> <selectionProcess> <name>ICMP</name> <selectionSequenceId>2</selectionSequenceId> <selector> <name>ICMP
>>> filter</name>
>>> <selectorId>2</selectorId> <filterMatch> <ieId>4</ieId> <value>1</value> </filterMatch> </selector> <cache>Packet
>>> <cache> <name>Flow cache</name> <cacheMode>timeout</cacheMode> <maxRecords>4096</maxRecords> <activeTimeout>5</activeTimeout> <inactiveTimeout>10</inactiveTimeout> <cacheLayout> <cacheField> <name>Field 1</name> <ieName>sourceIPv4Address</ieName> <isFlowKey/> </cacheField> <cacheField> <name>Field 2</name> <ieName>destinationIPv4Address</ieName> <isFlowKey/> </cacheField> <cacheField> <name>Field 6</name> <ieName>flowStartMilliSeconds</ieName> </cacheField> <cacheField> <name>Field 7</name> <ieName>flowEndSeconds</ieName> </cacheField> <cacheField> <name>Field 8</name> <ieName>octetDeltaCount</ieName> </cacheField> <cacheField> <name>Field 9</name> <ieName>packetDeltaCount</ieName> </cacheField> </cacheLayout> <exportingProcess>Export</exportingProcess> </cache>
>>> <cacheMode>immediate</cacheMode> <maxRecords>512</maxRecords> <cacheLayout> <cacheField> <name>Field 1</name> <ieId>313</ieId> <ieLength>64</ieLength> </cacheField> <cacheField> <name>Field
>>> 2</name> <ieId>154</ieId> </cacheField> </cacheLayout> <exportingProcess>Export</exportingProcess> </cache>
>>> collector</name> <transportProtocol>sctp</transportProtocol> <destinationIpAddress>192.0.2.1</destinationIpAddress> <timedReliability>0</timedReliability> <numberOfStreams>2</numberOfStreams> </destination> </exportingProcess>
>>> </ipfix>
>>> The Observed Packet Stream at the input of the Selection Process "ICMP" originates from the Selection Process "Sampling", which thus
>>> constitutes a pseudo Observation Point.  In order to inform the Collector about the cascaded Selection Processes, the Exporting Process exports two Selection Sequence Report Interpretations as defined in [RFC5476], section 6.5.1, including the following
>>> fields:

Fine, thank you.

Best regards,
  Lothar

--
Lothar Braun
Chair for Network Architectures and Services (I8)
Department of Informatics
Technische Universität München
Boltzmannstr. 3, 85748 Garching bei München, Germany
Phone:  +49 89 289-18010       Fax: +49 89 289-18033
E-mail: [email-masked]

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 03 Mar 2010 10:36:45 +0100

Hi Lothar,

Ok, we have an open issue regarding the Cache Layout, which is important.

So, other IPFIX people are encouraged to also read this e-mail and 
comment :-)

It's about section 4.3.1, which looks like this after some revisions:

> 4.3.1.  CacheLayout Class
> 
>             +--------------+
>             | CacheLayout  |
>             +--------------+   1..* +--------------------------+
>             |              |<>------| CacheField               |
>             |              |        +--------------------------+
>             |              |        | name                     |
>             |              |        | ieId/ieName              |
>             |              |        | ieLength {opt.}          |
>             |              |        | ieEnterpriseNumber[0..1] |
>             |              |        | isFlowKey[0..1]          |
>             +--------------+        +--------------------------+
> 
>    stream of packets.  Using the CacheField class, the CacheLayout class
>    specifies the superset of fields that are included in the Packet
>    Reports or Flow Records (see Figure 12).
> 
>    If Packet Reports are generated (i.e., if Cache Mode is "immediate"),
>    every field specified by the Cache Layout MUST be included in the
>    resulting Packet Report unless the corresponding Information Element
>    is not applicable or cannot be derived from the content or treatment
>    of the incoming packet.
> 

First question:

What happens if the Cache Layout contains an Information Element the 
Monitoring Device cannot handle?

In my opinion, the only possible answer is that the Monitoring Device 
returns an error. In general, the Monitoring Device MUST notify the user 
about any part of the configuration which is not supported. It MUST NOT 
silently accept a configuration which cannot be completely implemented.

I propose to add these two sentences to the introduction. This will 
implicitly answer the question.

Second question:

What happens to a field in the Cache Layout which cannot be determined 
from an incoming packet?

Possible answers:

1) It MUST NOT be present in the resulting Packet Report or Flow Record.

2) It MAY still be present in the resulting Packet Report or Flow
    Record, but it MUST be obvious from the field value that the field
    was not be determined from the packet.

3) It MUST still be present in the resulting Packet Report or Flow
    Record and contain a default value which is configured by the user.

4) We use a bit vector field which marks the fields with valid data.

Currently, we do not specify this case, having in mind that 2) is 
allowed. Example: transport port 0 is invalid and may be used for ICMP.

1) is a clean solution, but it may results in 2^n different Templates 
per Cache, n being the number of Flow Key fields. Implementing this is 
more complicated.

3) does not work because the Collector cannot distinguish between 
default values and real values.

4) is disliked by many people, e.g.:
http://www.ietf.org/mail-archive/web/ipfix/current/msg05275.html

So, I see 1) and 2) as the only feasible solutions.
Depending on the decision, I will add some text to the document.

Regards,
Gerhard

---

From: Brian Trammell <[email-masked]>
Date: Wed, 3 Mar 2010 10:39:47 +0100

hi Gerhard, all,

Short, short answers inline below. A much longer message follows.

Cheers,

Brian

On Mar 3, 2010, at 10:36 AM, Gerhard Muenz wrote:

> So, other IPFIX people are encouraged to also read this e-mail and comment :-)
> 
> What happens if the Cache Layout contains an Information Element the Monitoring Device cannot handle?
> 
> In my opinion, the only possible answer is that the Monitoring Device returns an error. In general, the Monitoring Device MUST notify the user about any part of the configuration which is not supported. It MUST NOT silently accept a configuration which cannot be completely implemented.
> 
> I propose to add these two sentences to the introduction. This will implicitly answer the question.

Yes.

> Second question:
> 
> What happens to a field in the Cache Layout which cannot be determined from an incoming packet?
> 
> Possible answers:
> 
> Currently, we do not specify this case, having in mind that 2) is allowed. Example: transport port 0 is invalid and may be used for ICMP.
> 
> 1) is a clean solution, but it may results in 2^n different Templates per Cache, n being the number of Flow Key fields. Implementing this is more complicated.
> 
> 3) does not work because the Collector cannot distinguish between default values and real values.
> 

(2).

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 04 Mar 2010 15:42:59 +0100

Hi Brian, all,

>> I propose to add these two sentences to the introduction. This will implicitly answer the question.

This text will appear in Section 5 which is about "Adaptation to Device 
Capabilities" and therefore should be the right place in the document:

    The deviation concept enables a device to announce deviations from
    the standard model using the "deviation" statement.  For example, it
    is possible to restrict the value range of a specific parameter or to
    define that the configuration of a certain parameter is not supported
    at all.  Hence, deviations are typically used to specify limitations
    due to resource constraints or functional restrictions.  Deviations
    concern existing parameters of the original configuration data model
    and must not be confused with model extensions.  Model extension are
    specified with the "augment" statement and allow adding new
    parameters to the original configuration data model.

    If certain device-specific constraints cannot be formally specified
    with YANG, they MUST be expressed with human-readable text using the
    "description" statement.  The provided information MUST enable the
    user to define a configuration which is entirely supported by the
    Monitoring Device.  On the other hand, if a Monitoring Device is
    configured, it MUST notify the user about any part of the
    configuration which is not supported.  The Monitoring Device MUST NOT
    silently accept configuration data which cannot be completely
    enforced.

    Just like features, deviations and model extensions are announced in
    NETCONF's <hello> message.  A usage example of deviations is given in
    Section 7.5.

Fine.

I'll wait for more feedback before writing more text.

Regards,
Gerhard

---

From: Lothar Braun <[email-masked]>
Date: Thu, 4 Mar 2010 16:43:35 +0100

Hi Gerhard,

On Mar 4, 2010, at 3:42 PM, Gerhard Muenz wrote:
> This text will appear in Section 5 which is about "Adaptation to Device Capabilities" and therefore should be the right place in the document:
> 

It is unclear to me what happens when there is a configuration that cannot be completely enforced.

The device MUST inform the user that the configuration is not fully supported. But what about the configuration: Does the monitor have to apply the configuration anyways or must it reject the configuration with an error? Or can the implementer decide on what to do?

It is not clear to me if "MUST NOT silently accept" corresponds to "reject" or "protest against the configuration but accept it anyways". 

>>> Depending on the decision, I will add some text to the document.

I would also prefer Option 2 but I have no idea if this is possible for all fields.

Best regards,
  Lothar

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 05 Mar 2010 13:33:19 +0100

Hi Lothar, Brian, all,

Please read my comments inline.

> It is not clear to me if "MUST NOT silently accept" corresponds to "reject" or "protest against the configuration but accept it anyways". 

In general, I do not want to specify any constraints regarding the 
handling of configuration errors.

Anyway, if NETCONF is used, the error handling is specified by the error 
option [RFC4741]:

       error-option:

          The error-option element has one of the following values:

          stop-on-error: Abort the edit-config operation on first error.
             This is the default error-option.

          continue-on-error: Continue to process configuration data on
             error; error is recorded, and negative response is generated
             if any errors occur.

          rollback-on-error: If an error condition occurs such that an
             error severity <rpc-error> element is generated, the server
             will stop processing the edit-config operation and restore
             the specified configuration to its complete state at the
             start of this edit-config operation.  This option requires
             the server to support the :rollback-on-error capability
             described in Section 8.5.

This option is sent with the configuration data (e.g., in an edit-config 
operation) and instructs the NETCONF agent what to do in case of a 
configuration error.

So, I added one more sentence to the paragraph to refer to NETCONF error 
handling:

    If certain device-specific constraints cannot be formally specified
    with YANG, they MUST be expressed with human-readable text using the
    "description" statement.  The provided information MUST enable the
    user to define a configuration which is entirely supported by the
    Monitoring Device.  On the other hand, if a Monitoring Device is
    configured, it MUST notify the user about any part of the
    configuration which is not supported.  The Monitoring Device MUST NOT
    silently accept configuration data which cannot be completely
    enforced.  If the NETCONF protocol is used to send configuration data
    to the Monitoring Device, the error handling is specified in the
    NETCONF protocol specification [RFC4741].

Is this sufficient in your opinion?

> I would also prefer Option 2 but I have no idea if this is possible for all fields.

No, it is not possible for all Information Elements because there is not 
always an invalid value.

For example, icmpTypeIPv4 = 0 is valid (Echo reply).

However, we might conclude that the icmpTypeIPv4 field is invalid if we 
have transportProtocol != 1 in the same record.

The description then would be more complicated:

2) It MAY still be present in the resulting Packet Report or Flow
    Record, but it MUST be obvious from the field value itself or from
    the values of other fields in same Data Record that the field
    was not be determined from the packet.

Opinions?

Gerhard

---

From: Lothar Braun <[email-masked]>
Date: Fri, 5 Mar 2010 13:45:04 +0100

Hi,

On Mar 5, 2010, at 1:33 PM, Gerhard Muenz wrote:

> In general, I do not want to specify any constraints regarding the handling of configuration errors.
> 
> Anyway, if NETCONF is used, the error handling is specified by the error option [RFC4741]:
> 
>      error-option:
> 
> This option is sent with the configuration data (e.g., in an edit-config operation) and instructs the NETCONF agent what to do in case of a configuration error.
> 
> So, I added one more sentence to the paragraph to refer to NETCONF error handling:
> 

Perfect.

> No, it is not possible for all Information Elements because there is not always an invalid value.
> 
> However, we might conclude that the icmpTypeIPv4 field is invalid if we have transportProtocol != 1 in the same record.
> 

I think this text is good. Can you add examples for this two conditions? 

Like "value 0 as field value, if the TCP source port is extracted from an ICMP Message" for a scenario where the field value is invalid itself and "icmpTypeIpv4 = 0 and transportProtocol = TCP" for the other case.

Best regards,
  Lothar

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 05 Mar 2010 15:52:42 +0100

Hi Lothar, Brian, all,

Still about the Cache Layout...

> Like "value 0 as field value, if the TCP source port is extracted from an ICMP Message" for a scenario where the field value is invalid itself and "icmpTypeIpv4 = 0 and transportProtocol = TCP" for…

I've realized that packets with zero ports exist in reality:
http://isc.sans.org/port.html?port=0
So, this is a bad example for an invalid field value.
And I do not have a better example for invalid field values.

This is how the text looks like now (new parts underlined):

    If Packet Reports are generated (i.e., if Cache Mode is "immediate"),
    every field specified by the Cache Layout MUST be included in the
    resulting Packet Report unless the corresponding Information Element
    is not applicable or cannot be derived from the content or treatment
    of the incoming packet.  _Any other field specified by the Cache
    Layout MAY only be included in the Packet Report if it is obvious
    from the field value itself or from the values of other fields in
    same Packet Report that the field value was not determined from the
    packet._

    For example, if a field is configured to contain the TCP source port
    (Information Element tcpSourcePort [RFC5102]), the field MUST be
    included in all Packet Reports which are related to TCP packets.
    _Although the field value cannot be determined for non-TCP packets,
    the field MAY be included in the Packet Reports if another field
    contains the transport protocol identifier (Information Element
    protocolIdentifier [RFC5102])._

    If Flow Records are generated (i.e., if Cache Mode is "timeout",
    "natural", or "permanent"), every Flow Key field specified by the
    Cache Layout MUST be included as Flow Key in the resulting Flow
    Record unless the corresponding Information Element is not applicable
    or cannot be derived from the content or treatment of the incoming
    packet.  _Any other Flow Key field specified by the Cache Layout MAY
    only be included in the Flow Record if it is obvious from the field
    value itself or from the values of other Flow Key fields in same Flow
    Record that the field value was not be determined from the packet._
    Two packets MUST be accounted by different Flow Records if they
    differ in at least one Flow Key field.  Hence, if a Flow Key field
    can be determined for one packet but not for the other, the two
    packets are accounted in different Flow Records.

    Every non-key field specified by the Cache Layout MUST be included in
    the resulting Flow Record unless the corresponding Information
    Element is not applicable or cannot be derived for the given Flow.
    _Any other non-key field specified by the Cache Layout MAY only be
    included in the Flow Record if it is obvious from the field value
    itself or from the values of other fields in same Flow Record that
    the field value was not be determined from the packet._  Two packets
    are accounted by the same Flow Record if they only differ in non-key
    fields but not in any of the Flow Key fields.  The same applies if
    one or more non-key fields can be determined for one packet but not
    for the other.

Opinions?

Regards,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Fri, 05 Mar 2010 22:15:21 +0000

Hi Gerhard

In this first part of my reply I'd like to address a specific point.

Gerhard Muenz wrote:
[SNIP]
> would indirectly prevent the existence of more than one Cache per 

I didn't say that you were PSAMP restricted you to one cache per 
Metering Process.  My point was about Selection Processes and my concern 
is over the statistics that PSAMP requires exported for each Selection 
Process.

For example, imagine a simple selection process configuration that 
selects 1 in 10 packets.  That configuration could first be used with 
Cache A:

     Selection Process A1:
       Observed packet stream count: 100
       Selected packet stream count:  10

     Cache A:
       Total packets in: 10

Now that same configuration can be used to populate another Cache later, 
but the statistics need kept separate for each Metering Process:

     Selection Process A1:
       Observed packet stream count: 130
       Selected packet stream count:  13

     Cache A:
       Total packets in: 10

     Selection Process B1:
       Observed packet stream count:  30
       Selected packet stream count:   3

     Cache B:
       Total packets in:  3

PSAMP requires that these stats are kept and exported, which means that 
the stats for A1 and B1 need to be exported under different Selection 
Process IDs.  I guess the real restriction is that a new cache cannot be 
populated from an existing Selection Process.  While a template 
configuration can be used it needs to spawn a separate instances so that 
the stats can kept separate.

Cheers

Andrew

---

From: Gerhard Muenz <[email-masked]>
Date: Sat, 06 Mar 2010 10:51:33 +0100

Hi Andrew,

Thanks for you reply.

> the stats can kept separate.

This is a good example. I see it differently, though.

In order to forward the output of the Selection Process to a second
Cache, you need to change the configuration of the Selection Process.
See how it looks in XML:

On Cache:

SP A1 -> Cache A

  <selectionProcess>
    <name>A1</name>
    <selector>
      ...
    </selector>
    <cache>A</cache>
  </selectionProcess>

After adding the second Cache:

       +-> Cache A
SP A1 -+
       +-> Cache B

  <selectionProcess>
    <name>A1</name>
    <selector>
      ...
    </selector>
    <cache>A</cache>
    <cache>B</cache>
  </selectionProcess>

So, switching from the first configuration to the second one involves a
reconfiguration of the Selection Process A1.

Regardless of how it is realized, the counters need to be reset because
the configuration of the SP has changed.

Before the configuration of the SP is changed, it makes sense to export
all old counters. Otherwise, they will be lost.

Probably, the cacheDataRecords counter of the Cache should also be reset
if the Cache is reconfigured.

Should I add a sentence which says that packetsObserved, packetsDropped,
cacheDataRecords need to be reset upon reconfiguration?

BTW: There are several ways to implement a switch from one configuration
to a new one. If your processes do not support reconfiguration at
runtime, you can replace them by newly instantiated processes. In this
case, the counters will be reset, anyway.

Cheers,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Sat, 06 Mar 2010 13:25:34 +0000

Hi Gerhard

>    configuration or state parameters of the Monitoring Device.

I think this looks good.

> Processes and multiple Caches.

If a Metering Process can have multiple caches and an IPFIX device can 
have multiple Metering Processes, how do we associate the caches with 
Metering Processes?  Can this configuration system only be used to 
configure a single Metering Process within a device?

> me if this conclusion is wrong.

A Selection process should only have a single feed, but can have 
multiple observers.  This results in the number of possible chains 
(Selection Sequences) being equal to the number of Selectors that 
directly feed a cache.

> This is how the text reads now:

See other mail.

>>      |    | localIpAddress[0..*] {SCTP only}               
>> |<>-----+                |    | timedReliability = 0 {SCTP 
>> only}               |       | 0..1         |    | numberOfStreams 
>> {opt.} {SCTP only}             |       |                |    | 
>> sourceIpAddress[0..1] {UDP only}               |   +------------+
>>     | sourceIpAddress[0..1] {UDP only}               
>> |<>-----+               | templateRefreshTimeout = 600 {UDP 
>> only}        |       | 0..1        | optionsTemplateRefreshTimeout = 
>> 600 {UDP only} |       |               | templateRefreshPacket[0..1] 
>> {UDP only}         |   +------------+

I'm not that familiar with YANG, perhaps it deals with the situation 
more gracefully than the UML diagram shows.  Can you point the 
Destination to one of a choice of protocol specific objects?

> There was some discussion on the mailing list whether we should add any 0

OK.

> I thought that the interface/linecard is doing the filtering.

I don't see it as filtering, just part of the definition of the 
Observation Point.  The Interface is most likely able to send and 
receive traffic, but the observation point could be on only the send or 
receive channel.  i.e. It's not a input only interface, but the 
Observation Point could only see input packets.

Currently you have a 1:1 mapping of OP to interface.  If the direction 
is moved to the Observation Point then you could have two observation 
points on the same interface, one on the input side and one on the output.

> I assume that the MIB standardization cannot be stopped any more, right?

I don't know.  Why follow the MIB exactly?  Any idea about the history 
of this choice or dropped rather than selected?

> Do you want to add such a restriction?

A cache without keys will result in a single flow, which seems a bit 
daft.  I guess there is no pressing reason to add this restriction though.

> order. So, the fields occur in the same order as specified in the 

If we're worried about implementation specifics can't they extend the 
YANG model?

> Is there any advantage of having separate lists?

I was just thinking about how you'd specify that at least one key was 
required.

> configurations might make sense in specific use cases.

OK.

> Could you provide a sentence or two?

Perhaps a more general note that not all fields will be appropriate for 
all configurations.  Is the Exporting Device free to change the fields 
if it needs total counters for a permanent cache?

> Shall we go beyond IPFIX-MIB and add such state parameters?

I don't see why not.  Is it possible to add to MIB after the initial 
definition?  Not necessarily now, but perhaps after it's been in use a 
while and certain additions are in demand.

> the case of a secondary export destination.

Well let's imagine and Exporting process with a Primary Destination and 
a Secondary Destination.  The Primary Destination starts a Transport 
Session, which works for a while and then is lost.  The Secondary 
Destination then opens a Transport Session, which works fine.  The 
Exporting Process then manages to reconnect with Primary Destination and 
opens a new Transport Session with it and closes the Secondary 
Destination's Transport Session.

So, we have seen three Transport Sessions and if the Exporting Process 
kept a full history of the statistics it could show:

   Primary Destination (10.2.100.2 port 4739)
     - Transport Session (Open)
         Start Time: 10:01.19
         End Time: --:--.--
         Records sent: 5345
         ...
    -  Transport Session (Closed)
         Start Time: 08:00.00
         End Time: 09:45.14
         Records sent: 123210

   Secondary Destination (10.3.0.15 port 4739)
    - Transport Session (Closed)
        Start Time: 09:45.16
        End Time: 10:01.20
        Records sent: 14022

I guess it would be up to the implementation to control how much history 
was kept.

> not, depending on whether the Transport Session exists.

I think this makes sense, otherwise you have to match the Destination 
configuration to the Transport Session statistics by matching the 
various address and port field.

> you say, there can be multiple Transport Sessions per Receiver.

Well it seems that every field in the Destination Class is present in 
the Transport Session Class.  In fact, the Transport Session seems to be 
a copy of the Destination Class plus some statistics.  This is not true 
of the Receiver class.

Perhaps the Destination Class could be replaced by a Transport Session 
class that has some readonly stats and the Receiver Class can then 
contain multiple Transport Sessions.

In this case and Exporting Process wouldn't have a history of closed 
Transport Sessions, but would have information about the most recent one 
per Destination.  The above example would become:

   Primary Transport Session
     Destination IP address: 10.2.100.2
     Destination port:       4739
     Status:                 Open
     Start Time:             10:01.19
     End Time:               --:--.--
     Records sent:           5345
     ...

   Secondary Transport Session
     Destination IP address: 10.3.0.15
     Destination port:       4739
     Status:                 Closed
     Start Time:             09:45.16
     End Time:               10:01.20
     Records sent:           14022

> Would you agree with these changes?

I think we need a little more discussion :)

Cheers

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Sat, 06 Mar 2010 20:26:07 +0000

Hi Gerhard

> the configuration of the SP has changed.

Well, it might not be clear that the stats need cleared because of the 
nature of the change doesn't effect the selection behaviour of the 
Selection Process.

What about this case:

     SP A1 -> SP B1 -> Cache A

   becomes

                     +-> Cache A
     SP A1 -> SP B1 -+
                     +-> Cache B

In this case SP A1 didn't have its configuration changed at all, yet it 
must have its stats reset for Cache B.  In fact, if you only have one 
set of stats for each SP, then adding a new Cache means resetting the 
stats of the entire tree.  i.e. if the above was part of a tree like 
this, then all every SP in the tree would need its stats reset when 
Cache B was added:

                           +-> Cache A
                 +-> SP Y -+
                 |         +-> Cache C
    OP1 -> SP Z -+
                 |                   +-> Cache A
                 +-> SP A1 -> SP B1 -+
                                     +-> Cache B

Since Cache B needs fresh stats, B1 needs reset, therefore A1 needs 
reset, therefore Z needs reset, which then means all its children need 
reset.

> if the Cache is reconfigured.

Is the cache being reconfigured or just populated from a different source?

You seem to be making assumptions here about how I need to manage my 
stats.  I can conceive of a PSAMP compliant implementation that can be 
configured using your YANG model, that doesn't need to reset any stats 
when a new cache is added to a Selection Sequence.

It would be acceptable for each selection process in the chain to keep 
separate stats for each cache that is feeds.  In fact the configuration 
would act as a template and a new instance of each Selector would be 
created for each use.

> cacheDataRecords need to be reset upon reconfiguration?

That would make sense.  However, my concern was over the stats that are 
exported by PSAMP, which is a different story.

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 11 Mar 2010 11:29:16 +0100

Hi Andrew, all,

Please find my answer at the end of this e-mail.
There are text revisions to approve and a question from my side.

>> case, the counters will be reset, anyway.

So, your concerns are not related to the counters included in the 
configuration data model, but about how PSAMP statistics would be 
exported in Selection Sequence Statistics Reports.

We have discussed two possibilities to maintain the necessary statistics:
- reset all statistics with every reconfiguration of the device
- maintain serapate statistics for every cache
We can conclude that the configuration data model does not collide with 
RFC5476, right?

Nevertheless, I reworked the corresponding parts of the text with the 
following objectives:
- no reference to parameter descriptions in the MIB (as proposed by
   Brian)
- make adaptations to the state parameters adopted from the MIB where it
   is useful
- recommend re-initialization of counters at reconfiguration

As an example, there exists YANG counter types which are set to zero at 
initialization (in contrast to MIB counters which start with an 
arbitrary value). I propose to use these counter types.

So, please comment the current text:

4.2.1.  Selector Class

     +---------------------------------------+
     | Selector                              |
     +---------------------------------------+      1 +-----------------+
     | name                                  |<>------+ SelectAll/      |
     | selectorId {readOnly}                 |        | SampCountBased/ |
     | packetsObserved {readOnly}            |        | SampTimeBased/  |
     | packetsDropped {readOnly}             |        | SampRandOutOfN/ |
     | selectorInitializationTime {readOnly} |        | SampUniProb/    |
     |                                       |        | FilterMatch/    |
     |                                       |        | FilterHash/     |
     +---------------------------------------+        +-----------------+

                          Figure 8: Selector class

[snip]

    As state parameters, the Selector class also contains the Selector
    statistics packetsObserved and packetsDropped as well as
    selectorInitializationTime:

    packetsObserved:  The number of packets observed at the input of the
       Selector since the point in time indicated by
       selectorDiscontinuityTime.  In contrast to
       ipfixSelectionProcessStatsPacketsObserved in the IPFIX MIB module
       [I-D.ietf-ipfix-mib], the counter is initialized with zero.

    packetsDropped:  The number of packets discarded by the Selector
       since the point in time indicated by selectorDiscontinuityTime.
       In contrast to ipfixSelectionProcessStatsPacketsDropped in the
       IPFIX MIB module [I-D.ietf-ipfix-mib], the counter is initialized
       with zero.

    selectorInitializationTime:  The value of sysUpTime at the most
       recent occasion at which the Selector was re-initialized and at
       which the counters packetsObserved and packektsDropped were reset
       to zero.  In contrast to
       ipfixSelectionProcessStatsDiscontinuityTime in the IPFIX MIB
       module [I-D.ietf-ipfix-mib], the zero value does not have any
       specific meaning.

    The counter values of packetsObserved and packetsDropped MUST both be
    set to zero at the time indicated by selectorInitializationTime.  It
    is RECOMMENDED to reset the counters after every reconfiguration of
    the Selector.

[snip]

4.3.  Cache Class

      +-------------------------------------+
      | Cache                               |
      +-------------------------------------+         1 +-------------+
      | name                                |<>---------| CacheLayout |
      | cacheMode                           |           +-------------+
      | maxRecords {opt.}                   |
      | activeTimeout {opt.} {Cache Modes   | 0..*
      |   "timeout" and "natural" only}     |---------------+
      | inactiveTimeout {opt.} {Cache Modes |               | 0..*
      |   "timeout" and "natural" only}     |               V
      | exportInterval {opt.} {Cache Mode   |      +------------------+
      |   "permanent" only}                 |      | ExportingProcess |
      | activeFlows {readOnly}              |      +------------------+
      | inactiveFlows {readOnly}            |
      | cacheDataRecords {readOnly}         |
      | cacheInitializationTime {readOnly}  |
      +-------------------------------------+

                           Figure 11: Cache class

[snip]

    As state parameters, the Cache class contains the Metering Process
    statistics activeFlows, inactiveFlows, and cacheDataRecords, as well
    as cacheInitializationTime:

    activeFlows:  The number of Flows currently active in this cache.
       This value corresponds to ipfixMeteringProcessCacheActiveFlows in
       the IPFIX MIB module [I-D.ietf-ipfix-mib].

    inactiveFlows:  The number of Flows currently inactive in this cache.
       This value corresponds to ipfixMeteringProcessCacheInactiveFlows
       in the IPFIX MIB module [I-D.ietf-ipfix-mib].

    cacheDataRecords:  The number of Data Records generated by this Cache
       since the point in time indicated by cacheIntitializationTime.
       In contrast to ipfixMeteringProcessDataRecords in the IPFIX MIB
       module [I-D.ietf-ipfix-mib], the counter is initialized with zero.

    cacheInitializationTime:  The value of sysUpTime at the most recent
       occasion at which the Cache was re-initialized and at which the
       counter cacheDataRecords was reset to zero.  In contrast to
       ipfixMeteringProcessDiscontinuityTime in the IPFIX MIB module
       [I-D.ietf-ipfix-mib], the zero value does not have any specific
       meaning.

    The counter value of cacheDataRecords MUST be set to zero at the time
    indicated by cacheInitializationTime.  It is RECOMMENDED to reset the
    counter after every reconfiguration of the Cache.

Question:
Is it clear what inactive Flows are?
I can imagine what this number is supposed to by, but I'm not sure.

Regards,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 11 Mar 2010 13:31:04 +0100

Hi Andrew, all,

> configure a single Metering Process within a device?

The model does not group Selection Processes and Caches into Metering 
Processes. This is done on purpose because there is not a single correct 
view.

All Selection Process and all Caches of a Monitoring Device can be 
regarded as one Metering Process.

Alternatively, Selection Processes and Caches which are not directly or 
indirectly linked to each other can be seen as belonging to different 
Metering Processes.

> (Selection Sequences) being equal to the number of Selectors that 

Ok. Single feed means single Observation Point as specified in the draft.

> Destination to one of a choice of protocol specific objects?

Agreed. It is easier to model and describe protocol-specific Destination 
classes. To be consistent, I propose to have protocol-specific Receiver 
classes in the CollectingProcess class as well.

4.4.  ExportingProcess Class

              +--------------------+
              | ExportingProcess   |
              +--------------------+   0..* +------------------+
              | name               |<>------| SctpDestination  |
              |                    |        +------------------+
              |                    |
              |                    |   0..* +------------------+
              |                    |<>------| UdpDestination   |
              |                    |        +------------------+
              |                    |
              |                    |   0..* +------------------+
              |                    |<>------| TcpDestination   |
              |                    |        +------------------+
              |                    |
              |                    |   0..* +------------------+
              |                    |<>------| FileWriter       |
              |                    |        +------------------+
              |                    |
              |                    |   0..* +------------------+
              |                    |<>------| Options          |
              |                    |        +------------------+
              |                    |
              |                    |   0..* +------------------+
              |                    |<>------| TransportSession |
              +--------------------+        +------------------+

4.5.  CollectingProcess Class

            +-------------------+
            | CollectingProcess |
            +-------------------+
            | name              |       0..* +------------------+
            |                   |<>----------| SctpReceiver     |
            |                   |            +------------------+
            |                   |
            |                   |       0..* +------------------+
            |                   |<>----------| UdpReceiver      |
            |                   |            +------------------+
            |                   |
            |                   |       0..* +------------------+
            |                   |<>----------| TcpReceiver      |
            |                   |            +------------------+
            |                   |
            |                   |       0..* +------------------+
            |                   |<>----------| FileReader       |
            |                   |            +------------------+
            |                   |
            |                   | 0..*  0..* +------------------+
            |                   |----------->| ExportingProcess |
            |                   |            +------------------+
            |                   |
            |                   |       0..* +------------------+
            |                   |<>----------| TransportSession |
            +-------------------+            +------------------+

> points on the same interface, one on the input side and one on the output.

With the current model, there is no problem to configure two OPs on the 
same interface:

      <observationPoint>
        <name>interface ingress</name>
        <interface>
          <ifIndex>3</ifIndex>
          <direction>ingress</direction>
        </interface>
        ...
      </observationPoint>

      <observationPoint>
        <name>interface egress</name>
        <interface>
          <ifIndex>3</ifIndex>
          <direction>egress</direction>
        </interface>
        ...
      </observationPoint>

> of this choice or dropped rather than selected?

No idea.

IMO, it does not make sense to have three counters (packetsDropped, 
packetsSelected, packetsObserved) because two of them are sufficient.

For the moment, I'll keep packetsObserved and packetsDropped like in the 
MIB.

Anyway, as concluded in the other e-mail thread, these counters are not 
necessarily identical to those exported in the PSAMP statistic reports.

> If we're worried about implementation specifics can't they extend the 

Yes, they can.

> I was just thinking about how you'd specify that at least one key was 

Caches of type "immediate" do not have Flow Keys. So, the restrictions 
would not apply to such Caches.

 > if it needs total counters for a permanent cache?

No.

I added a new paragraph at the end of 4.3.1. (CacheLayout Class):

    Note that the use of Information Elements can be restricted to
    certain Cache Modes or to Flow Key or non-key fields.  Such
    restrictions may result from Information Element definitions or from
    device-specific constraints.  According to Section 5, the Monitoring
    Device MUST notify the user if a Cache field cannot be configured
    with the given Information Element.

Ok?

>> How would you configure "primary" and "secondary with a single Exporting 

This issue is on my todo list.

> I guess it would be up to the implementation to control how much history 

But what happens if you remove the primary destination from the 
configuration?

If the transport session history is linked to the destination, it will 
be lost.

If you want to keep a history of Transport Sessions, it is probably 
better to keep the information at a distinct place at it is now.

> a copy of the Destination Class plus some statistics.  This is not true 

There is an important difference between configuration parameter and 
state parameters:

- configuration parameters are not supposed to change dynamically, only
   after the (re)configuration
- state parameters may change at any time

For example, if the user does not configure a source IP address in the 
Destination class, the source IP address in the IPFIX packets changes if 
the outgoing interface changes.

> contain multiple Transport Sessions.

IMO, it is wrong to think that Transport Sessions are configurable.
Transport Sessions represent states which may change dynamically.

If you cut a link, the Transport Session does not exist any more, but 
the configuration of the Exporting Process is still there.

Similarly, if you configure a backup destination, the Transport Session 
won't exist until the backup is needed.

What we can do is having one Transport Session per Destination:

     +------------------------------+
     | SctpDestination              |
     +------------------------------+    0..1 +------------------------+
     | name                         |<>-------| TransportLayerSecurity |
     | exportMemberType = parallel  |         +------------------------+
     | ipfixVersion = 10            |
     | sourceIpAddress[0..*]        |    0..1 +------------------------+
     | destinationIpAddress[1..*]   |<>-------+ TransportSession       |
     | destinationPort = 4739|4740  |         +------------------------+
     | ifName/ifIndex[0..1]         |
     | sendBufferSize {opt.}        |
     | rateLimit[0..1]              |
     | timedReliability = 0         |
     | numberOfStreams {opt.}       |
     | numberOfStreams {opt.}       |
     +------------------------------+

If the Transport Session exists, the TransportSession object exists and 
includes the complete state of the current Transport Session (including 
state parameters which result from configuration, such as 
destinationIpAddress).

In the case of the Collecting Process, we can do the same, allowing 
multiple Transport Sessions per Receiver:

       +--------------------------+
       | SctpReceiver             |
       +--------------------------+    0..1 +------------------------+
       | name                     |<>-------| TransportLayerSecurity |
       | localIpAddress[0..*]     |         +------------------------+
       | localPort = 4739|4740    |
       | maxAllowedStreams {opt.} |    0..* +------------------------+
       +--------------------------+<>-------+ TransportSession       |
                                            +------------------------+

However, historic Transport Session would not be included.

Opinions?

Regards,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 11 Mar 2010 16:00:09 +0000

Hi Gerhard

Gerhard Muenz wrote:
>>> explicitly says that a Metering Process may contain multiple 
>>> Selection Processes and multiple Caches.
>>

OK, at this point I think I'm uncomfortable with the idea that a 
Metering Process can have more than one cache.  What is the reasoning 
behind this decision?

>>> Furthermore, it says that only packets observed at a single 
>>> Observation Point may enter a Selection Process. IMO, this results 
>>> from the PSAMP definitions of Observed Packet Stream and Selection 
>>> Process. Please tell me if this conclusion is wrong.
>>

No, by "feed" I mean input stream.  Either from an Observation point or 
the output of a different Selection Process.

> 4.4.  ExportingProcess Class

OK, this approach seems fine to me.

>>> If the current position of the direction parameter bothers you, we 
>>> can move it.
>>>
>> receive traffic, but the observation point could be on only the send 
>> or receive channel.  i.e. It's not a input only interface, but the 

OK, but my point was always that the interface is not defined by its 
direction.

These two "interfaces" are actually two observation points on the same 
interface:
        <interface>
          <ifIndex>3</ifIndex>
          <direction>ingress</direction>
        </interface>

        <interface>
          <ifIndex>3</ifIndex>

>>> There is another small advantage: The current position of the 
>>> parameter makes it easier to add vendor or implementation-specific 

OK, on the other thread I think I want to go as far as saying they are 
unrelated.  Will reply to that thread shortly.

>>> A the moment, there is no restriction which says that a flow key 
>>> field must exist.
>>>

Really?  Why not say that Caches of an "immediate" type have no 
non-keys?  For example, the counter fields should not be key fields, 
they make no sense as keys.  They also make no sense with the immediate 
cache.  It seems that an immediate cache is made up of entirely key fields.

>>> However, we could add a note that totalCounters are typically used 
>>> for "permanent" and deltaCounters for "timeout".
>>>
>> Perhaps a more general note that not all fields will be appropriate 
>> for all configurations. Is the Exporting Device free to change the fields

Yes, seems reasonable.

>>>             duplicate exporting i.e., all group members identified> 

>>>             as one target for load-balancing. This means that a> 

>>> How would you configure "primary" and "secondary with a single 
>>> Exporting Process parameter?
> 

OK, it seems from a configuration point of view you have three modes. 
Parallel, loadBalancing and fallback.  In fallback mode you try the 
destinations in order until you get a connection.  The mode is a 
property of the Exporting Process and defines how to treat the list of 
destinations.

i.e.

   <Exporter>
     <name>Test 1</name>
     <mode>fallback</mode>
     <destination> ... </destination>
     <destination> ... </destination>
     <destination> ... </destination>
   </Exporter>

So parallel mode sends to all three at once, load balancing takes it in 
turns and fallback treats the first as primary, the second and secondary 
and the third as tertiary.

>>> Currently, TransportSession is a mere copy of 
>>> ipfixTransportSessionTable which exists for both, Exporters and 
>>> Collectors.
>>>
>>> I think that there can be at most one Transport Session per 
>>> Destination and per FileWriter object. At least at the same point in 
>>> time.
>>> However, it may happen that no Transport Session exists, for example 
>>> in the case of a secondary export destination.
>>
>> Well let's imagine and Exporting process with a Primary Destination 
>> and a Secondary Destination.  The Primary Destination starts a 
>> Transport Session, which works for a while and then is lost.  The 
>> Secondary Destination then opens a Transport Session, which works 
>> fine.  The Exporting Process then manages to reconnect with Primary 
>> Destination and opens a new Transport Session with it and closes the 
>> Secondary Destination's Transport Session.
>>
>> I guess it would be up to the implementation to control how much 
>> history was kept.
> 

Well you've changed the config, so I think that would be acceptable.

>>> So, we could move a single set of Transport Session state parameters 
>>> to the Destination and FileWriter classes. The parameters would 
>>> appear or not, depending on whether the Transport Session exists.
>>
>>> In the case of the _Collecting Process_ , we can also move the 
>>> Transport Session state parameters to the _Receiver_ and _FileReader_ 
>>> classes. As you say, there can be multiple Transport Sessions per 
>>> Receiver.
>>
>> the Transport Session Class.  In fact, the Transport Session seems to 
>> be a copy of the Destination Class plus some statistics.  This is not 
>> true of the Receiver class.
> 

OK, that seems fine then.

> What we can do is having one Transport Session per Destination:

I'm happy with that.

Cheers

Andrew

>> Transport Sessions, but would have information about the most recent 
>> one per Destination.  The above example would become:
>>
>>> Transport Session. This container may appear multiple times in the 
>>> case of the Receiver class, and only once in all other classes.
>>>

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 11 Mar 2010 16:21:58 +0000

Hi Gerhard

Gerhard Muenz wrote:
[SNIP]
> We can conclude that the configuration data model does not collide with 

They don't directly collide, but I don't think they directly align 
either.  I am not happy with the idea that a Metering Process has more 
than one cache.  I think one Metering Process should collect one set of 
statistics from a set of observation points.

That doesn't mean that you can't provide Selector configurations that 
feed more than one cache, it just means that more than one Selection 
Process will be created in response to that Selector configuration and 
each Selector Process will only need one set of stats.

> So, please comment the current text:

OK.  Clearing the stats on reconfiguration seems like a good idea. 
Perhaps we should make it clear that the statistics captured on a per 
Selector class basis are not the same as those that get exported.  The 
exported ones are captured on a Selection Sequence basis and can be 
reset whenever the implementation feels the need.

> 4.3.  Cache Class

Again resetting counters on reconfiguration seems reasonable to me, 
however does changing a timeout value really necessitate resetting the 
counters?

> I can imagine what this number is supposed to by, but I'm not sure.

No it isn't.  So if I have a cache with 1000 entries, some might be 
active, some might be free, ready to be populated with a new flow, and I 
suppose some might be considered ready for export, sitting in Limbo 
until the Exporting Process has the resources to send them.  Are these 
the inactive flows?

Cheers

Andrew

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 12 Mar 2010 15:09:25 +0100

Hi Andrew, all,

> Metering Process can have more than one cache.  What is the reasoning 

It should be possible that the output of the same Selection Process 
enters two different Caches.

Example:

           +-> Cache 1 (PSAMP)
OP -> SP -+
           +-> Cache 2 (IPFIX)

It is difficult to say that this corresponds to two MPs.

> the output of a different Selection Process.

Yes, this is what I wanted to say. Therefore, we have "0..1" in -05 (it 
was "0..*" in -04):

                         +------------------+
                         | ObservationPoint |
                         +------------------+
                              0..1 |
                              ==== |
                                   |
                              0..* V
                         +------------------+
                         | SelectionProcess |
                         +------------------+<-+
                              0..* |  0..1 |   | 0..*
                                   |  ==== +---+
                              0..* V
                         +------------------+
                         | Cache            |
                         +------------------+
                              0..* |
                                   |
                              0..* V
                         +------------------+
                         | ExportingProcess |
                         +------------------+

Sorry, if this was not clear from the beginning.

>           <direction>ingress</direction>

Ok. So it seems to you that <direction> belongs to the definition of the 
interface.

If we moved <direction> to the Observation Point, we could remove the 
Interface/Linecard containers because there would be only one parameter 
left:

     +-------------------------------------------------+
     | ObservationPoint                                |
     +-------------------------------------------------+
     | name                                            |
     | observationPointId {readOnly}                   |
     | observationDomainId                             |
     | ifIndex/ifName/entPhysicalIndex/entPhysicalName | 0..*
     | direction                                       |----+
     +-------------------------------------------------+    | 0..*
                                                            V
                                           +--------------------+
                                           | SelectionProcess   |
                                           +--------------------+

Would you agree with this change?

> cache.  It seems that an immediate cache is made up of entirely key fields.

IMO, a Packet Report does not contain any Flow information, so it does 
not make sense to talk about Flow Keys.

In -05, I restricted the <isFlowKey> parameter to Caches generating Flow 
Records because I think that it does not make sense to distinguish Flow 
Key fields and non-key fields if Packet Reports are generated.

> and the third as tertiary.

Yes, this sounds good!

Cheers,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Fri, 12 Mar 2010 15:52:44 +0000

Hi Gerhard

Gerhard Muenz wrote:
>>>> If a Metering Process can have multiple caches and an IPFIX device 
>>>> can have multiple Metering Processes, how do we associate the caches 
>>>> with Metering Processes?  Can this configuration system only be used 
>>>> to configure a single Metering Process within a device?
>>> Processes. This is done on purpose because there is not a single 
>>> correct view.
>>>
>>> Alternatively, Selection Processes and Caches which are not directly 
>>> or indirectly linked to each other can be seen as belonging to 
>>> different Metering Processes.
>>

Is the only argument for these two caches being in the same MP is so 
they can share a selection process?

I say that the cache is the heart of a Metering Process and it should 
only have one!  I realise that it may not be explicitly stated in 
current IPFIX documents, but I think it's implied in the Architecture 
document and is the paradigm in which IPFIX and PSAMP have been developed.

It seems that what you really object to is the PSAMP declaration that a 
selection process belongs to a single Metering Process, but as we've 
seen, the statistics held by a Metering Process need to be kept on a 
per-cache basis.

I don't object to the configuration shorthand of using the same config 
across different Metering Processes, but with this model the stats are 
going to be quite different to those needed for PSAMP.  That's fine though!

>> No, by "feed" I mean input stream.  Either from an Observation point 
>> or the output of a different Selection Process.
> 

This has always been clear... I'm not sure what the original points were 
now.

>>>> Currently you have a 1:1 mapping of OP to interface.  If the 
>>>> direction is moved to the Observation Point then you could have two 
>>>> observation points on the same interface, one on the input side and 
>>>> one on the output.
>>> With the current model, there is no problem to configure two OPs on 
>>> the same interface:
>>>
>>>      <observationPoint>

Yes, this seems fine.

>>>>> packet sources for which the direction parameter has no meaning, 
>>>>> for example:
>>>>>
>>>>> +------------------+
>>>>> | PcapFile         |
>>>>> +------------------+
>>>>> | filename         |
>>>>> +------------------+
>>>>>
> 
> 
>>>>> A Cache implementation may generate records using the specified 
>>>>> field order. So, the fields occur in the same order as specified in 
>>>>> the configuration.
>>>> If we're worried about implementation specifics can't they extend 
>>>> the YANG model?
>>> Yes, they can.
>>>
>>>>> With separate lists for flow key fields and non-key fields, we 
>>>>> cannot arbitrarily mix these two types of fields.
>>>>>
>>>> I was just thinking about how you'd specify that at least one key 
>>>> was required.
>>> Caches of type "immediate" do not have Flow Keys. So, the 
>>> restrictions would not apply to such Caches.
>>
>> they make no sense as keys.  They also make no sense with the 
>> immediate cache.  It seems that an immediate cache is made up of 
>> entirely key fields.
> 

Ah I see.  I guess it makes sense to simply not say anything about 
whether it is a key or not.  So these fields are not keys, but neither 
are they non-keys.  The are simply collected fields.  Certain fields 
only make sense for flows and not for packet reports, which might be 
worth mentioning.

>>>>> This is not enforced. I would not strictly enforce it because 
>>>>> strange configurations might make sense in specific use cases.
>>>> OK.
>>>>
>> So parallel mode sends to all three at once, load balancing takes it 
>> in turns and fallback treats the first as primary, the second and 
>> secondary and the third as tertiary.
> 
> Yes, this sounds good!

Great.

Thanks

Andrew

---

From: Brian Trammell <[email-masked]>
Date: Fri, 12 Mar 2010 17:35:28 +0100

hi Andrew, Gerhard,

Let me step in here quickly, as I can easily see a place for a multicache MP.. the following is something I was asked to do once upon a time:

Suppose we have an MP at an observation point in a security monitoring context. This MP is driving two separate processes: one which filters traffic according to some current set of criteria defining "bad" traffic, then displays in "real time" (~5s delay) this "bad" traffic volume (the fast stream), and one which prefilters and stores full flow data for retrospective analysis (the full stream). The fast stream gets used to drive a display and maybe turn on a blinkenlight somewhere, then it gets thrown away. The full stream has richer data about each flow and gets stored and used by the analysis tools for doing actual work.

It seemed to me at the time that the right way to handle this situation is two caches in the same MP: one with very fast active timeouts (we don't care about termination conditions, we just want to see flow-level volumes), and one with reasonable timeouts (because the analyses are simpler when you don't have to stitch together all the flows from active timeouts.) Everything else -> observation point, selection process, even the EP and CP, are the same.

Or is this two MPs in your reading of the architecture?

Cheers,

Brian

On Mar 12, 2010, at 4:52 PM, Andrew Johnson wrote:

>>>>>> Note: The text has slightly changed after Atsushi's review. Now, it explicitly says that a Metering Process may contain multiple Selection Processes and multiple Caches.
>>>>> If a Metering Process can have multiple caches and an IPFIX device can have multiple Metering Processes, how do we associate the caches with Metering Processes?  Can this configuration system only be used to configure a single Metering Process within a device?
>>>> The model does not group Selection Processes and Caches into Metering Processes. This is done on purpose because there is not a single correct view.
>>>> 
>>>> All Selection Process and all Caches of a Monitoring Device can be regarded as one Metering Process.
>>>> 
>>>> Alternatively, Selection Processes and Caches which are not directly or indirectly linked to each other can be seen as belonging to different Metering Processes.
>>> 
>>> OK, at this point I think I'm uncomfortable with the idea that a Metering Process can have more than one cache.  What is the reasoning behind this decision?
>> It should be possible that the output of the same Selection Process enters two different Caches.
>> Example:
>>          +-> Cache 1 (PSAMP)
>> OP -> SP -+
>>          +-> Cache 2 (IPFIX)
> Is the only argument for these two caches being in the same MP is so they can share a selection process?
> 
> I say that the cache is the heart of a Metering Process and it should only have one!  I realise that it may not be explicitly stated in current IPFIX documents, but I think it's implied in the Architecture document and is the paradigm in which IPFIX and PSAMP have been developed.
> 
> It seems that what you really object to is the PSAMP declaration that a selection process belongs to a single Metering Process, but as we've seen, the statistics held by a Metering Process need to be kept on a per-cache basis.
> 
> I don't object to the configuration shorthand of using the same config across different Metering Processes, but with this model the stats are going to be quite different to those needed for PSAMP.  That's fine though!
> 
> 
>>>>>> Furthermore, it says that only packets observed at a single Observation Point may enter a Selection Process. IMO, this results from the PSAMP definitions of Observed Packet Stream and Selection Process. Please tell me if this conclusion is wrong.
>>>>> A Selection process should only have a single feed, but can have multiple observers.  This results in the number of possible chains (Selection Sequences) being equal to the number of Selectors that directly feed a cache.
>>> No, by "feed" I mean input stream.  Either from an Observation point or the output of a different Selection Process.
>> Yes, this is what I wanted to say. Therefore, we have "0..1" in -05 (it was "0..*" in -04):
>>                        +------------------+
>>                        | ObservationPoint |
>>                        +------------------+
>>                             0..1 |
>>                             ==== |
>>                                  |
>>                             0..* V
>>                        +------------------+
>>                        | SelectionProcess |
>>                        +------------------+<-+
>>                             0..* |  0..1 |   | 0..*
>>                                  |  ==== +---+
>>                             0..* V
>>                        +------------------+
>>                        | Cache            |
>>                        +------------------+
>>                             0..* |
>>                                  |
>>                             0..* V
>>                        +------------------+
>>                        | ExportingProcess |
>>                        +------------------+
> This has always been clear... I'm not sure what the original points were now.
> 
> 
>>>>>> If the current position of the direction parameter bothers you, we can move it.
>>>>>> 
>>>>>> - If it appears in the ObservationPoint class, it looks like the Observation Point is doing some filtering.
>>>>>> - If it appears in the Interface/Linecard class, it looks like the filtering is already done at the interface or linecard.
>>>>>> 
>>>>> I don't see it as filtering, just part of the definition of the Observation Point.  The Interface is most likely able to send and receive traffic, but the observation point could be on only the send or receive channel.  i.e. It's not a input only interface, but the Observation Point could only see input packets.
>>>>> 
>>>>> Currently you have a 1:1 mapping of OP to interface.  If the direction is moved to the Observation Point then you could have two observation points on the same interface, one on the input side and one on the output.
>>>> With the current model, there is no problem to configure two OPs on the same interface:
>>>> 
>>>>     <observationPoint>
>>> OK, but my point was always that the interface is not defined by its direction.
>>> 
>>> These two "interfaces" are actually two observation points on the same interface:
>>>        <interface>
>>>          <ifIndex>3</ifIndex>
>> Ok. So it seems to you that <direction> belongs to the definition of the interface.
>> If we moved <direction> to the Observation Point, we could remove the Interface/Linecard containers because there would be only one parameter left:
>>>>>> There is another small advantage: The current position of the parameter makes it easier to add vendor or implementation-specific packet sources for which the direction parameter has no meaning, for example:
>>>>>> 
>>>>>> +------------------+
>>>>>> | PcapFile         |
>>>>>> +------------------+
>>>>>> | filename         |
>>>>>> +------------------+
>>>>>> 
>>>>>> A the moment, there is no restriction which says that a flow key field must exist.
>>>>>> 
>>>>> A cache without keys will result in a single flow, which seems a bit daft.  I guess there is no pressing reason to add this restriction though.
>>>> >
>>>>>> A Cache implementation may generate records using the specified field order. So, the fields occur in the same order as specified in the configuration.
>>>>> If we're worried about implementation specifics can't they extend the YANG model?
>>>> Yes, they can.
>>>> 
>>>>>> With separate lists for flow key fields and non-key fields, we cannot arbitrarily mix these two types of fields.
>>>>>> 
>>>>> I was just thinking about how you'd specify that at least one key was required.
>>>> Caches of type "immediate" do not have Flow Keys. So, the restrictions would not apply to such Caches.
>>> 
>>> Really?  Why not say that Caches of an "immediate" type have no non-keys?  For example, the counter fields should not be key fields, they make no sense as keys.  They also make no sense with the immediate cache.  It seems that an immediate cache is made up of entirely key fields.
>> IMO, a Packet Report does not contain any Flow information, so it does not make sense to talk about Flow Keys.
>> In -05, I restricted the <isFlowKey> parameter to Caches generating Flow Records because I think that it does not make sense to distinguish Flow Key fields and non-key fields if Packet Reports are generated.
> 
> Ah I see.  I guess it makes sense to simply not say anything about whether it is a key or not.  So these fields are not keys, but neither are they non-keys.  The are simply collected fields.  Certain fields only make sense for flows and not for packet reports, which might be worth mentioning.
> 
> 
>>>>>> This is not enforced. I would not strictly enforce it because strange configurations might make sense in specific use cases.
>>>>> OK.
>>>>> 
>>>>>> However, we could add a note that totalCounters are typically used for "permanent" and deltaCounters for "timeout".
>>>>>> 
>>>>> Perhaps a more general note that not all fields will be appropriate for all configurations. Is the Exporting Device free to change the fields
>>>>>> "parallel" and "loadBalancing" can be configured once with a single ExportingProcess class parameter.
>>>>>> 
>>>>>> How would you configure "primary" and "secondary with a single Exporting Process parameter?
>>> OK, it seems from a configuration point of view you have three modes. Parallel, loadBalancing and fallback.  In fallback mode you try the destinations in order until you get a connection.  The mode is a property of the Exporting Process and defines how to treat the list of destinations.
>>> 
>>> i.e.
>>> 
>>>   <Exporter>
>>>     <name>Test 1</name>
>>>     <mode>fallback</mode>
>>> So parallel mode sends to all three at once, load balancing takes it in turns and fallback treats the first as primary, the second and secondary and the third as tertiary.
>> Yes, this sounds good!
> 
> Great.
> 
> Thanks
> 
> Andrew
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 12 Mar 2010 18:09:55 +0100

Hi Brian,

If it does not matter whether the Selected Packet Stream at the input of 
the two Caches is identical, you could use two distinct Selection 
Processes (with identical configuration). If so, you can see it as two 
different Metering Process, each having a distinct SP and distinct Cache.

If the Selected Packet Stream at the input of the two Caches needs to be 
identical (e.g., because you want to generate Packet Reports and Flow 
Records from the identical subset of packets), then the same SP is 
shared by two Caches.

There is no IPFIX Standard Track document which disallows such a 
configuration. Therefore, the configuration data model allows both 
configurations.

I think that such a configuration can be useful. And I consider it as a 
single MP with two Caches.

Andrew feels uncomfortable with such a configuration because it 
contradicts "the paradigm in which IPFIX and PSAMP have been developed" ;)

Regards,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 15 Mar 2010 11:06:44 +0100

Hi Andrew,

> reset whenever the implementation feels the need.

To make it clearer, I will add an explicit note that the counters in the 
Selector class correspond to the counters in the IPFIX MIB.

Just like in the MIB specification, we do not specify how the counters 
are related to the PSAMP statistics reports.

> however does changing a timeout value really necessitate resetting the 

"RECOMMEND" allows you to keep the counters. I do not think that it 
makes sense to discuss any possible situation where not resetting the 
counters might be appropriate.

> until the Exporting Process has the resources to send them.  Are these 

This counter was adopted from the MIB. I'll send another e-mail to 
discuss this together with other issues.

Regards,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 15 Mar 2010 11:50:22 +0100

Dear all,

I would appreciate some feedback regarding counter values which have 
been adopted from IPFIX MIB.

In the IPFIX MIB, packet or record counters start at certain point in 
time which is specified as "value of sysUpTime".
(sysUpTime is a timer in MIB-2 for "the time (in hundredths of a second) 
since the network management portion of the system was last 
re-initialized.")

What we usually want to know is one of the following:
- absolute time when the packet/record counters were initialized, or
- duration of the current counting period.
To obtain these values, it is necessary to retrieve sysUpTime first from 
MIB-2, and then do some calculation.

In the current configuration data model, we have adopted the counters 
and copied the definitions from the IPFIX MIB.

Since IPFIX-CONFIG is not located in the SNMP world, I propose the 
following changes:

- Counters start with zero (instead of arbitrary values)
   => yang type zero-based-counter64
- Time at which counters start is specified as an absolute time stamp
   (instead of sysUpTime value)
   => yang type time-and-date
   (Alternatively, we could measure the duration of the counting period
   in timeticks or milliseconds.)

Are there any opinions/objections?

Then, I do not understand what the following value in IPFIX MIB is 
supposed to be:

    ipfixMeteringProcessCacheInactiveFlows OBJECT-TYPE
        SYNTAX      Gauge32
        MAX-ACCESS  read-only
        STATUS      current
        DESCRIPTION
            "The number of Flows currently inactive at this cache."
        ::= { ipfixMeteringProcessStatsEntry 2 }

Well, I'm a coauthor of IPFIX MIB but I have to admit that it is not 
clear to me what inactive flows are supposed to be.

Can anybody give an explanation?

Regards,
Gerhard

---

From: Paul Aitken <[email-masked]>
Date: Mon, 15 Mar 2010 13:16:55 +0000

Gerhard,

> Can anybody give an explanation?

The definitions of ipfixMeteringProcessCacheActiveFlows and 
ipfixMeteringProcessCacheInactiveFlows are poor.

However, when a cache is a pre-allocated block of N entries (where each 
entry may represent a flow), then

N = (ipfixMeteringProcessCacheActiveFlows + 
ipfixMeteringProcessCacheInactiveFlows)

where

* ipfixMeteringProcessCacheActiveFlows is the number of flows currently 
in the cache (ie, the number of cache entries currently in use) and

* ipfixMeteringProcessCacheInactiveFlows is the number of further cache 
entries capable of holding new flows.

With a dynamic cache, ipfixMeteringProcessCacheInactiveFlows may 
indicate how much further the cache can grow, or it may be meaningless.

An alternative (and contradictory) explanation is that 
ipfixMeteringProcessCacheInactiveFlows is the number of flows in the 
cache which have exceeded the inactive timeout but have not yet been 
expired. However, given the transient nature of this statistic, it 
doesn't seem to be a very useful one.

Cheers,
P.

-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

---

From: Andrew Johnson <[email-masked]>
Date: Mon, 15 Mar 2010 13:22:32 +0000

> Or is this two MPs in your reading of the architecture?
I would treat then as two different Metering Processes.  I have no 
problem with 2 MPs sharing the same Observation Point and Exporting 
Process.  In your example one cache is receiving a filtered subset of 
traffic, so the Selection Process is not the same.

I don't see the advantage to having more than one cache per Metering 
Process.  I do see a disadvantage in the complexity it brings.  For 
example, if you were to have implemented the above system and the 
Metering Process Statistics Options and/or the Metering Process 
Reliability Statistics, would you count a packet in the 
"ignoredPacketTotalCount" if it successfully entered one cache and not 
the other?

I would say you should only report the loss if both caches failed to 
observe the packet.  From RFC 5470, section 6.4. Reporting Responsibilities:

     From time to time, an IPFIX Device may not be able to observe all
    the packets reaching one of its Observation Points.  This could
    occur if a Metering Process finds itself temporarily short of
    resources... In such situations, the IPFIX Device should attempt to
    count the number of packet losses that have occurred, and report
    them to its Collector(s).

Would it not be more useful to have separate statistics per cache?

Furthermore, the current suggestion is that the configuration draft 
could configure two caches, each populated from completely different 
Observation Points, Selection Process and Exporting to different 
destinations, yet they could be the same Metering Process.  If we 
configure four caches then we would expect between one and four Metering 
Processes to be configured, possibly grouping the caches in one of a 
hundred different ways.

The only reason for this seems to be that it would be convenient to have 
different caches share Selection configuration.  What I don't see is why 
a shared Selector configuration must result in a shared Selection 
Process and therefore a single Metering Process.

Cheers

Andrew

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 15 Mar 2010 15:08:08 +0100

Hi Paul

> entries capable of holding new flows.

Ok, thank you for your interpretation.

Let's assume that it is correct.

So, since we do have

    maxRecords:  Maximum number of Data Records in the Cache.  If the
       Cache stores this number of Data Records, no additional record can
       be created before any of the existing records is removed.  If not
       configured by the user, the Monitoring Device sets this parameter.

(= N) in the configuration data model, we can omit inactiveFlows, right?

Cheers,
Gerhard

---

From: Paul Aitken <[email-masked]>
Date: Mon, 15 Mar 2010 14:37:43 +0000

Gerhard,

>> However, when a cache is a pre-allocated block of N entries (where 
>> each entry may represent a flow), then
>>
>> * ipfixMeteringProcessCacheActiveFlows is the number of flows 
>> currently in the cache (ie, the number of cache entries currently in 
>> use) and
>>
>> * ipfixMeteringProcessCacheInactiveFlows is the number of further 
>> cache entries capable of holding new flows.
> 

I'd like to think so.

> So, since we do have
> 
>    maxRecords:  Maximum number of Data Records in the Cache.

Yuck. Data Records are produced by the exporting process, while a cache 
is maintained by the Metering Process. Data obtained from the cache may 
be transformed, either in whole or in part, and may be combined with 
other data, to form Data Records.

>       If the

Yes. However, in the real world people like to know how full their cache 
is and how much space is left. Providing only "maxRecords" requires them 
to perform an (admittedly simple) calculation, whereas providing 
inactiveFlows makes it immediately obvious.

Is there any reason not to provide all three?

BTW, maxRecords would be the only configurable parameter, if 
configuration were possible.

P.

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 15 Mar 2010 15:52:44 +0100

Paul,

Ok.

> configuration were possible.

Sure.

Regards,
Gerhard

---

From: "Thomas Dietz" <[email-masked]>
Date: Tue, 16 Mar 2010 14:36:40 +0100

Dear Gerhard,

find my replies inline.

-- 
Thomas Dietz
NEC Europe Ltd., NEC Laboratories, Network Research Division, 69115
Heidelberg, Germany

NEC Europe Limited, Registered in England 2832014
Registered Office: NEC House, 1 Victoria Road, London W3 6BL

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Gerhard Muenz
> Sent: Montag, 15. März 2010 11:50
> To: IETF IPFIX Working Group
> Subject: [IPFIX] IPFIX-CONFIG: Questions about counters adopted from
> IPFIXMIB
> 
> 
> Dear all,
> 

Fine with me. You have to make sure though that nobody mixes up the values
from IPFIX CONFIG and IPFIX MIB somehow. If you make it an absolute time
stamp you must make sure somehow that you really have the correct time. This
may be difficult on device reboot and within network environments that don't
give you a timeserver that keeps a correct time.

Anyway I doubt that you profit from making it an absolute time stamp since
counters are only meaningful if you take the difference between two counter
readouts. So you also have to take the difference of the times between last
readout of the counter and current readout to know how long it took to count
the number of whatsoever.

Best Regards,

Thomas

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 16 Mar 2010 15:28:04 +0100

Hi Thomas,

> from IPFIX CONFIG and IPFIX MIB somehow. 

Yes. The time parameters will be called differently (not discontinuity 
time as in the MIB) to avoid confusion.

> If you make it an absolute time stamp you must make sure somehow that 
> you really have the correct time. This

I am aware of this problem. However, in the IPFIX Message header, we have:

    Export Time

       Time, in seconds, since 0000 UTC Jan 1, 1970, at which the IPFIX
       Message Header leaves the Exporter.

So, at least the Exporter must know the absolute time anyway.

> the number of whatsoever.

As we use zero-based counters in IPFIX-CONFIG, you know the counters 
from the indicated point in time.

Thanks and regards,
Gerhard

---

From: "Thomas Dietz" <[email-masked]>
Date: Tue, 16 Mar 2010 15:43:46 +0100

Hi Gerhard,

-- 
Thomas Dietz                 E-mail: [email-masked]
NEC Europe Ltd.              Phone:  +49 6221 4342-128
NEC Laboratories Europe      Fax:    +49 6221 4342-155
Network Research Division
Kurfuersten-Anlage 36
69115 Heidelberg, Germany    http://www.nw.neclab.eu

NEC Europe Limited           Registered in England 2832014
Registered Office: NEC House, 1 Victoria Road, London W3 6BL

> From: Gerhard Muenz [mailto:[email-masked]]
> Sent: Dienstag, 16. März 2010 15:28
> To: Thomas Dietz
> Cc: IETF IPFIX Working Group
> Subject: Re: [IPFIX] IPFIX-CONFIG: Questions about counters adopted
> from IPFIXMIB
> 
> 
> Hi Thomas,
> 
> >> > To obtain these values, it is necessary to retrieve sysUpTime
> first
> >> > from

Be careful here: the discontinuity time is the time when anything went wrong
and you cannot rely on the value of the counter anymore because it may have
missed some packets or whatsoever. This is not the time the counter was last
updated or started!

By the way this is also something you may need?

> from the indicated point in time.

Fine but what do you do if your counter wraps?

Regards,

Thomas

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 16 Mar 2010 16:31:29 +0100

Hi Thomas,

> Fine but what do you do if your counter wraps?

Good points.

My original point was that I would like to get away from sysUpTime 
because it is not sure from where you can get it with Netconf.

Discontinuity times like in the MIB are ok, but then they should contain 
absolute time stamps.

Maybe it is the easiest to only make this little change and keep the 
definitions of the other counters as they are in the MIB.

What do you think?

Regards,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Tue, 16 Mar 2010 21:56:44 +0000

Hi Gerhard

Gerhard Muenz wrote:
[SNIP]
> are related to the PSAMP statistics reports.

I think the current configuration model has a few problems:
   * Implementations seem forced to consider more than one cache part
     of the same Metering Process to be able to use the cascading
     selection configuration.
   * The "SelectionSequenceID" can not be a property of a
     "SelectionProcess Class" because that class does not map to a
     Selection Sequence (i.e. OP, selectors, Metering Process).
   * Selectors can not be re-used fully across Selection Sequences as
     they are done in the PSAMP model (so I don't think you should have
     a "SelectorID" property either).

The PSAMP model is to define selectors as simply configuration.  There 
are no statistics on a per Selector basis.  Instead, each use of a 
selector needs separate statistics.  From RFC 5476, section 6.5.1:

  Example of two Selection Sequences:

    Selection Sequence 7 (Filter->Sampling):
      ingressInterface    5
      selectorId          5 (Filter, match IPV4SourceAddress 192.0.2.1)
      selectorId         10 (Sampler, Random 1 out-of ten)

    Selection Sequence 9 (Sampling->Filtering):
      ingressInterface    5
      selectorId         10 (Sampler, Random 1 out-of ten)
      selectorId          5 (Filter, match IPV4SourceAddress 192.0.2.1)

If we're not going to map the PSAMP model in the configuration model 
then I think we should separate them completely.  In fact, I don't think 
any attempt should be made to directly configure a "Selection Process", 
instead the selection configuration is provided and the Exporting Device 
can sort it out.

This simply avoids the question about whether a Metering Device can (or 
should) have more than one cache.  It also allows greater flexibility in 
configuration because a Selector Configuration can have more than one 
Packet Stream as input.

I think this is a useful concept.  Imagine this relatively common 
configuration, where we want to monitor 10 interfaces, doing 1 in 100 
sampling for each.  Rather than define a new sampling selector for each 
interface, wouldn't it be better to simply do?:

   Selector MY_SAMPLER: sampling 1 in 100 -> cache: MY_CACHE
   OP A -> MY_SAMPLER
   OP B -> MY_SAMPLER
   ...

This is closer to the PSAMP model where you would do something like:

   Selector MY_SAMPLER: sampling 1 in 100

   Selection Sequence: OP A, MY_SAMPLER, MY_CACHE
   Selection Sequence: OP B, MY_SAMPLER, MY_CACHE
   ...

Cheers

Andrew

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 17 Mar 2010 10:58:32 +0100

Hi Andrew,

>      of the same Metering Process to be able to use the cascading

Not at all!

As I have written multiple times here on the list, no implementation is 
expected to support the entire configuration data model. That is just 
impossible. Every implementation will have restriction at one end or the 
other. This is not a specific problem to the Selection Process or Cache 
at all.

YANG allows you to specify restrictions and extensions in a formal way.

If you do not think that this becomes clear from the text in the draft, 
then please provide some better text.

>      Selection Sequence (i.e. OP, selectors, Metering Process).

No. The Selection Process may be composed of multiple Selectors. 
Therefore, the Selection Process represents a Selection Sequence and has 
its Selection Sequence ID as state parameter.

>      a "SelectorID" property either).

You have always stressed that every Selection Sequence has its own 
Selector instances with separate selection statistics and that only a 
Selector configuration template can be shared.

This is reflected in the model!

The data model specifies instances, no configuration templates (again, 
I'm repeating myself). So, if you need the same Selector config in two 
Selection Sequences, you copy the data.

Just like the Selection Sequence ID, the Selector ID is a state 
parameter assigned by the Monitoring Device. It gives the link to the 
information exported via PSAMP protocol. Therefore it is useful to have 
it in the configuration data model.

The Monitoring Device may assign the same Selector ID for two Selector 
instances if the config is identical. There is no problem. Obviously, 
the Monitoring Device needs to compare the configurations of the 
different instances.

>       selectorId          5 (Filter, match IPV4SourceAddress 192.0.2.1)

What if these are two different Observation Domains?

    * Selector ID

       The Selector ID is the unique ID identifying a Primitive Selector.
       The ID is unique within the Observation Domain.

It does not say: unique per PSAMP Device.

Configuration templates for Selectors would introduce another level of 
indirection to the model. This will complicate things a bit.

If the Selector ID would be a unique state parameter of the 
configuration template, then you force Selector IDs to be unique per 
Monitoring Device. Or you must disallow reusing Selector configuration 
templates across Observation Domains. Formally specifying such 
restrictions in YANG results in complicated XPath expresions (or is not 
possible at all).

Though, if you insist on configuring templates for Selectors instead of 
instances, we can change the model accordingly.

However, this decision will have no effect on the question whether one 
Selection Process can feed multiple Caches.

> configuration because a Selector Configuration can have more than one 

The questions whether the Selector configuration is organized with 
reusable configuration templates and whether the same Selection Process 
may feed two Caches are completely orthogonal.

You must not mix up these two questions!

>    Selector MY_SAMPLER: sampling 1 in 100 -> cache: MY_CACHE

What do you mean with "simple"?
You think that you do not have to copy some lines of XML?
I assume that configuration management tools can do this work for you.

>    Selection Sequence: OP B, MY_SAMPLER, MY_CACHE

Do you also request configuration templates for Caches? Or is this a 
single instance?

Regards,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 17 Mar 2010 14:29:50 +0000

Hi Gerhard

One question before I comment inline.  How do you directly attach a
cache to an observation point?  i.e. no sampling or filtering.

Gerhard Muenz wrote:
> 
> Hi Andrew,
> 
> Andrew Johnson wrote:
>> Hi Gerhard
>>
>> Gerhard Muenz wrote:
>> [SNIP]
>>> To make it clearer, I will add an explicit note that the counters in 
>>> the Selector class correspond to the counters in the IPFIX MIB.
>>>
>>> Just like in the MIB specification, we do not specify how the 
>>> counters are related to the PSAMP statistics reports.
>>

You seem to have misunderstood my point.  I will try to clarify it: An
implementation that treats each cache as a separate Metering Process
will be forced to use a limited subset of the configuration model.

To put this another way, the cascading selector model offers a reasonably
concise way to state the configuration and more implementations could use
this model no attempt was made to map the PSAMP model onto the
configuration model.

> its Selection Sequence ID as state parameter.

Since a "Selection Process" can pass a packet stream on to a second
"Selection Process", it does not necessarily represent the full
Selection Sequence and therefore won't necessarily have a Selection
Sequence ID.

In the case where you have:

               +-> SP2 -> cache A
  OP1 -> SP1 --+
               +-> SP3 -> cache B

SP1 is part of two selection sequences.

> Selection Sequences, you copy the data.

Exactly!  Doesn't it strike you as odd that the configuration model
needs every use of a configuration to be repeated?

> it in the configuration data model.

I'm not convinced any run-time state really belongs in the configuration
data model.  Worse, the attempt to make that state available has made it
really difficult to model simple configurations with configuration model!

> the Monitoring Device needs to compare the configurations of the 

And you don't think this is a problem?!  Here is one of my assumptions
about a good configuration model.  It is relatively simple for a Metering
Device to generate it's configuration in the model and then read that
configuration back and produce an internal representation of that
configuration that is isomorphic to the original.

> What if these are two different Observation Domains?

The example is very specifically within the same observation domain.

> It does not say: unique per PSAMP Device.

That doesn't matter if you don't put the selector ID in the configuration
model.

> gIf the Selector ID would be a unique state parameter of the 

For me it will have an impact.  Or rather, I have no objection to a
"Selection Process" feeding multiple caches, what I object to is the
forcing of the sharing of selection state between caches, just because
they share configuration.

>> then I think we should separate them completely.  In fact, I don't 
>> think any attempt should be made to directly configure a "Selection 
>> Process", instead the selection configuration is provided and the 
>> Exporting Device can sort it out.
>>
>> This simply avoids the question about whether a Metering Device can 
>> (or should) have more than one cache.  It also allows greater 
>> flexibility in configuration because a Selector Configuration can have 
>> more than one Packet Stream as input.
> 

They aren't completely orthogonal.  An implementation that considers each
cache a separate Metering Process should not be forced to build an
excessively complicated configuration model.

>> sampling for each.  Rather than define a new sampling selector for 
>> each interface, wouldn't it be better to simply do?:
>>

It's the same cache.

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 17 Mar 2010 15:14:11 +0000

Hi Gerhard

Perhaps we can come to a better understanding if you help me
see how my implementation would use the configuration data
model.  I was originally very optimistic about the configuration
draft because many of the concepts seemed to map very neatly to
how my implementation models its config via the command line.

Would you be willing to map out some real-world configurations
into the configuration data model?  It doesn't have to be the
full YANG output, but I think some text showing the basic
structure would be very useful.

Example 1:

  flow record MY_RECORD
    match ipv4 source address
    match ipv4 destination address
    collect counter bytes

  flow monitor MY_MON
    record MY_RECORD

  interface ethernet 0/0
    ip flow monitor MY_MON input

  interface ethernet 0/1
    ip flow monitor MY_MON input

Typically a customer would configure a monitor on many more
interfaces, perhaps even using a virtual-template or interface
group to add the monitor to hundreds of interfaces, but for now
lets look at just two.

The above configuration directly applies a single monitor
to two interfaces.  In our feature each monitor would be
considered a separate Metering Process, but we don't have to
worry about that here.

Example 2:

  sampler MY_SAMPLER
    mode random 1 out-of 100

  flow record MY_RECORD
    match ipv4 source address
    match ipv4 destination address
    collect counter bytes

  flow monitor MY_MON
    record MY_RECORD

  interface ethernet 0/0
    ip flow monitor MY_MON sampler MY_SAMPLER input

  interface ethernet 0/1
    ip flow monitor MY_MON sampler MY_SAMPLER input

Very similar, except that a we have added a sampling selector.

As I think about these common configurations, what strikes
me as very typical is a reasonably simple configuration that
is then re-used over many interfaces.  It seems to me that
in the current configuration data model that leads to a lot
of repetition.

Cheers

Andrew

---

From: "Thomas Dietz" <[email-masked]>
Date: Fri, 19 Mar 2010 14:58:53 +0100

Hi Gerhard,

> Sent: Dienstag, 16. März 2010 16:31
> To: Thomas Dietz

If you are sure that you can always get the correct absolute time then this
is fine. I am not an Netconf expert, so it's up to you to judge.

> definitions of the other counters as they are in the MIB.

This would be the easiest option from my point of view.

Best Regards,

Thomas

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 19 Mar 2010 16:27:33 +0100

Hi Thomas,

> is fine. I am not an Netconf expert, so it's up to you to judge.

Each IPFIX Message contains the export time in UTC.
So, as IPFIX requires an absolute timestamp, I assume that devices have
a more or less synchronized clock.

> This would be the easiest option from my point of view.

Good.

So, at the end, we keep counter64 (non-zero-based) and have absolute
discontinuity timestamps.

Regards,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 23 Mar 2010 14:17:51 +0100

Hi Andrew,

> cache to an observation point?  i.e. no sampling or filtering.

Citation from -05 of the draft:

  "Every Metering Process contains at least one Selection Process and at
   least one Cache.  If all packets in the Observed Packet Stream are
   selected without any Filtering or Sampling, this corresponds to a
   Selection Process implementing a trivial Selector which selects all
   packets."

Ok.

Though, I would not call the limited configuration model "the PSAMP model".

> SP1 is part of two selection sequences.

http://tools.ietf.org/html/draft-ietf-ipfix-configuration-model-05#section-7.3

No.

Network management is about managed objects which have their own
configuration and state parameters. Two Selection Process represent two
such managed objects which are independent from each other.

> really difficult to model simple configurations with configuration model!

Netconf is a protocol for both monitoring and configuration.
The configuration data model is to cover the monitoring part as well.
Therefore, the MIB parameters have been included.

> configuration that is isomorphic to the original.

The mapping from the configuration data model to the
implementation-dependent representation of the data is most important.
The other direction is secondary.

I do not see a problem because the configuration data model can be
quite easily transformed into the configuration file format of our
IPFIX/PSAMP implementation.

> That doesn't matter if you don't put the selector ID in the configuration

See arguments above why we want to have state parameters in the model.

> they share configuration.

There is no such "forcing". This depends on the implementation.

Some e-mails ago, you told me that you could imagine a Selection Process
which keeps separate counters for all Caches which aggregate the
selected packets.

> excessively complicated configuration model.

I assume that you have one specific implementation in mind, which is
Cisco Flexible NetFlow and its CLI, right?

I know that Cisco decided to introduce configuration templates for
Selectors in the FNF CLI. Yet, this has nothing to do with PSAMP.
There are other implementations which do not use such configuration
templates.

Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 23 Mar 2010 14:49:28 +0100

Hi Andrew,

> structure would be very useful.

There will never be YANG output. Just XML ;)

> considered a separate Metering Process, but we don't have to

This is similar to the example in the draft:
http://tools.ietf.org/html/draft-ietf-ipfix-configuration-model-05#section-7.2

Is it one observation domain? Or two?
The model allows to configure the Observation Domain ID.
Implementations which do not allow this can restrict this parameter.

Please also see the note below the XML part.

  <observationPoint>
    <name>OP at eth0</name>
    <observationDomainId>123</observationDomainId>
    <ifName>eth0</ifName>
    <selectionProcess>All packet selection 1</selectionProcess>
  </observationPoint>

  <observationPoint>
    <name>OP at eth1</name>
    <observationDomainId>123</observationDomainId>
    <ifName>eth1</ifName>
    <selectionProcess>All packet selection 2</selectionProcess>
  </observationPoint>

  <selectionProcess>
    <name>All packet selection 1</name>
    <selector>
      <name>Select all</name>
      <selectAll/>
    </selector>
    <cache>Flow cache</cache>
  </selectionProcess>

  <selectionProcess>
    <name>All packet selection 2</name>
    <selector>
      <name>Select all</name>
      <selectAll/>
    </selector>
    <cache>Flow cache</cache>
  </selectionProcess>

  <cache>
    <name>Flow cache</name>
    <cacheMode>timeout</cacheMode>
    <maxRecords>4096</maxRecords>
    <activeTimeout>5000</activeTimeout>
    <inactiveTimeout>10000</inactiveTimeout>
    <cacheLayout>
      <cacheField>
        <name>Field 1</name>
        <ieName>sourceIPv4Address</ieName>
        <isFlowKey/>
      </cacheField>
      <cacheField>
        <name>Field 2</name>
        <ieName>destinationIPv4Address</ieName>
        <isFlowKey/>
      </cacheField>
      <cacheField>
        <name>Field 8</name>
        <ieName>octetDeltaCount</ieName>
      </cacheField>
    </cacheLayout>
    <exportingProcess>SCTP export with UDP backup</exportingProcess>
  </cache>

  <exportingProcess>
    <name>SCTP export with UDP backup</name>
    ...
  </exportingProcess>

</ipfix>

Note:
I had the idea that we could group multiple interfaces which belong the
same observation domain in one observation point (this possibility is
mentioned in RFC5101):

  <observationPoint>
    <name>OP at eth0 and eth1</name>
    <observationDomainId>123</observationDomainId>
    <ifName>eth0</ifName>
    <ifName>eth1</ifName>
    <selectionProcess>All packet selection 1</selectionProcess>
  </observationPoint>

However, if <direction> was configured, it would affect all interfaces:

  <observationPoint>
    <name>OP at eth0 and eth1</name>
    <observationDomainId>123</observationDomainId>
    <ifName>eth0</ifName>
    <ifName>eth1</ifName>
    <direction>ingress</direction>
    <selectionProcess>All packet selection 1</selectionProcess>
  </observationPoint>

If this makes sense, we can allow multiple occurrences of
ifIndex/ifName/entPhysicalIndex/entPhysicalName per Observation Point.

> Very similar, except that a we have added a sampling selector.

I assume that this configuration results in two independent Selection
Process instances in your system. Otherwise, the selection of packets
from the two interfaces would not be independent from each other.

Then, the configuration is the same as above, but you change the
configuration parameters of the Selection Processes:

      <selector>
         <name>1-out-of-100 sampler</name>
         <sampRandOutOfN>
           <size>1</size>
           <population>100</population>
         </sampRandOutOfN>
       </selector>

> in the current configuration data model that leads to a lot

As you have two independent Selection Process instancees, you have the
same repetition in the running system. So, the configuration reflects
the instances in the system.

Regards,
Gerhard