Subject: [IPFIX] [Fwd: questions regarding rfc 5101 implementation]

From: Nevil Brownlee <[email-masked]>
Date: Wed, 20 Jan 2010 11:08:09 +1300

Forwarded as requested :-)

-------- Original Message --------
Subject: questions regarding rfc 5101 implementation
Date: Wed, 20 Jan 2010 09:21:52 +1300
From: Allwyn Carvalho <[email-masked]>
To: [email-masked] <[email-masked]>
CC: Nevil Brownlee <[email-masked]>

[I have been trying to send this for some time, but my email does not 
make it
through.  Neville, can you help?.  Thanks in advance.]

I have a few of questions regarding RFC 5101 implementation which I 
don't see
addressed in RFC 5153 (or at least the answer is not clear to me).

Consider a Selection Process that has been configured to capture the 
following
Information Elements (key and non-key):

    * sourceIPv4Address
    * destinationIPv4Address
    * protocolIdentifier
    * sourceTransportPort
    * destinationTransportPort
    * bgpSourceAsNumber
    * bgpDestinationAsNumber
    * flowStartMilliseconds
    * flowEndMilliseconds
    * octetDeltaCount
    * packetDeltaCount

Clearly, all these IEs are not applicable to all flows.  For example, 
source and
destination transport port are not applicable to ICMP flows.  And bgp 
source
and/or destination AS number are not applicable if source/destination 
address
was not learned via BGP.

So, the first question is regarding the use of templates.  Is an 
implementation
supposed to maintain multiple templates for the different types of 
traffic?  Or,
is it supposed to have a single template and to use "dummy" values when 
a field
is not applicable?  I realize this could be implementation specific, but is
there a guideline?

Assuming the guideline is to use multiple templates, this leads me to my 
next
question: is necessary for the Collecting process to know that the flow 
records
(with different template ids) all come from the same Cache, and if so 
how will
the Exporting Process ensure that this info is provided?

I believe the answer to this could be to use the same Observation Domain 
ID in
the Packet Header for all these flow records.  But this leads me to the 
final
question.  Is the Observation Domain ID tied to a Cache, or is it tied 
to a set
of Observation Points?

In particular, here is the dilemma.  Suppose it is required to aggregate 
the
same packet streams onto multiple aggregation caches.  Consider for 
example the
following requirement:

Cache 1:

    * sourceIPv4Address (key)
    * destinationIPv4Address (key)
    * flowStartMilliseconds (non-key)
    * flowEndMilliseconds (non-key)
    * octetDeltaCount (non-key)
    * packetDeltaCount (non-key)

Cache 2:

    * sourceIPv4Address (key)
    * destinationIPv4Address (key)
    * bgpSourceAsNumber (key)
    * bgpDestinationAsNumber (key)
    * flowStartMilliseconds (non-key)
    * flowEndMilliseconds (non-key)
    * octetDeltaCount (non-key)
    * packetDeltaCount (non-key)

The requirement is that every selected packet must get counted in each 
of the
two caches.

Now, when flow records are reported to the Collecting Process, should we be
using the same or different Observation Domain ID for each Cache?

The reason for the dilemma is that if we use a common Observation Domain 
ID when
reporting each cache, it will not be possible for the Collecting Process 
to know
which cache it came from.  However, from RFC 5101, the Observation 
Domain ID is
based only on the set of Observation Points, and it does not say what to 
do if
there are multiple aggregation caches within the same Observation 
Domain.  Here
is the text from RFC 5101:

     Observation Domain

        An Observation Domain is the largest set of Observation Points for
        which Flow information can be aggregated by a Metering Process.
        For example, a router line card may be an Observation Domain if it
        is composed of several interfaces, each of which is an Observation
        Point.  In the IPFIX Message it generates, the Observation Domain
        includes its Observation Domain ID, which is unique per Exporting
        Process.  That way, the Collecting Process can identify the
        specific Observation Domain from the Exporter that sends the IPFIX
        Messages.  Every Observation Point is associated with an
        Observation Domain.  It is RECOMMENDED that Observation Domain IDs
        also be unique per IPFIX Device.

Final question: will the authors of RFC 5153 consider adding some 
language in
the guidelines to address some of these issues?

Thanks.
Allwyn.

-- 
---------------------------------------------------------------------
  Nevil Brownlee                    Computer Science Department | ITS
  Phone: +64 9 373 7599 x88941             The University of Auckland
  FAX: +64 9 373 7453   Private Bag 92019, Auckland 1142, New Zealand

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 20 Jan 2010 09:33:38 +0100

Hi Allwyn,

Some quick answers inline.

> Information Elements (key and non-key):

The Selection Process does not capture any IEs nor does it define key 
and non-key fields. It's the Metering Process, or - as we define it in
http://tools.ietf.org/html/draft-ietf-ipfix-configuration-model-04#section-4.3.1
- the Cache (as part of the Metering Process).

> is not applicable?  I realize this could be implementation specific, but is

That is not specified in the standard, so it is implementation specific. 
There is no guideline.

Cisco uses dummy values in NetFlow.V9 (e.g. zero port-numbers for ICMP 
flows).

The problem is that you may have a large number of templates, which can 
be inefficient. On the other hand, such invalid dummy values do not 
exist for all IEs.

If we use a single template, the solution will be to have a new IE which 
defines a bitvector, and 1s in the bit vector indicate "valid" fields in 
the record, 0s "invalid" fields (similar to flowKeyIndicator).

The idea with this new IE is not mine, Paul Aitken mentioned it to me 
some time ago.

> The requirement is that every selected packet must get counted in each 

That is true (and the expected behavior).

> using the same or different Observation Domain ID for each Cache?

The ODID specifies where the packet was observed, not the Cache! So, it 
will be the same.

> which cache it came from.  

If you know that you have configured two parallel caches, you also know 
that you will receive two flow records for every observed flow. In this 
case, you do not need any further information.

PSAMP specifies how to report the selection sequence, but this does not 
include the cache.

Reporting internals of IPFIX metering processes has not been 
standardized in similar detail as in PSAMP. However, you can export an 
option template to associate flow records with the metering process ID 
or alternatively standardize a new cacheId IE to specify a cache.

It's true that the standard is not complete with this regard. In 
practice, you will probably know the configuration of your exporters. 
And you probably won't configure parallel caches as you propose. So, you 
usually won't have this problem.

Regards,
Gerhard

 > However, from RFC 5101, the Observation
> Domain ID is

-- 
Dipl.-Ing. Gerhard Münz
Chair for Network Architectures and Services (I8)
Technische Universität München - Department of Informatics
Boltzmannstr. 3, 85748 Garching bei München, Germany
Phone:  +49 89 289-18008       Fax: +49 89 289-18033
E-mail: [email-masked]    WWW: http://www.net.in.tum.de/~muenz

---

From: Paul Aitken <[email-masked]>
Date: Wed, 20 Jan 2010 11:19:06 +0000

Gerhard, Allwyn,

>> implementation supposed to maintain multiple templates for the
>> different types of traffic? Or, is it supposed to have a single
>> template and to use "dummy" values when a field is not applicable?
>> I realize this could be implementation specific, but is there a
>> guideline?
>
> That is not specified in the standard, so it is implementation
> specific. There is no guideline.
>
> Cisco uses dummy values in NetFlow.V9 (e.g. zero port-numbers for
> ICMP flows).
>
> The problem is that you may have a large number of templates, which
> can be inefficient. On the other hand, such invalid dummy values do
> not exist for all IEs.
>
> If we use a single template, the solution will be to have a new IE
> which defines a bitvector, and 1s in the bit vector indicate "valid"
> fields in the record, 0s "invalid" fields (similar to
> flowKeyIndicator).
>
> The idea with this new IE is not mine, Paul Aitken mentioned it to
> me some time ago.

I'm planning to write a new draft to express this idea. We may use
structured data rather than a bitfield to express the invalid
(unobserved) fields, since there would be issues with mediators that
don't understand the bitfield. The idea is not mine; Andrew Johnson
mentioned it to me some time ago.

>> Assuming the guideline is to use multiple templates, this leads me
>> to my next question: is necessary for the Collecting process to
>> know that the flow records (with different template ids) all come
>> from the same Cache,

I don't think so.

>> and if so how will the Exporting Process ensure that this info is
>> provided?
>>
>> I believe the answer to this could be to use the same Observation
>> Domain ID in the Packet Header for all these flow records. But
>> this leads me to the final question. Is the Observation Domain ID
>> tied to a Cache, or is it tied to a set of Observation Points?

The OD ID is related to the Observation Points. From RFC 5101:

    Observation Domain

       Point.

A single cache could potentially observe data from multiple Metering
Processes (eg, from multiple line cards). However, the key thing from
the above definition is that these observations are not aggregatable -
so the MP would have to keep each line card's observations separate from
other line cards, eg by using the line card ID or OD ID as a key field.

Clearly there's little advantage to using a single cache for this
purpose; one cache per OD makes more sense. In this implementation, the
OD ID and cache become synonymous.

>> The reason for the dilemma is that if we use a common Observation
>> Domain ID when reporting each cache, it will not be possible for
>> the Collecting Process to know which cache it came from.

Why is it necessary to know?

>> However, from RFC 5101, the Observation Domain ID is based only on
>> the set of Observation Points, and it does not say what to do if
>> Domain.

If each cache is observing the same OD, then they would all report the
same OD ID.

Cheers,
P.

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 20 Jan 2010 12:08:06 +0000

Paul Aitken wrote:
[SNIP]
> other line cards, eg by using the line card ID or OD ID as a key field.

A single cache that observes data from Multiple Metering Process doesn't
make sense to me, unless it is some sort of mediation device which would
mean it was a Metering Process in its own right, with a new Observation
Domain that was the sum of the parts.

>From RFC 5470, "The typical functions of a Metering Process may include
... Maintaining database(s) of all the Flow Records from an Observation
Domain."

I think it's fair to say that database and cache are synonymous here.

If you mean that a single "cache" is shared between multiple Metering
Processes simply as a memory management model and each Metering Process
is still in control of its own separate set of data, then logically
each Metering Process still has its own database / cache.

regards

Andrew

---

From: Brian Trammell <[email-masked]>
Date: Wed, 20 Jan 2010 13:36:29 +0100

> mentioned it to me some time ago.

I remember a conversation with Andrew, too, maybe in San Diego?

Basically, "yes" to this thread, "but".

I'll note that template flexibility is one of the big advantages of IPFIX, and that I don't really buy the multiple template inefficiency argument. In the worst case export, with multiple templates representing the same underlying data model but omitting nulls or substituting reduced- for full-length encoding IEs, the template switch overhead is 4 octets per record (plus, unless you're using UDP (which you shouldn't be) with a ridiculous template refresh time, epsilon octets per record for additional template export). This worst case represents each record in its own data set, and the average case is probably somewhat better than that, especially depending on implementation and deployment specifics such as maximum buffer delay time (which leads to messages being exported without being MTU-sized), path MTU, and per-template buffering. 

A null bitfield introduces for all but the most trivial templates a 2- to 4-octet overhead, and only allows null omission, not concurrent export of full- and reduced-length encoded IEs. There are advantages to the FLE/RLE switch: for a common example, consider octet and packet count IEs, which are by default 8 octets, but in almost all flows (by flow count, not by octet volume) can be represented with 2 or 4. This saves 8 octets per record (16 per biflow) when allowing 4- and 8- octets encodings, which saves 4 (12) net octets even in the pathological case, where every other record in the stream is RLE. Of course, once you're using this multiple-template export trick for RLS, you can apply the same for null omission, with no implementation pain.

In any case, in general we do need something better than simple dummy values. They're okay for counters (where a 0 and a "dummy" value have exactly the same semantics) and identifiers where the number space has an explicit null value, probably okay in cases (such as ports) where additional information (such as a protocolIdentifier value set to a portless transport) allows disambiguation, and only maybe okay in cases where protocol-specific values (think 0.0.0.0 address for DHCP) or nonsensical ones (e.g. 0 UTC 1 January 1970) can be disambiguated with special-casing or human intervention. Unless and until null bitfields are available, multiple-template export is what we have.

Cheers,

Brian

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 20 Jan 2010 13:50:30 +0100

Brian,

Brian Trammell wrote:
>>>>  different types of traffic? Or, is it supposed to have a
>>>> single template and to use "dummy" values when a field is not
>>>> applicable? I realize this could be implementation specific,
>>>> but is there a guideline?
>>> The problem is that you may have a large number of templates,
>>> which can be inefficient. On the other hand, such invalid dummy
>>> values do not exist for all IEs.
>>> 
>>> If we use a single template, the solution will be to have a new
>>> IE which defines a bitvector, and 1s in the bit vector indicate
>>> "valid" fields in the record, 0s "invalid" fields (similar to 
>>> flowKeyIndicator).
>>> 
>>> The idea with this new IE is not mine, Paul Aitken mentioned it
>>> to me some time ago.
>> (unobserved) fields, since there would be issues with mediators
>> that don't understand the bitfield. The idea is not mine; Andrew
>> Johnson mentioned it to me some time ago.
> 
> I'll note that template flexibility is one of the big advantages of
> IPFIX, and that I don't really buy the multiple template inefficiency
> argument. In the worst case export, with multiple templates
> representing the same underlying data model but omitting nulls or
> substituting reduced- for full-length encoding IEs, the template
> switch overhead is 4 octets per record (plus, unless you're using UDP
> (which you shouldn't be) with a ridiculous template refresh time,
> epsilon octets per record for additional template export). This worst
> case represents each record in its own data set, and the average case
> is probably somewhat better than that, especially depending on
> implementation and deployment specifics such as maximum buffer delay
> time (which leads to messages being exported without being
> MTU-sized), path MTU, and per-template buffering.

I think the argument against using many templates is that the exporting 
process implementation becomes more complex.
With "invalid" fields, it can use a single template per cache/metering 
process. Without, it has to decide among multiple templates.

Note that in general, I'm also not in favor of invalid fields in data 
records.

Regards,
Gerhard

> A null bitfield introduces for all but the most trivial templates a
> 2- to 4-octet overhead, and only allows null omission, not concurrent
> export of full- and reduced-length encoded IEs. There are advantages
> to the FLE/RLE switch: for a common example, consider octet and
> packet count IEs, which are by default 8 octets, but in almost all
> flows (by flow count, not by octet volume) can be represented with 2
> or 4. This saves 8 octets per record (16 per biflow) when allowing 4-
> and 8- octets encodings, which saves 4 (12) net octets even in the
> pathological case, where every other record in the stream is RLE. Of
> course, once you're using this multiple-template export trick for
> RLS, you can apply the same for null omission, with no implementation
> pain.
> 
> In any case, in general we do need something better than simple dummy
> values. They're okay for counters (where a 0 and a "dummy" value have
> exactly the same semantics) and identifiers where the number space
> has an explicit null value, probably okay in cases (such as ports)
> where additional information (such as a protocolIdentifier value set
> to a portless transport) allows disambiguation, and only maybe okay
> in cases where protocol-specific values (think 0.0.0.0 address for
> DHCP) or nonsensical ones (e.g. 0 UTC 1 January 1970) can be
> disambiguated with special-casing or human intervention. Unless and
> until null bitfields are available, multiple-template export is what
> we have.
> 
> Cheers,
> 
> Brian _______________________________________________ IPFIX mailing
> list [email-masked] https://www.ietf.org/mailman/listinfo/ipfix

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 20 Jan 2010 13:53:30 +0100

Hi Paul, Andrew,

> Domain that was the sum of the parts.

I'm a bit confused by the terms you use. The cache is part of the 
Metering Process. Do you mean that a single cache should/should not 
account packets from multiple selection processes?

Regards,
Gerhard

> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

---

From: Brian Trammell <[email-masked]>
Date: Wed, 20 Jan 2010 14:07:55 +0100

hi Gerhard,

On Jan 20, 2010, at 1:50 PM, Gerhard Muenz wrote:

>>>> That is not specified in the standard, so it is implementation specific. There is no guideline.
>>>> "valid" fields in the record, 0s "invalid" fields (similar to flowKeyIndicator).
>>> I'm planning to write a new draft to express this idea. We may use structured data rather than a bitfield to express the invalid (unobserved) fields, since there would be issues with mediators
> I think the argument against using many templates is that the exporting process implementation becomes more complex.
> With "invalid" fields, it can use a single template per cache/metering process. Without, it has to decide among multiple templates.

This is a point. In stripped-down EPs, the difference between one and two templates is a big deal. In slightly more complicated ones, the difference between two and three is trivial. It's been a while since I looked at the code for this, but YAF does it with maybe ten lines of C, looking at the record and choosing an appropriate template ID (which, coincidentally, is calculated as a bitmap of which features the record has...)

But yes, the null-bitmap field is a good idea, if _only_ for the simple-EP case.

> Note that in general, I'm also not in favor of invalid fields in data records.

this, I already knew. :)

Cheers,

Brian

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 20 Jan 2010 13:52:00 +0000

> But yes, the null-bitmap field is a good idea, if _only_ for the simple-EP case.

No bitmaps!  I'm really don't think they're a good idea.  Since they can only
be interpreted within the context of the record they are currently in, then each
stage of flow handling that can manipulate that context has to understand these
fields or, worse than simply losing the information, the fields become misleading.

Look at the way the "flowKeyIndicator" field is supposed to be used.  An option
template is sent that shows you how to interpret a different template!

For example, imagine a mediation device that accepts records from several sources,
perhaps does some filtering, and then adds a "source ID" to each flow before
passing them on to a central collector.  We can't expect this simple device to
understand every new "bitmask" field that gets invented.

Consider a simple collector that stores the flows in a big database, for later
interpretation.  The only data that is lost in the process is the order the
received fields are in.  This is enough to make the bitmask fields useless.

Cheers

Andrew

---

From: Atsushi Kobayashi <[email-masked]>
Date: Thu, 21 Jan 2010 23:27:46 +0900

Hi Andrew, and all,

On Wed, 20 Jan 2010 13:52:00 +0000
Andrew Johnson <[email-masked]> wrote:

[snip]

> > But yes, the null-bitmap field is a good idea, if _only_ for the simple-EP case.

I agree to export the information whatever bitmap or structured-data.

There have already been issues in our flow measurement operation.
For example, when the value of origin ASN is 0, we cannot see if it
comes from null AS path in BGP routedb or if the destination address
cannot be found in BGP routedb that is bogon. It is a big difference.

> received fields are in.  This is enough to make the bitmask fields useless.

I think the bit field for "invalid/valid" depends on its own flow. Thus,
this information needs to be included in a flow record. I assume your
proposal is to export the information as optional separated data. 
But, I cannot come up with your detailed idea. Could you explain more?

Regards,
Atsushi

> >> Note that in general, I'm also not in favor of invalid fields in data records.

--- 
Atsushi KOBAYASHI  <[email-masked]>
NTT Information Sharing Platform Lab.
tel:+81-(0)422-59-3978 fax:+81-(0)422-59-5637

---

From: Atsushi Kobayashi <[email-masked]>
Date: Thu, 21 Jan 2010 14:30:21 +0900

Hi Andrew, and all,

[snip]

Regards,
Atsushi

---