Subject: [IPFIX] Cache configuration issues

From: Gerhard Muenz <[email-masked]>
Date: Mon, 08 Sep 2008 17:53:42 +0200

Dear all,

IPFIX devices offer more and more configuration possibilities regarding
the fields contained in the generated Flow Records. In the IPFIX
configuration draft, this is called Cache Layout.

The flexible combination of Flow Key fields and non-Flow Key fields in a
Cache Layout may lead to situations where a more precise specification
of the Flow Record generation and update functions is necessary.

It is best to explain these issues with an example of a Cache Layout:
* Flow Key fields:
  - src IP address
  - dst IP address
  - src transport port
  - dst transport port
* non-Flow Key field:
  - tcp control bits
  - tcp sequence number
  - packet count
  - timestamps

Note that there is no protocol field!

Four packets enter the Cache:
(A) TCP 1.2.3.4:100 => 4.3.2.1:101 (SYN)
(B) UDP 1.2.3.4:100 => 4.3.2.1:101
(C) ICMP 1.2.3.4 => 4.3.2.1
(D) UDP 1.2.3.4:200 => 4.3.2.1:201

Issue 1:
--------
What happens if any of the Flow Key fields is not available in the packet?

In the example, packet C does not have src and dst transport port.
There are two ways to handle this situation:

1) Ignore/drop the packet, which means that the Cache Layout implicitly
performs a kind of packet filtering.

2) Create a Flow Record without the missing ports fields and make sure
that packets with port fields are not accounted in this Flow Record.

The advantage of 1) is that all generated Flow Records contain all
configured fields. However, in order to generate ICMP Flow Records, we
would need to configure a second Cache without port fields. In addition,
we need a packet filter in front of the ICMP Cache in order to avoid
that TCP/UDP packets are From [email-masked]  Mon Sep  8 08:53:46 2008
Date: Mon, 08 Sep 2008 17:53:42 +0200
From: Gerhard Muenz <[email-masked]>
To: IPFIX WG <[email-masked]>
Subject: [IPFIX] Cache configuration issues
Sender: [email-masked]

Dear all,

IPFIX devices offer more and more configuration possibilities regarding
the fields contained in the generated Flow Records. In the IPFIX
configuration draft, this is called Cache Layout.

The flexible combination of Flow Key fields and non-Flow Key fields in a
Cache Layout may lead to situations where a more precise specification
of the Flow Record generation and update functions is necessary.

It is best to explain these issues with an example of a Cache Layout:
* Flow Key fields:
  - src IP address
  - dst IP address
  - src transport port
  - dst transport port
* non-Flow Key field:
  - tcp control bits
  - tcp sequence number
  - packet count
  - timestamps

Note that there is no protocol field!

Four packets enter the Cache:
(A) TCP 1.2.3.4:100 => 4.3.2.1:101 (SYN)
(B) UDP 1.2.3.4:100 => 4.3.2.1:101
(C) ICMP 1.2.3.4 => 4.3.2.1
(D) UDP 1.2.3.4:200 => 4.3.2.1:201

Issue 1:
--------
What happens if any of the Flow Key fields is not available in the packet?

In the example, packet C does not have src and dst transport port.
There are two ways to handle this situation:

1) Ignore/drop the packet, which means that the Cache Layout implicitly
performs a kind of packet filtering.

2) Create a Flow Record without the missing ports fields and make sure
that packets with port fields are not accounted in this Flow Record.

The advantage of 1) is that all generated Flow Records contain all
configured fields. However, in order to generate ICMP Flow Records, we
would need to configure a second Cache without port fields. In addition,
we need a packet filter in front of the ICMP Cache in order to avoid
that TCP/UDP packets araccounted as well. You see that with 1), even
simple configurations may become quite complex.

The advantage of 2) is that we can account all packets in a single Cache
(with a single Cache Layout). However, the resulting Flow Records are
unpredictable.

My proposed solution is to support both Cache behaviors. This does not
mean that devices need to implement both functionalities. Yet, the
configuration data model foresees that devices may behave in one way or
the other. A parameter would specify the Cache behavior in the
configuration file.

Issue 2:
--------
What about non-Flow Key fields which are not available in all packets of
a flow?

In the example, packet A and packet B share the same Flow Keys, so they
are accounted as a single Flow. However, TCP control bits and TCP
sequence number do not appear in packet B.

According to RFC5102, the TCP control bits field ORs all control bits
seen in the packets of the flow. As packet B does not have any control
bit field, we would treat it as a TCP packet with all control bits set
to zero.

According to RFC5102, the TCP sequence number is taken from the first
packet of the Flow. If packet A is the first packet, we would take the
sequence number from this packet. However, if packet B is the first
packet and packet A the second, there is no such field in the first
packet of the Flow.

There are three ways to handle this situation:

1) Non-Flow Key fields which are not available in the first packet of a
Flow, are omitted in the resulting Flow Record.

2) Non-Flow Key fields which are not available in the first packet of a
Flow, are set to zero in the resulting Flow Record.

3) Non-Flow Key fields are set to the value of the first packet in the
Flow which contains this field, or omitted if none of the packets
contains this field.

I propose to go for 1) because 2) may lead to misinterpretations and 3)
is not quite in-line with RFC 5102.

Finally, packet D, which does not share the Flow Keys with any other
packet, would be accounted in a separate Record which contains a TCP
control bits field with value zero. Yet, the TCP sequence number field
is not included since this field is not available in the first packet of
the Flow.

You are invited to comment and to make different suggestions.
Otherwise, I'll adopt the proposed caches behaviors in the configuration
draft.

Regards,
Gerhard

-- 
Dipl.-Ing. Gerhard Münz
Computer Networks and Internet
Wilhelm Schickard Institute for Computer Science
University of Tuebingen
Sand 13 (Room B309), D-72076 Tuebingen, Germany
Phone:  +49 7071 29-70534       Fax: +49 7071 29-5220
E-mail: [email-masked]
WWW:    http://net.informatik.uni-tuebingen.de/~muenz

_______________________________________________
IPFIX mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ipfix

e accounted as well. You see that with 1), even
simple configurations may become quite complex.

The advantage of 2) is that we can account all packets in a single Cache
(with a single Cache Layout). However, the resulting Flow Records are
unpredictable.

My proposed solution is to support both Cache behaviors. This does not
mean that devices need to implement both functionalities. Yet, the
configuration data model foresees that devices may behave in one way or
the other. A parameter would specify the Cache behavior in the
configuration file.

Issue 2:
--------
What about non-Flow Key fields which are not available in all packets of
a flow?

In the example, packet A and packet B share the same Flow Keys, so they
are accounted as a single Flow. However, TCP control bits and TCP
sequence number do not appear in packet B.

According to RFC5102, the TCP control bits field ORs all control bits
seen in the packets of the flow. As packet B does not have any control
bit field, we would treat it as a TCP packet with all control bits set
to zero.

According to RFC5102, the TCP sequence number is taken from the first
packet of the Flow. If packet A is the first packet, we would take the
sequence number from this packet. However, if packet B is the first
packet and packet A the second, there is no such field in the first
packet of the Flow.

There are three ways to handle this situation:

1) Non-Flow Key fields which are not available in the first packet of a
Flow, are omitted in the resulting Flow Record.

2) Non-Flow Key fields which are not available in the first packet of a
Flow, are set to zero in the resulting Flow Record.

3) Non-Flow Key fields are set to the value of the first packet in the
Flow which contains this field, or omitted if none of the packets
contains this field.

I propose to go for 1) because 2) may lead to misinterpretations and 3)
is not quite in-line with RFC 5102.

Finally, packet D, which does not share the Flow Keys with any other
packet, would be accounted in a separate Record which contains a TCP
control bits field with value zero. Yet, the TCP sequence number field
is not included since this field is not available in the first packet of
the Flow.

You are invited to comment and to make different suggestions.
Otherwise, I'll adopt the proposed caches behaviors in the configuration
draft.

Regards,
Gerhard

-- 
Dipl.-Ing. Gerhard Münz
Computer Networks and Internet
Wilhelm Schickard Institute for Computer Science
University of Tuebingen
Sand 13 (Room B309), D-72076 Tuebingen, Germany
Phone:  +49 7071 29-70534       Fax: +49 7071 29-5220
E-mail: [email-masked]
WWW:    http://net.informatik.uni-tuebingen.de/~muenz

_______________________________________________
IPFIX mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ipfix

---

From: Andreas Bourges <[email-masked]>
Date: Tue, 9 Sep 2008 08:55:58 +0200

Hi Gerhard,

On Monday 08 September 2008, Gerhard Muenz wrote:
> that TCP/UDP packets are accounted as well. You see that with 1), even

Have you ever thought of daisy-chaining flow-cache processing? You could have 
several cache-groups and within each cache group you could have an ordered 
list of caches to be processed. Once a flow is processed, the caches could be 
checked in their configured order for matching key/value fields. If the flow 
does not match the first cache, try the second cache and so on...

In terms of flexibility this would be a good solution to categorize flows into 
different caches (according to the flow-type, e.g.tcp/udp/icmp/other) without 
explicitly filtering unwanted flows for a cache...

Regards,

Andreas

_______________________________________________
IPFIX mailing list
[email-masked]
https://www.iFrom [email-masked]  Mon Sep  8 23:56:07 2008
From: Andreas Bourges <[email-masked]>
Organization: IsarNet AG
To: [email-masked]
Date: Tue, 9 Sep 2008 08:55:58 +0200
Subject: Re: [IPFIX] Cache configuration issues
Reply-To: [email-masked]
Sender: [email-masked]

Hi Gerhard,

On Monday 08 September 2008, Gerhard Muenz wrote:
> that TCP/UDP packets are accounted as well. You see that with 1), even

Have you ever thought of daisy-chaining flow-cache processing? You could have 
several cache-groups and within each cache group you could have an ordered 
list of caches to be processed. Once a flow is processed, the caches could be 
checked in their configured order for matching key/value fields. If the flow 
does not match the first cache, try the second cache and so on...

In terms of flexibility this would be a good solution to categorize flows into 
different caches (according to the flow-type, e.g.tcp/udp/icmp/other) without 
explicitly filtering unwanted flows for a cache...

Regards,

Andreas

_______________________________________________
IPFIX mailing list
[email-masked]
https://www.ietetf.org/mailman/listinfo/ipfix

f.org/mailman/listinfo/ipfix

---

From: Christoph Sommer <[email-masked]>
Date: Tue, 09 Sep 2008 09:24:08 +0200

Hi Andreas,

Andreas Bourges wrote:
>> we would need to configure a second Cache without port fields. In addition,
>> that TCP/UDP packets are accounted as well.
> 
> Have you ever thought of daisy-chaining flow-cache processing? [...] If the flow

Yes, this is discussed in draft-dressler-ipfix-aggregation, more
specifically in:

Section 6.2. ("Multiple Aggregation Rules")
<http://tools.ietf.org/html/draft-dressler-ipfix-aggregation-05#page-8>

It does complicate the export of information about the aggregation
process, though.

btw (just in case you were wondering what stance the draft takes
regarding the other issue): It's currently mandating the presence of all
of a cache's configured fields:

Section 6.1 ("Single Aggregation Rule")
<http://tools.ietf.org/html/draft-dressler-ipfix-aggregation-05#page-7>

Regards,

  Christoph

---

From: Christoph Sommer <[email-masked]>
Date: Tue, 09 Sep 2008 09:24:08 +0200

Hi Andreas,

Regards,

  Christoph

---

From: Andreas Bourges <[email-masked]>
Date: Tue, 9 Sep 2008 09:36:02 +0200

Hi Christoph,

On Tuesday 09 September 2008, Christoph Sommer wrote:
> Andreas Bourges wrote:
> > Have you ever thought of daisy-chaining flow-cache processing? [...] If
> > the flow does not match the first cache, try the second cache and so
> > on...

:)) - yes, that's exactly what I thought of. Should have read you draft till 
the end (which I actually wanted to do...), but forgot about it during my 
hollidays ;-)

> It does complicate the export of information about the aggregation

Yes it does. And implementation at the observing device is likely to be more 
complex, too... 

Regards,

Andreas

---

From: Andreas Bourges <[email-masked]>
Date: Tue, 9 Sep 2008 09:36:02 +0200

Hi Christoph,

Regards,

Andreas