Subject: Re: [IMRG] Flow sampling (or selection)

From: Maurizio Molina <[email-masked]>
Date: Tue, 01 Jul 2003 20:13:47 +0200

Mark Allman wrote:

>>As far as I know IPFIX deals with reporting per flow information,
>>PSAMP with sampling and reporting per packet information, selected
>>with some criteria.  Has there in any of these groups (or
>>elsewhere) been any discussion about FLOW sampling? I've seen
>>papers about it, e.g.
>>
>>http://www.research.att.com/~duffield/pubs/DLT02-optimal.pdf
>>
>>and I wondered whether this IMRG is the right place to discuss
>>about it.
>>    
>>
>
>Indeed this is the right place to discuss such things.
>
>allman
>
>
>--
>Mark Allman -- BBN/NASA GRC -- http://roland.grc.nasa.gov/~mallman/
>  
>
Actually, what I would try to discuss is if there's the need, in the 
standards, to agree on common procedures for coupling packet sampling 
and flow selection.
To better clarify, I copy hereafter a section that has been introduced 
in the last revision of the PSAMP draft
http://www.ietf.org/internet-drafts/draft-ietf-psamp-sample-tech-02.txt
that I'm co-authoring.
I'm trying to get some reaction about this in the IETF but also some 
discussion in this IMRG would help.
P.S.: is there any planned meeting in Vienna?
Regards,
Maurizio Molina

****************
3.1.2.2.3    Non-Uniform flow State dependent sampling

Another type of sampling that can be classified as Non-Uniform (and, 
possibly, probabilistic) is closely related to the flow concept as 
defined in [QuZC02], and it is only used jointly with a flow monitoring 
function (IPFIX monitoring function). Packets are selected, 
probabilistically or deterministically, dependent on a selection state. 
The point, here, is that the selection state is determined also by the 
state of the flow the packet belongs to and/or by the state of the other 
flows currently being monitored by the associated flow monitoring 
function. This type of sampling is also content dependent because the 
identification of the flow the packet belongs to requires analyzing part 
of the packet content. If the packet is selected, then it is passed as 
an input to the IPFIX monitoring function (this is called "Local Export" 
in [DuGG03].
Selecting the packet depending on the state of a flow cache is useful 
when memory resources of the flow monitoring function are scarce (i.e. 
there's no room to keep all the flows that have been scheduled for 
monitoring). The algorithms leading to the selection decision are not 
subject to standardization, but they may take into account factors like 
flow volume, flow grow rate, flow monitoring priority. An example of 
such an algorithm is described in [EsVa01].

---

From: Nevil Brownlee <[email-masked]>
Date: Wed, 2 Jul 2003 15:10:50 -0700 (PDT)

Hi Maurizio et al:

I haven't seen much discussion of flow sampling, in the sense of
'deciding which flows should be exported' from a flow measuring
device.

There's been some recent work on determining which flows are
the most significant/interesting, e.g.
Cristian Estan's   www.icir.org/vern/imw2001-papers/71.pdf
and my "Dragonflies and Tortoises," IEEE Communications, October 2002

In my work with NeTraMet I've implemented a coarse filter for flows,
you can tell NeTraMet's meter reader to ignore flows which have seen
less than n packets, so you can ignore things like port scans and
concentrate of high-volume or long-running flows.

Mark's description of the IPFIX and PSAMP WGs (below) is about right;
IPFIX has recently published a set of four new drafts, you can find
them via the IETF WG pages.  But IPFIX hasn't considered the question
of flow sampling, and I guess it's out of scope for PSAMP (which looks
at packet sampling).

Cheers, Nevil

-------------------------------------------------------------
   Nevil Brownlee                     Internet Researcher
   Phone: (858) 534 8338                 CAIDA, San Diego

> P.S.: is there any planned meeting in Vienna?

_______________________________________________
IMRG mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/imrg

---

From: Maurizio Molina <[email-masked]>
Date: Thu, 03 Jul 2003 18:21:34 +0200

>of flow sampling, and I guess it's out of scope for PSAMP (which looks
Nevil,
It's true that it's difficult to "position"  such a sampling 
methodology, that it's actually a sort of "packet sampling oriented to 
flow selection", but if  we believe it can play a role in future 
implementations we should try to do it. I think the best is to stimulate 
the discussion in both groups (PSAMP and IPFIX).
As regards PSAMP, it's true that the WG deals with packet sampling, but 
stateful sampling (e.g. sampling based on router state) has always been 
considered. Then, I tried to insert it in the draft this paragraph 
describing it as "flow state dependent sampling". Hope to get some 
reaction on this.
As regards IPFIX, it's true that IPFIX hasn't considered the question of 
flow sampling, but the requirement to do so is still there, also in the 
most recent version of the requirement draft, see 
draft-ietf-ipfix-reqs-10.txt, section 5.2 and 5.3.
On the 21.05.2003 I initiated a discussion on this on the IPFIX ML 
(inside a thread "changes to architecture doc.) getting some negative 
reactions (Ganesh Sadasivan, Carter Bullard) and a positive one (Alex 
Audu). Perhaps it'd be interesting to have your opinion as WG chair.
Regards,
Maurizio