Subject: Re: single point of measurement problem: router traffic meter

From: Nevil Brownlee <[email-masked]>
Date: Mon, 11 Oct 1999 10:18:38 +1300 (New Zealand Daylight Time)

Hello Juergen:

re: your note dated 20 Sep 99

> the architecture as defined by RFC 2063 (and also by the current ID)
> does not support this task in a convenient way.
> 
> Our main problem is the point of measurement. It is defined to be
> a single interface of the router and not the IP forwarding engine.

Although the RTFM Architecture draft describes a meter as observing
traffic as a single point - the METERING POINT - it also says (as an 
example of how to implement a meter)

" A meter could be implemented in various ways, including:

  .......

  - A packet-forwarding device such as a router or switch.  This is
    similar to (b) except that every received packet should also be
    forwarded, usually on a different interface. "

To take this a step further, the Architecture includes among the address 
attributes 

" - Addresses for the flow's source and destination.  These include
    addresses and masks for various network layers (extracted from the
    packet header), and the identity of the interface on which the
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    packet was observed. "

Further, the attributes include both source- and dest- PeerType 
attributes, allowing the architecture to handle gateways, where packets 
came in on one interface as (say) AppleTalk, and went out another 
interface as IP packets!  Quite a while back, at one of the WG meetings, 
we disucssed whether this 'gateway' ability was useful; consensus was to 
keep it in the architecture.

> Solution 2:
>   Move the point of measurement to the IP forwarding engine and
>   run a single meter at this point.
> 
>   This solves the described problem, but it has several disadvantages:
> 
>   a. It is not compatible to the traffic flow measurement
>      architecture. (Solution 1 is compatible.)

Not a problem, see comments above :-)

>   b. The new "point" of measurement is rather large and may
>      be extended at runtime, e.g. by starting a multicast
>      daemon. The implementation of the Meter must somehow be
>      done "in parallel" with the entire IP forwarding engine.
>      This can be done for commercial router devices where all
>      software updates are checked by the manufacturer before
>      they are delivered to the customers, but it seems to be
>      hard for e.g. a Linux system used as router.
> 
>   c. The forwarding engine does a lot more with packets than
>      the interfaces. E.g. a packet may be dropped, because
>      of missing buffer space or another unability to deliver it.
> 
>      Also a single arriving packet may be sent out in several
>      copies to different interfaces, when mutlicast is supported.
>      Currently, the traffic flow archtitecture does not discuss
>      these issues, because they do not occur, if the point of
>      measurement is at a single interface.
> 
>      If handles well, this inconsistency with the architecture
>      might be turned to an advantage (at least in our application),
>      because you can measure these actions in the forwarding engine
>      and reflect them by new attributes, e.g. the number of dropped
>      packets.
> 
> Despite the list of disadvantages, we have implemented a prototype
> of solution 2 based on the NeTraMet distribution. We had to modify
> the Meter MIB a bit, e.g. supporting several output interfaces
> for a flow and adding COUNT attributes for the number of dropped
> packets and the number of used destination interfaces.

There is already one implementation of this - Cisco NetFlow provides the 
source and destination interfaces for the flows it sees, and NetFlowMet 
makes these accessible to rulesets.

I've had "implement NeTraMet within a modified gated" in mind as a 
research project for some time, but haven't got round to it yet.  I'm 
delighted that you've done this; I hope that sharing the experiences with 
this will encourage vendors of switches and routers to implement the 
Meter MIB in their devices.  Real soon now!

A comment on flows through a device with multiple interfaces, for a 
switch on a LAN each host will only appear on a single interface 
(although that could change with time if a link fails).  That means that 
a flow between two hosts, A and B will almost always have the same pair 
of interfaces, say A-1-2-B.  In this case the interfaces don't add much 
information to the flow.  But it could be very useful to look at flows 
between interfaces so as to spot the busiest paths through a switch.

For a router, however, packets could go to and from B via two (or more) 
interfaces, say 2 and 3.  So now there are four possible paths -
1->2 + 1<-2, 1->3 + 1<-3, 1->2 + 1<-3 and 1->3 + 1<-2.  Observing these 
four flows could be useful, say in a situation where load balancing was 
causing part of the A-B traffic to take two different paths. 

On the question of multicast, there's nothing in the architecture, 
although we know we need something to cover it!

One thought has occured to me - how about a DestMulticastCount attribute,
which indicated how many interfaces a multicast packet was being sent out 
over?  A zero value would indicate a unicast flow, higher values would 
indicate the coverage (within a switch) of the multicast.

Comments and suggestions please ...

Cheers, Nevil

PS: This is definitely a topic we could talk about in person at the 
    IETF meeting in Washington DC!

+---------------------------------------------------------------------+
| Nevil Brownlee                     Director, Technology Development |
| Phone: +64 9 373 7599 x8941        ITSS, The University of Auckland |
|   FAX: +64 9 373 7021      Private Bag 92019, Auckland, New Zealand |
+---------------------------------------------------------------------P

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 21 Oct 1999 14:38:22 +0200

Hello Neville,

Thank you for your deatiled reply. I am glad to learn that what we
implemented conforms to the RTFM architecture. However, there
are some open issues left, which I think the description of the
architecture should clarify.

1. input != output

  If I take the IP forwarding engine of a router as METERING POINT,
  then it is not clear to me, how to handle differences in packet
  input and packet output.

  A simple case is dropped packets. They enter the forwarding engine,
  but never leave it. Shall they be counted in the packet and byte
  counter or not?
  The obvious solution to this problem is introducing further COUNT
  attributes DataToPDUsDropped and DataFromPDUsDropped and maybe
  also DataToOctetsDropped and DataFromOctetsDropped.

  More complex is the problem of multicast packets. A single entering
  packet may be sent out in several copies via different interfaces.
  Shall the Meter count one packet or the number of outgoing packets?

2. ambiguous output interface

  An  problem related to counting multicast packets is matching
  multicast packets. Lets assume we want to measure in a ruleset
  how many multicast packets are leaving the router per interface.
  The naive rule definition would match multicast packets on any
  outInterface and make the outInterface part of the flow key,
  such that we get an entry in the flow table for each interface
  where multicast packets leave our router.

  How shall this be realized in the PME?

  We could send the multicast packet once to the PME with a list
  of outInterfaces attached. But then we would have to modify the
  PME, because currently it matches only outInterfaces to single
  attributes and not to lists.

  Another solution would be to send one instance of the multicast
  packet to the PME for each used outInterface. This would not
  require a change of the PME and appears to be a rather 'natural'
  solution for multicast.
  A slight problem I see with this solution occurs when you measure
  the following traffic:
  - flow 1: all packets entering at interf A and leaving at interf B
  - flow 2: all packets entering at interf A and leaving at interf C
  - flow 3: all packets entering at interf A (in another rule set)
  Note that with our multicast solution, the sum of the number of
  packets in flows 1 and 2 can be greater than the number of packets
  in flow 3. This may lead to confusion.

The description of these problems leads to the major point I want
to make in this message:

I think both problems (input != output and ambiguous output interface)
can be solved if we specify in the architecture, that we count only
packets _LEAVING_ the metering point and not (necessarily) packets
_ARRIVING_ at the metering.

With this specification,
  1. the introduction of counter for dropped packets and bytes is
     not necessary, because they are not to be counted.
  2. The second, more natural solution for multicast would be implied
     and the possible confusion about this solution (as described above)
     would be avoided.

    Juergen
-- 
Juergen Quittek    [email-masked]   http://www.ccrle.nec.de
NEC Europe Ltd., C&C Research Laboratories  Tel: +49 30 254230-19
Hardenbergplatz 2, 10623 Berlin, Germany    Fax: +49 30 254230-99

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 21 Oct 1999 15:10:42 +0200

Nevil Brownlee wrote:
 ...
> Comments and suggestions please ...

We implemented such an attribute in our prototype. We followed the
same approach you suggest, but we ended up with TWO new attributes.

The DestMulicastCount has to be a COUNT attribute, because it may
change its value over time. This implies that you cannot match on it
as you cannot match on the DataToPDUs. At least it would not make
much sense to have the DestMulticastCount in your flow key, because
it would use (or create) a different entry in the flow table whenever
the number of subscribers of your multicast flow changes.

However, it would be very helpful to have a matchable attribute
which can be used to distinguish multicast packets from other
traffic. Checking all possible multicast addresses is quiet
circumstantial. Therfore, we added the attribute IsMulticast
which has a length of one bit and which has the value 1 for
multicast packets and 0 otherwise.

    Juergen