Subject: ICMPv6: Rate Limiting Configuration Per-Node or Per-Interfaces

From: [email-masked]
Date: Wed, 18 Aug 2004 14:51:49 -0500

I think the discussion about the ICMPv6 rate limiting is
going in all directions and we are not going anywhere 
near the consensus.  This email is my effort to bring
some consensus on the issues.

I think everyone agrees that per-interface configuration
would be a perfect solution and will provide a fine grained
control to the user.  Is there anyone who disagrees with
this ? (Pekka ??)

The argument that I see from people (Pekka, Me, Havard)
is that is it really worth the efforts to implement/
configure it or per-node configuration is good enough
operationally.  This is why we are suggesting it to
be a MAY.

Now instead of going into all the reasons/arguments
and deviating the discussions from one direction to
the other, could people please send their preferences ?

So the question is:

In your opinion (no reasoning please), the rate limiting
configuration per-interface in the ICMPv6 spec should be a

1) SHOULD
2) MAY
3) Any of them is fine for you.

My choice is 3.

Regards
Mukesh
PS: If I got anything wrong in this mail, please direct
your flaims to me in unicast messages :)

--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]
Administrative Requests: https://www1.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------

---

From: Francis Dupont <[email-masked]>
Date: Thu, 19 Aug 2004 00:25:32 +0200

In your previous mail you wrote:

   1) SHOULD
   2) MAY
   
=> 2

[email-masked]

---

From: Alex Conta <[email-masked]>
Date: Wed, 18 Aug 2004 20:13:34 -0400

SHOULD choice #1

[email-masked] wrote:
> {..]In your opinion (no reasoning please), the rate limiting

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Thu, 19 Aug 2004 14:40:59 +0900

>>>>> On Wed, 18 Aug 2004 14:51:49 -0500, 
>>>>> [email-masked] said:

> My choice is 3.

I prefer 3, then 2.  I would not take option 1.

					JINMEI, Tatuya
					Communication Platform Lab.
					Corporate R&D Center, Toshiba Corp.
					[email-masked]

---

From: JINMEI Tatuya / 神明達哉<[email-masked]>
Date: Thu, 19 Aug 2004 15:22:24 +0900

>>>>> On Thu, 19 Aug 2004 14:40:59 +0900, 
>>>>> JINMEI Tatuya <[email-masked]> said:

> I prefer 3, then 2.  I would not take option 1.

Oops, sorry, I was not careful enough...I mistakenly understood that
option 3 was "neither of them (i.e., no RFC2119 keyword)".

Please let me correct my previous answer to:

My choice is 2.

					JINMEI, Tatuya

---

From: Radhakrishnan Suryanarayanan <[email-masked]>
Date: Thu, 19 Aug 2004 14:46:26 +0530

Choice is :
1) MAY

----- Original Message ----- 
From: <[email-masked]>
To: <[email-masked]>
Sent: Thursday, August 19, 2004 1:21 AM
Subject: ICMPv6: Rate Limiting Configuration Per-Node or Per-Interfaces

So the question is:

1) SHOULD
2) MAY

My choice is 3.

Regards
Mukesh

---

From: Pekka Savola <[email-masked]>
Date: Thu, 19 Aug 2004 13:51:25 +0300 (EEST)

> control to the user.  Is there anyone who disagrees with

My objection to this stems from the fact that an implementation which
would like to do something like that is likely doing something wrong
in the first place (or something which is implementation-specific in
any case, and doesn't need "IETF blessing" for their approach in any
case), and I slightly disagree with per-interface configuration.  

Obviously, implementors can provide whatever mechanisms they want,
even those not mentioned by the specification, so even if we don't
mention it at all, it would be OK to implement it.

Rather than first figuring out whether to use MAY or SHOULD it might 
be better to try to figure how to reword the section etc. to be more 
specific about the recommended method and where it applies to.

This could possibly be achived by changing "send" to "originate"  
elsewhere in the spec, and reword (f) to something like:

====
    (f) Finally, an IPv6 node MUST limit the rate of ICMPv6 error 
        messages it originates in order to limit the processing at the
        node and bandwidth and forwarding costs incurred on the 
        network by originating ICMPv6 error messages.  This
        situation may occur when a source sending a stream of erroneous
        packets fails to heed the resulting ICMPv6 error messages.

        Rate-limiting of forwarded ICMP messages is out of scope of this
        specification.
                                                                                
        A recommended method for implementing the rate-limiting function
        is a token bucket, limiting the average rate of transmission to
        N packets/second, but allowing up to B error messages
        to be originated in a burst, as long as the long-term average
        is not exceeded.
                                                                                
        Rate-limiting mechanisms which cannot cope with bursty traffic
        (e.g., traceroute) are not recommended; for example a simple
        timer-based implementation, allowing an error message every T
        milliseconds (even with low values for T), is not reasonable.
                                                                                
        The rate-limiting parameters SHOULD be configurable.  In the
        case of a token-bucket implementation, the best defaults depend
        on where the implementation is expected to be deployed (e.g., a
        high-end router vs. an embedded host).  For example, in a
        small/mid -sized device, the possible defaults could be B=10,
        N=10/s.
====

 - make it clearer that the justification is also to limit the 
processing on the node
 - add note of non-scope of generic rate-limiting (this is also so 
obvious that 2nd paragraph could possibly be removed now)
 - just specify packets/second in the recommended token bucket
rate-limiter to avoid confusion with interface-specific issues.  Note
that as this is just a recommended *EXAMPLE*, it's fully compliant
with the spec to provide another kind of rate-limiter as well, hence
bandwidth-based measurements don't belong here (and haven't been even
really implemented -- all the vendors I know of do pps + burst)!

> 3) Any of them is fine for you.

MAY.

Or just leave it out completely (prefable, as with Jinmei's initial 
message), but that wasn't the option. :)

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: David Malone <[email-masked]>
Date: Thu, 19 Aug 2004 11:35:45 +0100

> 3) Any of them is fine for you.

MAY...

	David.

---

From: Radhakrishnan Suryanarayanan <[email-masked]>
Date: Fri, 20 Aug 2004 09:48:00 +0530

Sorry for the typo error
choice is 2) MAY
:-)

---

From: Brian Haberman <[email-masked]>
Date: Fri, 20 Aug 2004 10:03:57 -0400

<WG Chair hat off>

>>control to the user.  Is there anyone who disagrees with

I find it very useful.

> specific about the recommended method and where it applies to.

I agree that re-wording will help with clarity...

>         Rate-limiting of forwarded ICMP messages is out of scope of this

I don't agree with this last addition.  Rate-limiting transit ICMP
traffic is useful in several ways.  One key area for rate-limiting
in routers is when ICMP messages are sent in response to multicast
data packets.

Putting this in a separate document seems like a waste of time.  It
would be a section in the ICMP spec as it relates to rate-limiting.

If that section is added and the clarification between "send" and
"originate" is made, I feel that is sufficient.

>>3) Any of them is fine for you.

SHOULD for routers since there are many interface speeds.
MAY for hosts.

Regards,
Brian

---

From: Pekka Savola <[email-masked]>
Date: Fri, 20 Aug 2004 21:59:14 +0300 (EEST)

> would be a section in the ICMP spec as it relates to rate-limiting.

But multicast data packets are already rate-limited by the senders,
the same way as unicast!  Or do you want to set more aggressive limits
to multicast packets?

I don't think anyone disagrees that rate-limiting transiting ICMP 
packets might be beneficial especially under certain situations.

The issue just is that it's an *operational* decision, not a 
*protocol* decision.  Every operator is likely to have entirely 
different set of requirements.  I know a large number of them which 
*do not* want to rate-limit anything -- they just want to forward the 
packets, and that's it.  Likewise, even if some would not object to 
preferring it, there could be no agreement on the exact values, and 
having to configure the values on all your routers (or disable the 
limiting if you don't want it) takes a lot of time.

And what does rate-limiting in transit help?  People can just send
ICMP echo replys/request or other informational ICMP messages instead
which wouldn't be rate-limited: it doesn't seem to protect from
anything.

The fact is, that there are dozens of protocols out there that are not
rate-limited *in transit* by the protocol specification -- in fact, I
can't recall even one such protocol!  I.e., rate-limiting transit
traffic is an operational procedure, not a protocol procedure.  IMHO,
protocol specs have no business specifying what happens to the packets
when they're forwarded: the packets are just plain IP.

---

From: Alex Conta <[email-masked]>
Date: Mon, 23 Aug 2004 13:53:11 -0400

Pekka Savola wrote:
> in the first place 

Traffic management in its very nature is linked to interfaces: its
functional components are dispersed among other protocol functional
components in a router's functional topology that start at an "ingress"
interface and ends at an "egress" interface. There is plenty of
documentation on this in IETF and elsewhere.

A separation of ICMP rate limiting, which is a form of traffic 
management, from interfaces goes against this very nature.

>(or something which is implementation-specific in

With a protocol purist perspective - nothing wrong - there is the
suggestion that ICMPv6 should bless or address only ICMP rate limiting 
that is implemented inside the ICMP protocol engine.

The reality is that ICMP protocol does not have a rate limiting
mechanism of its own - ICMP message headers do not have any rate
limiting fields.

The ICMP rate limiting is simply a set of operational requirements,
supported by "some" implementation mechanism: token bucket is suggested.

Consequently, placing the "token bucket" anywhere in the processing 
graph: from policing detected errored packets (A) to traffic management 
of 'egress' traffic on interface (B), as long as it achieves the 
operational requirements is valid, and legal, as far as ICMPv6 spec is 
concerned.

One place versus the other is selected based on implementation 
constraints, and operational purposes.

The latter - (B) above - is a choice for minimizing the number of 
vertices in the node's internal functional topology, combined with 
maximizing the operational capabilities of the node: it controls all 
"send" traffic, as well as "send" ICMP traffic (both "originated" and 
"in transit"). It is a common (cost effective) choice in modern router 
implementations, and possible choice in software (host) implementations 
(LINUX is known, but there may be others).

As configuring is the same (CLIs hide implementation details), and the 
effect is the desired ICMP rate limiting, the IETF spec should not be 
preferential in its blessing.

> [...]

Steve Deering, and Bob Hinden who last edited this part of the text in 
the past did a wonderful job, approved by the WG, and IESG: in a very 
succinct portion of text, they were able to capture the essential.

> elsewhere in the spec, and reword (f) to something like:

The judgment of how good a spec is, should not be based on the ability
of the spec to serve/support one side of a debate - lack of support 
triggered discussion of "send".

The judgment is the abundant existence of architectures and (host and
router) implementations of the traffic management model (above) and the 
abundant existence of hardware/software components to build it, and 
ultimately their use in operational networks. Based on that:

I am opposed to any rewording of the first 3 paragraphs  of
Section 2.4 (f) of the ICMPv6 specifications, that would exclude "in 
transit" ICMP traffic from the ICMPv6 scope, because it can break
current implementation compliance and being within the ICMPv6 scope,
with a particular effect on routers and hardware/software
implementations in network processors, which are components for routers.
It can also break operational capabilities of routers, and the reliance 
of network operators on these capabilities, which are  important for 
managing traffic in networks.

This includes opposition to the suggestion to use packets per second as
the ONLY metric for ICMPv6 rate limiting.

Packets/second is not an accurate metric: packets are not equal in size.
Different packet sizes have different impact on processing resources,
and thus different impact on implementations.

Bits/second is an accurate and widely recognized standard metric,
expressing link bandwidth/speed, or interface bandwidth/speed. It can be
easily used as it is, or translated into internal resource metrics (for
instance for memory, or internal bus) and should be the recommended metric.

>>3) Any of them is fine for you.

My amended preference is:

If an addition is made to current paragraph 4 of Section 2.4 (f):
1) SHOULD for routers, 2) MAY for hosts

If the addition is not made as the line above, leave the text unchanged.

Regards,
Alex

---

From: Pekka Savola <[email-masked]>
Date: Tue, 24 Aug 2004 00:17:59 +0300 (EEST)

> management, from interfaces goes against this very nature.

I don't disagree with traffic management being important, but the 
point is:

*Managing traffic ICMP traffic which is not originated from the node
in question is out of scope of this specification.*

Therefore if you need such knobs for traffic management, that's fine,
but out of scope of this spec, and don't belong here.
 
> mechanism of its own - ICMP message headers do not have any rate

Neither does UDP, TCP (in the sense that you can always send the 
packets even if the rate at which actual data is sent is adjusted 
based on congestion), or a hundred other protocols.  Why should ICMP 
have somethnig like that then?
 
> (LINUX is known, but there may be others).

As already discussed to death, "in transit" is an entirely different
rate-limiting concept with different requirements, assumptions, and
the place to specify it, than "origination by a protocol"  
rate-limiting.
 
> preferential in its blessing.

For all implementations I know of (Juniper, Cisco, Linux, BSD, ...)  
out there the configuration of ICMP origination rate-limiters and
generic traffic management (e.g., ICMP limiter for transit traffic) is
*not* the same.
 
> managing traffic in networks.

No, this wouldn't make any such implementations non-compliant. They
would just implement something more, something that's out of scope
from the spec.  One could call that a bonus feature if you want :-).

Believe me, this particular interpretation is not required for 
managing traffic in the networks.  Generic rate-limiting features 
already required, those which are not dependent on ICMPv6, and need to 
be provided in any case.  This is no place to do that work.

So, your proposal seems to be looking at a way not to implement
separate ICMP origination rate-limiters, just make do with the generic
rate-limiters which have been set-up appropriately.  That's not
something we as the IETF want to recommend (IMHO), but as long as it
does the job, it's still compliant with the spec AFAICS.

> instance for memory, or internal bus) and should be the recommended metric.

Pps the is only real useful metric in the context of traffic
origination.  It's the number of packets (not their size) which cause
processing, and it's the number of packets (not their size) which
freeze the forwarding (e.g., in flow-based architectures) when you get
hit by a flood of a million or two packets per second.

But again, this wouldn't make an implementation non-compliant, as 
token bucket w/ pps is just a recommended example.  And there are good 
reasons why it's recommended, but implementations can diverge from 
that if they really want.. the IETF just doesn't need to recommend 
that.

---

From: Fred Templin <[email-masked]>
Date: Mon, 23 Aug 2004 17:27:23 -0700

>3) Any of them is fine for you.

Bandwidth-based per-interface rate limiting is:

  1) SHOULD

In other words, leave current text of [RFC2463], section 2.4 (f) as-is.

Fred
[email-masked]

---

From: Fred Templin <[email-masked]>
Date: Mon, 23 Aug 2004 17:30:50 -0700

Hello,

With Nokia hat off, I would like to announce a proposal for IPng
called: "IPvLX - IP with virtual Link eXtension". See:

  http://www.ietf.org/internet-drafts/draft-templin-ipvlx-00.txt

IPvLX uses IPv4 as an L2 protocol for network traversal and IPv6
as an L3 addressing protocol. It inserts an L2.5 "shim" layer between
the IPv4 header and upper layer protocol headers, and establishes
virtual links between peers using IPv6 neighbor discovery messages
for control plane signalling.

IPvLX uses a special IPv4 header construction that encodes VCI/VPI
information for virtual circuit switching in IPvLX Gateways. This
allows virtual link extension between peers located in different
enterprises/sites.

IPvLX is compatible with legacy IPv4 hosts and routers, and requires
no "flag day" changes in the Internet. It provides an adaptation layer
similar to AAL5 for efficient multiaccess segment sizing based on
path MTU.

IPvLX is a unified package combining familiar mechanisms having
various degrees of operational experience, and presents a potential
"one size fits all" alternative for IPng. Please respond on the list
with any comments/questions.

Fred L. Templin
[email-masked]

P.S. An appendix on multiprotocol support that did not make
   it in time for the I-D submission is attached below.
Appendix B.  IPvLX as a Layer 2.5 Shim

   Section 4 presents IPvLX as boh an IPv4 encapsulation method and an
   IPv6 header compression method that is tightly coupled with IPv6 as
   both the addressing (control plane) mechanism and data exchange (data
   plane) protocol. IPvLX could instead be decoupled from IPv6 and
   specified as a distinct Layer 2.5 "shim" between IPv4 as the L2
   protocol and any IP protocol in the IANA "Protocol Numbers" registry
   as the L3 protocol. In this model, IPv6 would serve as the control
   plane mechanism to set up virtual links between peers that can
   support data plane exchanges using any IP protocol.

   The IPvLX Flow Header (see: section 4.4.2) could be trivially
   extended to support this multiprotocol operation by adding a 1 byte
   "Protocol" field the same as for the IPv4 "Protocol" ([RFC0791],
   section 3.1) and IPv6 "Next Header" ([RFC2460], section 3) fields as
   follows:

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Version|    Protocol   |           Flow Label                  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Version         4-bit version number - value TBA by IANA.

   Protocol        8-bit field; indicates the IANA IP protocol number
                   exactly as for the IPv4 "Protocol" and IPv6 "Next
                   Header" fields.

   Flow Label      20-bit flow label [RFC3697].

   With this extended format, the IPvLX Flow Header becomes 4 bytes in
   length instead of just 3 and would occur immediately following the
   20-byte IPv4 header in all IPvLX packets (see: section 4.2.1) as
   follows:

                  +-------------------------------+
                  |     IPv4 Header (20 bytes)    |
                  |                               |
                  | RF = DF = 1; MF = [0|1]       |
                  | 1 <= SEGID <= 31              |
                  | 23 < Length <= (20 + SEGSIZE) |
                  |                               |
                  +-----------------------+-------+
                  |  IPvLX Flow Header (4 bytes)  |
                  +-------------------------------+
                  |                               |
                  ~ Up to (SEGSIZE - 4) PDU bytes ~
                  |                               |
                  +-------------------------------+

                    Format for all IPvLX packets

   The extended format has the disadvantage of consuming 1 extra byte
   per IPvLX packet, and reducing the maximum IPvLX PDU Payload (see:
   section 4.2) to (2^16 - (32 * 4) - 1) = 65407 bytes. These
   disadvantages are outweighed by numerous advantages, including:

       -  IPvLX gateways can process all IPvLX packets in fast-path
          since the 4-byte IPvLX Flow Header occurs in all packets.

       -  Awkward, special-case encodings that lead to slow path
          handling (e.g., the IPvLX Fragment Header - see: section
          4.4.3) are no longer needed.

       -  The 4-byte IPvLX Flow Header following the 20-byte IPv4 header
          provides a natural 8-byte alignment needed for any protocol
          headers that follow (e.g., IPv6 header extensions).

       -  The "Protocol" field allows multiprotocol operation on the
          data plane with IPv6 as the control plane mechanism for
          virtual link setup.
--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]
Administrative Requests: https://www1.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------

---

From: Greg Daley <[email-masked]>
Date: Tue, 24 Aug 2004 13:06:34 +1000

Hi Fred,

(to the IPv6 wg only)

It may be worth mentioning, before you get a huge amount
of IPv6-WG ire flamed down upon you that the IPVLX BoF
was discussing some related ideas at the past IETF 60.

It was not in all proposals that IPv4 were being used
as transport, nor were the v4/v6 implications being
discussed.

The concept has some different applications than just
replacing IPv6 (;-) and these are being discussed on the
[email-masked] mailing list.

Greg Daley

Fred Templin wrote:
> ------------------------------------------------------------------------
> 
> ------------------------------------------------------------------------
>

---

From: Fred Templin <[email-masked]>
Date: Tue, 24 Aug 2004 10:01:09 -0700 (PDT)

Greg,

Thanks for the message. To clarify, however, IPvLX
*does not* propose to replace IPv6.

The IPvLX proposal incorporates IPv6 mechanisms as
core architectural elements.

Regards - Fred
[email-masked]

---

From: Stig Venaas <[email-masked]>
Date: Thu, 26 Aug 2004 16:12:31 +0200

> 3) Any of them is fine for you.

2 MAY

Stig