Subject: RE: can you be constructive? Re: pls read the specs Re: Section 2.4, item (f) of draft-ietf-ipngwg-icmp-v3-04.txt

From: [email-masked]
Date: Wed, 18 Aug 2004 12:14:39 -0500

Alex,

> It is amusing for you to say that. As a co-author of the ICMPv6 RFC 
> 2463, and RFC 1885, work which started in 1994, I have a good 
> insight on 
> the meaning of the text, and the meaning of the standard. 
> However I am 
> not interested in giving lectures, and am not interested in 
> proving how 
> wrong you are in interpreting the ICMP specs. I am rather 
> interested in 
> a quick positive outcome and a win-win situation.

So are you saying that the ICMPv6 spec is talking about 
rate limiting the ICMPv6 traffic that is being forwarded 
by a router ?

If that is the case, Pekka is not alone.  I also get the 
perception that it is talking about generating the ICMPv6 
messages and NOT forwarding ICMPv6 messages.  As far as I 
understand, rate limiting the ICMPv6 traffic being forwarded 
by a router is out of the scope of this spec.

> Are you ready to be constructive?

Please pardon me for saying this but it seems to me that 
you are not being constructive by changing the subjects 
line to those provocative sentences.

> If your concern, your vision and your target is "controlling internal 
> ICMP message generation" and the mechanism to achieve that is 
> the token 
> bucket, I do not understand why does it matter to you the number of 
> token buckets, whether it is one, or many?

Lets be clear about this.  Do you agree that the spec needs to
address the rate limiting of ICMPv6 packets that are being 
generated by the node (host or router) and not the traffic 
that is being forward by a router.

> I hope you understand that with one token bucket per node, 
> you cannot control ICMP packet generation in a modern router, 
> so basically, routers are left out.
> 
> Just to make sure we're on the same page:
> 
> in a modern router architecture there is distributed packet 
> forwarding, 
> i.e. multiple line cards, each line card with its own 
> implementation of 
> link, IP, and ICMP processing, as well as traffic management, and 
> support for multiple interfaces, that have bandwidth which 
> ranges from 
> fractions of 1.5Mbit/sec at one end of the spectrum, and 
> 1Gbit/sec, or 
> 10Gbit/sec at the other end of the spectrum.
> 
> So you understand now that because each line card may have at 
> least one IP/ICMP engine, the token-bucket per node cannot apply?
> 
> Therefore, the current recommendation in the ICMP specs should be 
> upgraded to support rate control per line card. As a line card has 
> several interfaces, in general of same type, it is sufficient 
> to specify 
> the rate control for one of the interfaces, which brings us to the 
> interface based rate control configuration, and the one sentence text 
> addition to the ICMP specs that I suggested.
> 
> So, in order to support distributed control of ICMP message 
> generation, 
> the addition to the spec which I suggested is necessary, 
> since one token 
> bucket per node cannot apply.
> 
> How is this with you?

As far I understand (and you can explain me if I am not able to see
your point), this case is no different than a router with various 
type of interfaces attached.

Moreover, we may need to be clear about what "per-node" means
in this distributed environment.  To me, "per-node" should
mean per-IPv6/ICMPv6 stack.  This means, you should have the
parameters configurable per line card.

I do not have any personal preferences about any of the choices
that are being discussed.

In terms of being perfect, I agree that per interface configuration
would provide a better control to the user.

But operationally, I think (and I am not always right), per 
interface configuration will be too complex and the extra control 
that it provides might not be worth the trouble.  Ofcourse it 
will be easy if you are just talking about using ACLs to 
implement it.

Regards
Mukesh

--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]
Administrative Requests: https://www1.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------

---

From: Fred Templin <[email-masked]>
Date: Wed, 18 Aug 2004 11:03:23 -0700

Mukesh,

>by a router is out of the scope of this spec.

I'm not sure whether this is what Alex was saying, but consider
the case that an L2 device on the path between routers A and B
starts sending lots of ICMPs along the reverse path back through A.
What should A do in that case? Attempt to forward all of the ICMPs,
or use rate-limiting?

It might be tempting to say that this is beyond the scope of
ICMPv6(bis), but I see that as side-stepping an issue that can
easily be helped. In particular, if router A can use the same rate
limiting mechanisms in ICMPv6(bis) (or rather, the mechanisms
we finally agree upon), then I think it should do so.

Fred
[email-masked]

---

From: Pekka Savola <[email-masked]>
Date: Wed, 18 Aug 2004 21:27:49 +0300 (EEST)

> we finally agree upon), then I think it should do so.

That's definitely out of scope of this *protocol* specification.

They're just forwarded IP packets. More often than not, the router
doesn't even know it's ICMPv6 (because it just looks at the
destination address), and *cannot* even know that (e.g., there are
extension headers, encryption, etc.).

(I don't argue that it might be useful for routers to deal with a
number of bad situations w/ traffic they forward like SYN floods, ICMP
storms, or whatever, BUT THAT'S OUT OF THE SCOPE OF RESPECTIVE
PROTOCOL SPECIFICATIONS -- if you want to fix the Internet, write
documents for operations people!)

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Fred Templin <[email-masked]>
Date: Wed, 18 Aug 2004 12:40:17 -0700

>extension headers, encryption, etc.).

If the router can know that they are error messages and can also know,
e.g., that the errors are arriving at a disproportionally high rate with
respect to the IPv6 packets that could have possibly generated them,
then it should perform rate limiting. (That would be per-interface
rate limiting, I suppose.)

Fred
[email-masked]

---

From: Havard Eidnes <[email-masked]>
Date: Wed, 18 Aug 2004 21:48:23 +0200 (CEST)

> [...], but consider the case that an L2 device on the path
> between routers A and B starts sending lots of ICMPs along the
> reverse path back through A.  What should A do in that case?
> Attempt to forward all of the ICMPs, or use rate-limiting?

Seen from A's side, doesn't this L2 device implement enough of L3
to be called a host?  I'm not sure I understand why you say that
it has to be an L2 device.

If a host is flooding the network with ICMPv6 messages, the bug
is in the host stack of the originating device (if there is
agreement that rate limiting of sourcing of ICMPv6 messages is to
be done, which seems to be the case).

I think it would be most unwise to mandate or recommend that IP
traffic of one type or other receive different forwarding
treatment by default.

This does of course not prevent an equipment vendor to provide
and an administrator to configure his router to defend against
the situation in his network by implementing an optional rate
limiter on an interface.  Such a rate limiter should of course be
different from the rate limiter used for locally originated
traffic.

Regards,

- Håvard

---

From: Alex Conta <[email-masked]>
Date: Wed, 18 Aug 2004 16:28:59 -0400

Mukesh,

[email-masked] wrote:

>> [...]I am rather 

 From a certain perspective, a router's system architecture is certainly 
more complex than a host's, and there are several basic possible valid 
situations, with many variations. For the sake of a well documented 
answer, please see bellow, and bear with me, cause it may be long:

1. IP forwarding and ICMP generation is done in one processor, which 
also controls all interfaces. A host with multiple NICs can act as such 
a router.

2. IP forwarding is distributed on line cards, each line card 
controlling its interfaces, while ICMP generation is done in one central 
processor. In this case the IP stack up to and including the IP layer is 
distributed, but the central processor has no interfaces, therefore, it 
uses the line cards to output ICMP packets. IP packets may be forwarded 
between two interfaces on the same line card, and between two 
interfaces, on different line cards. So, forwarding is the most 
important and powerfully feature, and everything revolves around it, or 
is done based on it. To send ICMP packets out, the central processor 
forwards the ICMP packet to the appropriate output line card, similarly 
to one line card forwarding a packet to another line card, so internally 
there is no difference between forwarding and generating packets. I hope 
this is clear.

3. IP forwarding and ICMP are distributed on line cards, each line card 
controlling its interfaces, while ICMP generation takes place on each 
line card. Again, forwarding is the most important and powerful feature, 
and everything revolves around it. In some cases, the ICMP message is 
sent on a different interface, or line card, then the errored packet 
arrived. Internally, on the line card, the IP and ICMP engines can be in 
the same processor, or in different processors. Packets are passed 
internally from one processor to another almost, or identical to 
forwarding. So the outputting of ICMP packets generated by the line card,
looks like forwarding.

NOTE: Both example 2, and 3 are implementing a distributed IP stack, 
which looks line one stack from outside. Internally, every member of the 
stack performs some IP functions independently. The distributed 
functions and elements of the stack are merged together through the 
internal packet forwarding mechanism.

Recently a lot of debating took place around what sending ICMP packets 
mean? There are questions like:

a.- Does sending ICMP packets start when an errored packet is passed to 
the ICMP message output engine?
b.- Does sending the ICMP packet start when an ICMP message has been 
assembled and is ready to be sent out?
c.- Does sending the ICMP packet start when an ICMP packet has passed 
through the IP output engine?
e.- Does sending the ICMP packet start when it went to the wire?

When this question is asked in relationship to rate limiting, the answer 
can depend one what the implementation choice is:

A.- rate control the IP errored packets coming from a particular interface

B.- rate control the ICMP messages supposed to go out on a certain 
interface.

C.- rate control the IP traffic, with ICMP protocol field, in a generic 
IP traffic management as packets are going out on a certain interface,
Distinction between local or remote packets can done by adding as source 
address filters the local IP addresses.

D.- rate control the IP/ICMP packet, in a generic traffic management 
engine implemented at L2, with IP and ICMP protocols as filters, when 
the packet is ready to be output on the wire. Distinction between local 
or remote packets can be done by adding as IP source address filters 
filters the local IP addresses.

Which one is a correct answer? A? B? C? D? Is one more correct than the 
other? or all of them are equally correct?

Debating this has the potential to turn into a faith based, as opposed 
to technical based debate.

Then the question is it always possible to make distinction between 
IP/ICMP locally generated and IP/ICMP forwarded packets? Obviously in 
some case it is, in some it is not.

IMHO, all answers above are correct, as long as the implementations do 
the right thing, and have the right or desired effect on the network.

So, then what is my interpretation?

I think the ICMPv6 specification is and should be left vague enough to 
leave room for multiple correct interpretations, i.e. implementations.
As an IETF specification, it MUST be a protocol and not implementation 
standard.

> by a router is out of the scope of this spec.

I hope I explained clear enough that in some cases, there is no internal 
distinction between forwarded and locally sourced packets. To make the 
distinction, a source address filter can be applied, but is that always 
necessary? As it turns out, it is not, and that is OK, as long as the 
effect on the network is OK.

> line to those provocative sentences.

I am sorry if I hurt anybody's feelings.

> that is being forward by a router.

I think I answered this above.

It seems that in modern routers, ICMP traffic controlled per interface, 
part of generic traffic control per interface, is cost effective. Cost 
is what ultimately counts.

The rate  control can be configured to filter locally generated, all 
ICMP messages (local and forwarded), or only remotely generated (forwarded).

Does it matter that ICMP message generation is not controlled internally 
in the ICMP engine? In some cases perhaps it does, in others it does not.

But let's not lose perspective. Remember, all of this started from one 
simple sentence to suggest a more appropriate suggestion for rate 
control for routers.
Can we still address that?

>[...] 

When looked from outside, it is true. From inside it may not, but it 
does not matter. The per interface rate control parameters can always be 
translated into the appropriate granularity: line card, or node 
mechanism. The opposite is not true.

> mean per-IPv6/ICMPv6 stack.  

I think your observation is right. "Per-node", in general, is IMHO 
clear, meaning "all interfaces".

> This means, you should have the

We do have the two concepts of Node, and Interface in IPv6. We could 
define a Line Card, as a group of interfaces. Keep in mind though, for 
what we need,... since an interface belongs to a line card, the line 
card can always be inferred from the interface. So we can get away, 
without the line card concept.

> that it provides might not be worth the trouble.  

I never saw in an IP stack, and I've seen a good number, the IP layer 
passing the IP packet without a reference to the incoming interface.

This of course applies to passing errored packets to the ICMP engine.
So, as this interface information is available, it can be used to 
reference a per interface rate control mechanism, or simply per 
interface rate control parameters. It is quite easy.

> Ofcourse it 

Router vendors do that indeed, so you're absolutely right!

The cost of generating ICMP messages dropped with the increase of 
processor power, to a level, where it is more cost effective to generate 
ICMP packets and drop them before they're sent out in the traffic 
management engines, than control their generation separately from 
traffic management.

> Mukesh
Regards,
Alex

---

From: Alex Conta <[email-masked]>
Date: Wed, 18 Aug 2004 22:46:45 -0400

Havard Eidnes wrote:

> limiter on an interface.  

If the router has traffic management per interface, and many if not most 
nowadays do, the ICMP traffic shaping is not different than shaping any 
other traffic.

> Such a rate limiter should of course be

Why should?

If the router implements ICMP rate limiting per interface, without any 
differentiation between local ICMP and forwarded ICMP the effect is 
still the same, and none need call the ICMP protocol police (-:. Who 
cares that the process may incure dropping some internally generated 
ICMP messages?

Regards,
Alex

---

From: Pekka Savola <[email-masked]>
Date: Thu, 19 Aug 2004 10:34:19 +0300 (EEST)

One set of answers to your earlier message..  This is probably past
its usefulness.

> also controls all interfaces. A host with multiple NICs can act as such 

Sure.  

I say these explicitly, just to be sure that my comments are not seen
as me saying "there must be only one ICMP origination engine per node
and it must be done on the CPU".  I've not said that.  If an
implementation has multiple linecards and they generate ICMP's, it's
just fine to put the generation on each line card -- you can even use
the same algorithm on each card.

> processor.

Sure.

> To send ICMP packets out, the central processor 

No - you may be using the term "forwarding" in a confusing manner
here.  In the context of IP documents such as ICMPv6 spec, we're
interested of IP forwarding (TTL decrease, etc.).  When you have
message passing between linecards and/or central processor, that's an
implementation specific approach to message passing.  It might even be
based on IP forwarding (and not just some implementation dependent
message passing mechanism -- I've seen both, unfortunately) -- but we
shouldn't need to care.

The point is, we'll just assume that *whoever* generates the ICMP
response (be it central processor, linecard or whatever) will perform
the origination rate-limitng.. or that the implementation makes sure
it will be done (and makes sure that it doesn't do rate-limiting for
other kinds of ICMP traffic) -- but if so, that's an implementor's
decision, not something we'll want to put in a spec.

> line card. 

Sure.

>  Packets are passed 

Good that you acknowledge that internal message passing is not IP
forwarding.

Remember, the IETF procudes protocol documents which describe the
external behaviour.  To do so, the architecture is often simplified,
and described based on that simplified architecture.  If an
implementation has more complex architecture, the details with such
architecture are beyond the scope of the specification, to be done as
the implementor sees fit as long as the external behaviour is the
same.
 
> e.- Does sending the ICMP packet start when it went to the wire?

These use the ambiguous word "send".  In any case, this is probably
implementation-specific, not something we need to be all that
concerned of.

> other? or all of them are equally correct?

B seems like the obvious choice here.  C and D are obviously not
something to be done in the ICMP specifications (but one is of course
free to do so internally as long as the behaviour is consistent).  A
is out of scope of ICMPv6 spec because the problem is addressed by B.

> the right thing, and have the right or desired effect on the network.

Approaches C & D impose constraints to the forwarded traffic.

But of course, your implementation is free to do something like that
(e.g., to rate-limit the ICMP origination by setting up generic
traffic limiters which list all the IP addresses of the specific
node), as long as it's externally consistent with the specification,
but it's not something we need or want to specify.  We want to keep
the concept very simple... which it isn't, at the moment.

> As an IETF specification, it MUST be a protocol and not implementation

No, it must not be too vague (as it seems to be now) that it's making 
it too difficult to interpret the meaning appropriately.

Even if we say that ICMP origination MUST be rate-limited, and IP
forwarding rate-limiting is out of scope (or MUST NOT be rate-limited
based on this specification), and we said nothing of
interface-specific configuration, nothing would prevent the
implementation the way you have described.  It just wouldnt be in the
spec to confuse matters.

> effect on the network is OK.

I think the difference stems from the fact that you think of this kind 
of distributed model as (IP) forwarding.  To the rest, it's just 
message passing, and even if it'd be done based on IP forwarding, it's 
just an implementation issue.

> ICMP messages (local and forwarded), or only remotely generated (forwarded).

I guess these ACLs are generated by an algorithm at the central
processor which monitors which IP addresses the node has.  Remember
that many hardware implementations also have constraints on
access-lists they can perform.  Does this work well on a router with
100 or 1000 IP addresses?  Maybe not.

---

From: Havard Eidnes <[email-masked]>
Date: Thu, 19 Aug 2004 13:28:39 +0200 (CEST)

> If the router has traffic management per interface, and many if
> not most nowadays do, the ICMP traffic shaping is not different
> than shaping any other traffic.

True.

> > different from the rate limiter used for locally originated

Because one should (I'd say MUST) differentiate between shaping
locally originated ICMP datagrams and forwarded ICMP datagrams
(the latter which should be outside the scope of this spec, per
Pekka's latest text which I agree with).

> If the router implements ICMP rate limiting per interface,
> without any differentiation between local ICMP and forwarded
> ICMP the effect is still the same, and none need call the ICMP
> protocol police (-:.

I disagree.  If origination of ICMP is vital to some function or
other (it would be for PMTUD in IPv4), a host could spew ICMP
messages to be forwarded and prevent or significantly hamper a
router along the path from originating any ICMP traffic of it's
own if both the forwarded and the originating traffic uses the
same rate limiter, and the originated rate exceeds the rate
specified by the rate limiter.

> Who cares that the process may incure dropping some internally
> generated ICMP messages?

See above.

Regards,

- Håvard

---

From: Alex Conta <[email-masked]>
Date: Thu, 19 Aug 2004 18:04:59 -0400

> locally originated ICMP datagrams and forwarded ICMP datagrams

This still does not elaborate on the reasoning. So the question stays:

why SHOULD, why MUST?

> Pekka's latest text which I agree with).

The ICMPv6 specification describes the ICMPv6 protocol, a set of ICMPv6 
error and informational messages, and some rules and/or suggestions on 
handling ICMPv6 messages. Why should "in transit" ICMP messages be 
outside its scope?

> specified by the rate limiter.

I am sorry, the text was not clear enough:

"without any differentiation" was meant that both 'locally generated' 
and 'in transit' ICMP packets are passed through the same "per interface 
traffic management engine". It was not meant to apply to operations 
performed by traffic management.

The valid issue, which you pointed out, is resolved by the router 
performing a preferential excess packet dropping.

>>Who cares that the process may incure dropping some internally

I need to clarify this too - sorry again.

This was meant that if for instance there is a sequence of 100 errored 
packets received from the same source, and the router generates 100 ICMP 
messages, the traffic management may drop a certain excess, let's say 
80, and from the network perspective this would look like the router 
generated internally only 20 ICMP messages.

Regards,
Alex

---

From: Brian Haley <[email-masked]>
Date: Thu, 19 Aug 2004 23:45:13 -0400

> handling ICMPv6 messages. Why should "in transit" ICMP messages be 

Others have pointed out that's not how they read rfc2463 and I'd have to 
agree.  To back this up I'll give you the following:

  2.2 Message Source Address Determination

    A node that sends an ICMPv6 message has to determine both the Source
    and Destination IPv6 Addresses in the IPv6 header before calculating
    the checksum.

"Send" here refers to the node generating the packet, since a router 
wouldn't be computing a checksum in an icmpv6 header while forwarding.

  2.4 Message Processing Rules

     (f) Finally, in order to limit the bandwidth and forwarding costs
         incurred sending ICMPv6 error messages, an IPv6 node MUST limit
         the rate of ICMPv6 error messages it sends.

Again the word "send" is used, seems like the same context as above.

And searching rfc2463 for "forward" or "router" only yields cases where 
a packet is being generated (too big, etc.), no mention of rate-limiting 
the forwarding of icmpv6 messages.

I have to agree with Pekka and Havard that this is out of scope here.

-Brian