Subject: Neighbor Unreachability Detection is too impatient

From: Erik Nordmark <[email-masked]>
Date: Mon, 23 May 2011 11:46:29 -0700

This draft proposes to change the requirement that NUD can not 
retransmit more than three times, so that NUD can be more robust against 
temporary network outages.

Comments?

    Erik

-------- Original Message --------
Subject: New Version Notification for 
draft-nordmark-6man-impatient-nud-00.txt
Date: Mon, 23 May 2011 11:43:16 -0700
From: [email-masked]
To: [email-masked]
CC: [email-masked]

A new version of I-D, draft-nordmark-6man-impatient-nud-00.txt has been 
successfully submitted by Erik Nordmark and posted to the IETF repository.

Filename:	 draft-nordmark-6man-impatient-nud
Revision:	 00
Title:		 Neighbor Unreachability Detection is too impatient
Creation date:	 2011-05-23
WG ID:		 Individual Submission
Number of pages: 5

Abstract:
    IPv6 Neighbor Discovery includes Neighbor Unreachability Detection.
    That function is very useful when a host has an alternative, for
    instance multiple default routers, since it allows the host to switch
    to the alternative in short time.  This time is 3 seconds after the
    node starts probing.  However, if there are no alternatives, this is
    far too impatient.  This document proposes an approach where an
    implementation can choose the timeout behavior to be different based
    on whether or not there are alternatives.

The IETF Secretariat

---

From: Wes Beebee <[email-masked]>
Date: Mon, 23 May 2011 15:25:51 -0400

Erik,

I have seen NUD packets dropped during congestion, and for traffic to
periodically drop out for re-resolution.  I agree with the goal of making
NUD more robust.  However, there may be other approaches besides
retransmitting more times.

- Wes

On 5/23/11 2:46 PM, "Erik Nordmark" <[email-masked]> wrote:

> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: Mark Townsley <[email-masked]>
Date: Mon, 23 May 2011 21:47:06 +0200

Erik Kline and I wrote up an experience we had with NUD and a broken IPv6 firewall on my home network.

http://sites.google.com/site/ipv6center/icmpv6-is-non-optional

In short, NUD thought that a host which was in the neighbor cache really wasn't available (due to incorrect FW blocking at the host) removed it from the neighbor cache and started dropping packets (and causing timeouts in gmail). In test, it took about 8 pings for the problem to manifest (time for NUD to retry enough to fail), which was followed by the classic IPv6 to IPv4 failover by the host (Windows Vista). 

On one hand, if NUD had been more patient we may have never noticed. However, tracking down a bug that showed up less often would have clearly been more difficult to diagnose. Double-edged sword.

Also, the Broadband Forum's WT-146 (which isn't a public document, but if you are BBF member you can take a look at it)  specifies NUD as a monitoring mechanism for "IPv6 sessions" over access links (which ultimately may be tied into billing, alerts, etc.). 

- Mark

On May 23, 2011, at 8:46 PM, Erik Nordmark wrote:

> 
> This draft proposes to change the requirement that NUD can not retransmit more than three times, so that NUD can be more robust against temporary network outages.
> 
> Comments?
> 
>   Erik
> 
> Subject: New Version Notification for draft-nordmark-6man-impatient-nud-00.txt
> A new version of I-D, draft-nordmark-6man-impatient-nud-00.txt has been successfully submitted by Erik Nordmark and posted to the IETF repository.
>

---

From: Philip Homburg <[email-masked]>
Date: Mon, 23 May 2011 21:09:44 +0200

>temporary network outages.

Do you have more data on how this problem actually shows up practice? 

The draft suggests that the main difference is more multicast traffic. 

How many hosts do you need to have on a single link for this to significantly
impact the performance of the link?

---

From: Thomas Narten <[email-masked]>
Date: Tue, 24 May 2011 08:45:23 -0400

I support this document.

As a general statement, ND has survived for 15 years with relatively
few changes from the original specification. We can all take pride in
that success.

That said, a number of the individual decisions made at that time were
somewhat arbitrary, and reflected a view of the world consisting of
ethernets (and other LAN technologies) defined by the state-of-the-art
available at that time.

As systems and technologies have evolved in the last 15 years, some of
the boundaries of the ND spec seem worthy of revisiting to take into
account what actual deployments are doing today (or will be in the
near future).

As one example, the cost of multicast on wifi is significantly higher
than unicast. Yet, the ND spec calls for deleting an NCE if NUD fails,
resulting in a fallback to multicast for address resolution. While
this is a simple and straightforward way to make the protocol work, I
can imagine worthwhile optimizations whereby ND is more conservative
in falling back to multicast and deleting an NCE that was working fine
just a few seconds ago. For example, one could continue probing a
neighbor using both unicast and multicast, and one could continue to
forward traffic to a unicast neighbor vs. discarding it because the
NCE has been deleted.

I'm sure there are other examples where tweaks to the basic algorithms
would be worth considering (so long as they are justified be real
operational concerns).

Thomas

---

From: [email-masked]
Date: Tue, 24 May 2011 09:21:27 -0400

On (05/23/11 11:46), Erik Nordmark wrote:
> 
> 
> This draft proposes to change the requirement that NUD can not retransmit
> more than three times, so that NUD can be more robust against temporary
> network outages.
> 
> Comments?

I think the intention is to use impatient-NUD only when the unicast NS
has been triggered by a packet whose IP destination is offlink, i.e.,
the IP destination is different from the target of the NS, so
impatient-NUD should only be applied for router NCE's (and only when
the triggering packet has a different IP destination than the NCE target). 

But if the motivation is to avoid sending mcast NS where possible, it might
not be achieved in a few other cases that we can still catch. 
E.g., even for offlink IP destinations, the switch to an alternative
default router might also result in triggering mcast NS if the new
router itself has not been resolved yet.  If the objective is to avoid
sending multicast NS, impatient-NUD would only be useful in reducing
mcast traffic if the alternative NCE is in REACHABLE/STALE/PROBE/DELAY
state. Would it make sense to REQUIRE impatient-NUD only under that
specific circumstance?

--Sowmini

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 24 May 2011 09:36:15 -0700

> Do you have more data on how this problem actually shows up practice?

One place where is shows up is when you have a stable network where the 
routers and hosts have neighbor cache entries for the peers they talk 
to. But then there is a short outage on the LAN, e.g., due to a switch 
or link failure causing spanning tree to recalculate things. Should 
e.g., the router start NUD at that point in time, then NUD is likely to 
discard the Neighbor Cache entries before STP is done.

Thus impact of the STP recalculation gets a lot worse, especially if 
this is in a massively scaled datacenter.

> impact the performance of the link?

Even if there are only two hosts, the issue is that the effect of an 
event like a STP recalculation expands due to NUD. We don't have that 
behavior with IPv4/ARP since it doesn't mandate short timeouts.

    Erik

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 24 May 2011 09:39:36 -0700

> retransmitting more times.

Agreed. For instance, it makes sense to consider whether the Address 
Registration option in draft-ietf-6lowpan-nd is something that should be 
applied outside of 6lowpan.

However, such approaches imply a change to hosts and routers.

Relaxing the constraints on the ND retransmissions is something which 
doesn't require a coordinated rollout of router and host changes.
Hence I think we need to relax the retransmission constraints now, while 
looking at future directions.

    Erik

> - Wes
>
> On 5/23/11 2:46 PM, "Erik Nordmark"<[email-masked]>  wrote:
>
>>

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 24 May 2011 10:05:22 -0700

> I support this document.

Thanks.

> forward traffic to a unicast neighbor vs. discarding it because the

Agreed.

The key thing is that when there is a choice (a host with multiple 
default routers and perhaps a NCE created by a redirect) that the 
failure of NUD cause it to try an alternative.

But if there is no such choice, keeping the NCE around and using it, 
while trying multicast NS packets, makes a lot of sense. Even if there 
is a choice, it can make sense to keep the NCE around but marked as 
UNREACHABLE, if that makes it easier to recover.

Should I add your suggestion to the draft?

    Erik

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Tue, 24 May 2011 12:09:33 -0500

I support this document.  I will send minor edits directly to the
author.

Hemant

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Erik Nordmark
Sent: Monday, May 23, 2011 2:46 PM
To: [email-masked]
Subject: Neighbor Unreachability Detection is too impatient

Comments?

    Erik

-------- Original Message --------

A new version of I-D, draft-nordmark-6man-impatient-nud-00.txt has been 
successfully submitted by Erik Nordmark and posted to the IETF
repository.

Abstract:
    instance multiple default routers, since it allows the host to
switch

The IETF Secretariat

---

From: Philip Homburg <[email-masked]>
Date: Tue, 24 May 2011 19:12:41 +0200

>this is in a massively scaled datacenter.

I'm surprised STP traffic has such a low priority that it is actually affected
by multicast traffic.

But if that's the case, then I guess that changing the NUD paramters to
something that takes more than 6 seconds would be a good idea.

But that effect can also be obtained by setting the Retrans Timer field in
the RA message to 3000ms or more.

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 24 May 2011 10:15:39 -0700

> the triggering packet has a different IP destination than the NCE target).

For an off-link destination, if there is only one choice (one default 
router) there is no need to be impatient.
Thus the applicability of patience is a bit broader.

> state. Would it make sense to REQUIRE impatient-NUD only under that

If a host has multiple default routers, it is likely (but not required 
by RFC 4861) that the host sends some flows to the different default 
routers. In that case there is an NCE for each default router. Thus 
declaring one of them as unreachable and moving those flows to use a 
different default router doesn't cause additional multicasts.

    Erik

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 24 May 2011 10:22:09 -0700

> I'm surprised STP traffic has such a low priority that it is actually affected

That isn't the issue.

During the STP topology change bridges go into blocking state for some 
amount of time. This causes packets to be dropped. If this last for more 
than 3 seconds, and a NUD probe is started at that point, then the three 
*unicast* NS messages that are used for NUD are all dropped.

With current RFC 4861 this means that the NCE has to be discarded, since 
the neighbor is deemed UNREACHABLE.

Without the NCE, the next time the node tries to send a packet, it will 
end up multicasting NS messages.

    Erik

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Tue, 24 May 2011 12:27:06 -0500

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Mark Townsley
Sent: Monday, May 23, 2011 3:47 PM
To: Erik Nordmark
Cc: [email-masked]
Subject: Re: Neighbor Unreachability Detection is too impatient

>In short, NUD thought that a host which was in the neighbor cache
really wasn't available (due to incorrect FW blocking at the host)
removed it from the neighbor cache and started dropping packets (and
causing timeouts in gmail). In test, it took about >8 pings for the
problem to manifest (time for NUD to retry enough to fail), which was
followed by the classic IPv6 to IPv4 failover by the host (Windows
Vista). 

In your test did you see how many NUD probes were sent out when eight
pings were sent?  The reason I am asking the question is because the
unicast NUD messages have to cap at three.    In so many pings, I
wouldn't be surprised if the host also issues an address resolution
multicast NS.   

>Also, the Broadband Forum's WT-146 (which isn't a public document, but
if you are BBF member you can take a look at it)  specifies NUD as a
monitoring mechanism for "IPv6 sessions" over access links (which
ultimately may be tied into billing, >alerts, etc.).

Note with this document's proposal  and if the BBF access link has no
alternative, the BBF nodes would have to implement NUD probing for a
longer time.  Alternatively, one could use BFD for monitoring. 

Hemant

---

From: Philip Homburg <[email-masked]>
Date: Wed, 25 May 2011 11:10:10 +0200

>end up multicasting NS messages.

So you are saying (with a bit exaggeration from my side) that some links
in data centers have so many hosts that a network disruption of a few seconds
will cause a multicast storm big enough to meltdown the network.

With modern CPUs and a Gbit/s network that seems very odd to me.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 25 May 2011 11:20:49 +0200 (CEST)

On Wed, 25 May 2011, Philip Homburg wrote:

> in data centers have so many hosts that a network disruption of a few 
> seconds will cause a multicast storm big enough to meltdown the network.
>

I have an example from reality:

At AMSIX, there are a few hundred routers on the same L2 network. 
Connected to this we have a router from a major well known vendor. This 
vendor does not filter IPv6 multicast messages on the linecard, but 
instead punts all of it to RP, and AMSIX L2 infrastructure treats all 
multicast as broadcast, so all multicast (mainly ND traffic) is sent out 
on all ports.

This traffic to the RP was enough to cause IPv6 BGP sessions to go down 
because it was continous ND messages coming in all the time (few hundred 
PPS).

ND is chatty, and it does cause problems on large networks. One might call 
this multiple faults on multiple levels (bad linecard code, bad L2 
platform, bad low-performing CPU on the RP), but it does cause problems in 
real life regardless.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Philip Homburg <[email-masked]>
Date: Wed, 25 May 2011 18:09:27 +0200

In your letter dated Wed, 25 May 2011 08:05:27 -0700 you wrote:
>I said that the current assumed maximum 3 NUD probes one second apart 
>makes IPv6/ND less flexible than IPv4/ARP, and I think we should remove 
>those constraints.
>
>I'm having a hard time understanding what point you are trying to make, 
>and whether or not it relates to the text in the draft. Please send some 
>suggested text changes to the draft if you think that would address your 
>concern.

In my opinion, the current spec is simple and it works well. The advantage
of precisely defining all the protocol contants also means that you predict
what nodes are going to do. With ARP you are never sure. 

One way to move forward is to define an option for router advertisements to
change some of the constants you want changed.

Another option is to just update the RFC with new constants.

NUD a lot of stuff. It detects that a system is down, allowing higher levels
to take an alternative path. It detects that a system got a new MAC address
(something you can easily found out from ARP broadcasts, but not as easily
from ND multicasts), it can expire routers that down, not just for default
routers but also after a redirect, and for prefixes with the onlink bit clear,
the same applies to hosts.

So there is a lot of stuff going on there. You do really want to just randomly
mess with that?

>From reading your draft, you never seemed to have considered the case that
a node would get a new ethernet address. 

So, I have two complaints. The problem statement is one of hand waving. Yes,
there will be a few more multicasts. No numbers on how badly it fails.

Second, I don't think it should be left to individual implementors. If you
want to create an additional nob for administrators, then just do so. Or if
3 NUD packets is just too few, then propose increasing it.

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 25 May 2011 08:05:27 -0700

> will cause a multicast storm big enough to meltdown the network.

Philip,

No that isn't what I said.

Thanks,
    Erik

---

From: Thomas Narten <[email-masked]>
Date: Wed, 25 May 2011 16:53:22 -0400

> Should I add your suggestion to the draft?

Kind of depends on where you see your document going.

Should we be looking at a more substantial effort to cleanup edge
cases in ND? What issues are there and how big an effort will that
turn into?

Or should we be publishing small "stand alone" tweaks individually,
consolidating them all at some point in the future?

I suspect that some changes will come out of the ARMD WG effort. But
it's a little early to know for sure yet what requirements might come
out of that effort.

Thomas

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 25 May 2011 15:06:50 -0700

> cases in ND? What issues are there and how big an effort will that

Covering all possible edge cases is too open ended IMHO.
But if we are going to clarify and relax things around retransmission 
behaviors in this draft, then we can attempt to address that whole area.

    Erik

---

From: Alan Kavanagh <[email-masked]>
Date: Tue, 31 May 2011 13:22:02 -0400

Hi Mark

That BBF WT-146 NUD for session monitoring was based on the idea that NUD is native to IPv6 compliant devices such as host, cpe and routers and was chosen as a passive way to do session monitoring.

Alan

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Townsley
Sent: May-23-11 3:47 PM
To: Erik Nordmark
Cc: [email-masked]
Subject: Re: Neighbor Unreachability Detection is too impatient

- Mark

Comments?

  Erik

-------- Original Message --------
Subject: New Version Notification for draft-nordmark-6man-impatient-nud-00.txt
Date: Mon, 23 May 2011 11:43:16 -0700
From: [email-masked]<mailto:[email-masked]>
To: [email-masked]<mailto:[email-masked]>
CC: [email-masked]<mailto:[email-masked]>

Abstract:

The IETF Secretariat
--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]<mailto:[email-masked]>
Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------