Subject: [6man] New Version Notification for draft-nordmark-6man-impatient-nud-00.txt

From: Ray Hunter <[email-masked]>
Date: Mon, 23 May 2011 22:03:50 +0200

re: http://tools.ietf.org/html/draft-nordmark-6man-impatient-nud-00

I'm afraid have more questions than answers.

Are there any implications for different nodes having different NUD 
timeout behavior on a link, and this no longer being symmetrical?

If I can think of two examples......

e.g. 1. Say Node A (router) declares node C (end node) unreachable but 
Node B (alternate back up router) has not yet timed out node C?

I'm guessing this case is just like a split-brain segment, so is not 
significant compared to existing failures.

Now the case of router failover....

e.g. 2. Say Node A (end host) declares node B (router) unreachable 
locally, but node B (router) is still up and running but has not yet 
timed out Node A.

Is that significant? I suspect so. After all if the raison d'etre of 
changing NUD timers is to quicken / slow down router failover, surely 
Node B (the router) also has to time out at the same speed as the end 
host (Node A) otherwise the router will continue to advertise valid 
routes to node A, and packets will black hole/queue anyway until NUD on 
node B also notices the failure.

Vice versa is also true, if the router notices the failure first, but 
the end node does not react to the failure and hangs around retrying 
NUD, packets may queue/black hole in the other direction.

In the good old days we had things like gratuitous ARP for such events 
to attempt to wake up end nodes to refresh their cache, but if they got 
lost in some layer 2 STP thrashing it didn't help much anyway.

Is there thus a need for any over-ridden NUD parameters to be 
synchronized across all nodes on a link e.g. via RA messages?

Is there a minimum and maximum timeout needed? To prevent danger of an 
update storm [as specified in RFC2461 that all Neighbor Solicitations 
are rate-limited on a per-neighbor basis] or "stuck in stale".

regards,
RayH

---

From: Ray Hunter <[email-masked]>
Date: Mon, 23 May 2011 23:10:09 +0200

Who says that NUD can't also be used to declare an interface down/ 
detect router neighbor loss?

Maybe think of a BGP process running over TCP receiving ICMP 
unreachables because the local NUD has declared the neighbor 
unreachable. Meanwhile the other BGP partner router is still retrying at 
TCP layer because NUD has not timed out on that node. Or am I seeing 
non-existent links here?

As someone who has had more than one evening / weekend ruined in a data 
centre with sniffers looking at interactions between ARP and STP 
thrashing about and Cisco HSRP and HP MC ServiceGuard and Network 
intrusion detection/ firewall failover, I would contend that these 
mechanisms are anything but independent, even of the designers intended 
them to be that way.

Especially when a layer 2 protocol like spanning tree is thrashing, or a 
local security box is performing a local non-transparent failover, 
things can get very unpredictable with all sorts of timers working at 
various levels in the stack.

IMVHO A reliable and predictable NUD is a blessing compared to ARP ;) 
Which is why I think that breaking synchronization/ symmetry of nodes 
declaring partners unreachable may have unforeseen side effects, and 
hence the questions.

best regards,
RayH

Philip Homburg wrote:
> In your letter dated Mon, 23 May 2011 22:03:50 +0200 you wrote:
>    
> I don't think I understand your model of how a router works.
>
> To a large extent both directions are independent. A host has to get a packet
> to any default router, and uses ND to figure out where they are and whether
> they are reachable.
>
> A router uses ND to find where a host is. If ND fails, then the router simply
> cannot deliver the packet.
>
> Both failures are independent.
>
>
>

---

From: Ray Hunter <[email-masked]>
Date: Mon, 23 May 2011 23:40:18 +0200

Philip Homburg wrote:
> In your letter dated Mon, 23 May 2011 23:10:09 +0200 you wrote:
>    
> Let's say router A declares router B unreachable because of some ND problem.
> Meanwhile router B still considers router A reachable.
>
> Now obviously, router A (and the routing system) will try to avoid routing
> packets from A to B because that link is down.
>
> B still assumes that A is reachable so it will continue to forward packets to
> A. As long a A does not drop those packets, everything will be fine. I don't
> think there is a reason to drop incoming packets when a neighbor on a link is
> unreachable, but if an implementation does that, then that will break the
> independence and will cause problems.
>
> But for relatively stable links consisting of just BGP peers, it may make more
> sense to just hardwire the ND entries and disable ND.
>
>
>    
You are probably 100% correct for BGP. And I'm reasonably convinced BGP 
is relatively bullet-proof (even if one well-known network provider that 
I know does not give BGP TCP sessions priority over normal user traffic, 
so BGP neighbors can flap on high user load.) It was just one example 
off the top of my head of a dependency.

The only point I'm making is that there may well be stuff out there that 
has become reliant on the current symmetrical behavior of NUD. ARP 
caches generally took a very long time before they started sending back 
an ICMP unreachable. AFAIK NUD is pretty instantaneous once it declares 
something unreachable.

It was after all a selling-point of NDP and neighbor discovery in 
general that it "fixed" the half-open link problem.

RFC 2461 "Unlike ARP, Neighbor Discovery detects half-link failures"

And now it mightn't live up to that promise: at least in transitory 
situations that might last for an extended period of comparable duration 
to timeouts in other protocol layers / applications, unless the timers/ 
retry counts are synchronized across all nodes on the link. Timers and 
race conditions can be very tricky to catch/ debug, as I'm sure you know.

Just raising a flag. Maybe this is significant, maybe not. Just thought 
I'd ask.

regards,
RayH

---

From: Philip Homburg <[email-masked]>
Date: Mon, 23 May 2011 22:33:46 +0200

---

From: Philip Homburg <[email-masked]>
Date: Mon, 23 May 2011 23:22:59 +0200

---

From: Thomas Narten <[email-masked]>
Date: Tue, 24 May 2011 08:49:51 -0400

> timeout behavior on a link, and this no longer being symmetrical?

I do not think so.

> Node B (alternate back up router) has not yet timed out node C?

IMO, this should not be an issue. 

> locally, but node B (router) is still up and running but has not yet 

At this point there is already a problem, because packets  do not seem
to be flowing. The sooner both peers recognize that the path is not
working and find an alternative the better.

But by definition, the two peers will *never* be able to determine
this situation at *exactly* the same time. That is, there will always
be cases where NodeA determines NodeB is unreachable before NodeB
detects NodeA is unreachable (and vice versa).

While it would be better to have both peers determine that the path to
each other is down ASAP, there is not and has never been a requirement
that both peers detect that a path is not working at the same time.

Thomas

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 24 May 2011 09:50:07 -0700

> timeout behavior on a link, and this no longer being symmetrical?

No - the reachability state in RFC 4861 is unidirectional. This can 
easily happen today on a host when TCP provides reachability 
confirmations to the IP stack, which just indicates forward reachability 
to the nexthop router. The return traffic might arrive to the host from 
a different router.

> node B also notices the failure.

Are you assuming that the routers inject host routes into the routing 
system based on the ND state? The routers inject a route for the subnet 
prefix which isn't tied to the ND state in any way.

(If you look at 6lowpan plus roll you'll see a different behavior, which 
is part of the reason for having explicit host registration in 
draft-ietf-6lowpan-nd.)

> synchronized across all nodes on a link e.g. via RA messages?

No.

> are rate-limited on a per-neighbor basis] or "stuck in stale".

A rate limit makes sense. RFC 4861 already has this with one per second. 
If ND retransmits more than three times it probably makes sense to 
recommend binary exponential backoff for the timer.

   Erik

> regards,
> RayH
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: Erik Nordmark <[email-masked]>
Date: Tue, 24 May 2011 10:45:51 -0700

> sense to just hardwire the ND entries and disable ND.

FWIW RFC 4861 allows for this.

Section 7.3 says:
    Neighbor Unreachability Detection is used for all paths between hosts
    and neighboring nodes, including host-to-host, host-to-router, and
    router-to-host communication.  Neighbor Unreachability Detection may
    also be used between routers, but is not required if an equivalent
    mechanism is available, for example, as part of the routing
    protocols.

    Erik

---

From: Ray Hunter <[email-masked]>
Date: Tue, 24 May 2011 20:33:13 +0200

Thanks very much for replying.

I think that I understand the motivation in that multicast is expensive 
on some media, and that you thus want to avoid it.

I'm always prepared to be dazzled by my lack of knowledge or incorrect 
assumptions. I'd much rather ask a dumb question and get a smart answer 
than just say nothing.

The idea of the draft seems to be that spending more time performing 
unicast neighbor solicitations in the "probe" state might avoid deleting 
the neighbor entry, and thus the relearning the entry via "expensive" 
multicast NS from the "-" state.

Seems perfectly reasonable and something worth pursuing.

Erik Nordmark wrote:
> system based on the ND state? The routers inject a route for the 
> subnet prefix which isn't tied to the ND state in any way. 

Yes, on the local link the router injects RA information based on a 
(statically configured or PD learned) prefix. But routers and other 
devices also redistribute reachability information elsewhere via other 
protocols.

The assumption that ND really is independent of everything else is what 
I'm questioning myself, although I freely admit to a large dose of hand 
waving here. ND isn't like ARP, as you of course know.

An ARP cache entry would sit there silently for 4 hours by default and 
do nothing, so packets could black hole if the next hop was learned via 
a static route. Higher level protocols would have to detect the problem 
themselves.

ND removing an entry by NUD probe failure retriggers next hop 
determination, and AFAIK also actively triggers replying to remote nodes 
with an ICMP unreachable message, and so ND can thus can effectively 
disseminate reachability information far further than just the local link.

A later post I made gave an example of reachability info being 
indirectly based on ND (via a BGP neighbor peering TCP session becoming 
unreachable due to an ICMPv6 unreachable, leading to route information 
changing). I can imagine the same for EIGRP, OSPF if their neighbors 
disappear due to receipt of an ICMPv6 unreachable (although 
traditionally these implementations have tended to ignore ICMP for good 
reason).

Another example sort of device that sometimes transmist reachability 
information via TCP are WAN accelerators, that auto build network 
tunnels, and then send routing information across these. Again, an 
ICMPv6 unreachable might cause the device to tear down the tunnel.

HSRP preference metrics can also potentially be influenced by 
reachability information (ND) from another link (via track commands).

Then there are also those dreaded silent devices (that we don't talk 
much about but which are generally plonked on the very most critical 
link into the main data centre), such as network intrusion detection 
systems and firewalls, that actively monitor traffic across their links, 
but that don't take part in any official routing protocol exchanges, and 
can fail over to a back up system without informing anyone else by 
marking interfaces up and down.

Using the example of spanning tree, waiting for STP would probably mean 
waiting 35 seconds (max_age + forwarding delay) in the default case for 
the root bridge to send out topology notification BPDU's. That's a long 
time in many protocols.

So I guess the question is also, do you want NUD to inform higher layers 
of the need for a fail over ASAP of a local link failure via ICMP 
unreachables (as Thomas seemed to suggest), or do you want ND to shut up 
and just keep on retrying locally and let those higher layer protocols 
hit their own time outs and take their own fail over actions?

Current ND seems to go the ASAP route with its 3 second timeout. 
Historically, ARP seems to go the silent route.

It just feels to me like all nodes on a common link should behave the 
same way in this respect (no scientific argument, just raw gut feeling 
of deja vu, and impending packet storms)

And if all nodes on the link aren't behaving the same way, don't you 
still get say 50% of the multicasts as the partner nodes revert to the 
"-" state by timing out "too fast" for that link type?

Just seems like another reason to have this as a "per link" parameter 
rather than a "per node" parameter.

Best regards,
RayH

---

From: Erik Nordmark <[email-masked]>
Date: Wed, 25 May 2011 08:59:52 -0700

> multicast NS from the "-" state.

If the NCE is retained, there is the added benefit of knowing that a 
node with that IPv6 address exists (or recently existed) on the link, 
which is helpful to know when dealing with DoS scans across the IPv6 
address space.

> Seems perfectly reasonable and something worth pursuing.

Good.

> waving here. ND isn't like ARP, as you of course know.

In terms of the interaction with routing, I certainly don't know of any 
case where RFC 4861 behaves differently than RFC 826. Perhaps you can 
enlighten me on the differences you see in the interaction with routing?

> a static route. Higher level protocols would have to detect the problem

Such an ARP implementation wouldn't follow RFC 1122.
Note that RFC 1122 also has a MUST for dead gateway detection (with 
"gateway" being an old term for "router"), but it doesn't specify how 
that is performed, and I don't think it is commonly implemented in hosts.
NUD was designed to solve the dead gateway detection, but generalizing 
it to nodes instead of just routers.

> traditionally these implementations have tended to ignore ICMP for good

Again, if you look at RFC 1122 you'll find that (in section 4.2.3.9) 
"TCP MUST NOT abort the connection" for the soft errors like many ICMP 
unreachables. Such soft errors might be for transient conditions, and 
the intention is that TCP survive such transients.
Thus I think your (hopefully hypothetical) example indicates bad TCP 
implementation.

> ICMPv6 unreachable might cause the device to tear down the tunnel.

Ditto.

> the root bridge to send out topology notification BPDU's. That's a long

Sure. And a protocol like BGP might be tuned to timeout in less time 
than 35 seconds (whether or not that is a good timeout isn't my point). 
But that wouldn't be impacted whether IPv4 or IPv6 is used.
I think there is good operational understanding of how to tune things 
for IPv4, and that shouldn't have to change as IPv6 gets more widely 
deployed. I don't see the ICMP address unreachable errors due to NUD 
failures making a difference, because TCP must not reset the connection 
as a result of them.
But I do see the three second NUD hard limit as being a case where 
IPv6/ND is different than IPV4/ARP for no good reason.

> hit their own time outs and take their own fail over actions?

There is a huge difference between making local decisions based on NUD 
failures (such as a host using a different default router) and trying to 
interpret the ICMP address unreachable from afar. My take is that the 
ICMP address unreachable is a useful diagnostic tool - ping and 
traceroute will report it - but it is a soft error hence it doesn't make 
sense to have TCP or other protocols give up as a result of such ICMP 
errors.

> of deja vu, and impending packet storms)

I don't understand why you think all the nodes on a link need to be 
coordinated in such a way; the Internet protocols are designed to be 
robust and not assume that all the nodes have the same code and tuning. 
For instance, we don't require that ARP or TCP on all the nodes on a 
link to have the same timer values, and things work just fine.

    Erik

---

From: Ray Hunter <[email-masked]>
Date: Wed, 25 May 2011 19:58:41 +0200

Thanks once again for replying

I think the differences are indeed down to bad implementations rather 
than specification.

Although it is acknowledged in RFC5461 section 4.1 as late as 2009 that 
there are non-compliant implementations in the field where TCP does 
react to soft errors. Some of the kit I work on is a lot older than that.

Also my operational experience of ARP says that dead gateway protection 
is not widely supported, and black holing was more the norm than the 
exception.

Erik Nordmark wrote:
> robust and not assume that all the nodes have the same code and 
> tuning. For instance, we don't require that ARP or TCP on all the 
> nodes on a link to have the same timer values, and things work just fine.
>
>    Erik 

>> rather than a "per node" parameter.
The last point was simply one from an operational perspective. Forgive 
me for being such a low level guy.

[side track]
One of my grumbles about IPv6 is that network managers just don't have 
the standard/generic tools to be able to tune the behavior of end nodes 
effectively. There are quite a lot of host behaviors that are set with 
local preferences and have default values, but which are not coordinated 
across implementations. e.g. dare I mention SLAAC v. DHCPv6.

As a network admin that's just a nightmare to manage in an environment 
where there are multiple operating systems, guest end nodes, traveling 
users, new nodes, old implementations...... half of the implementations 
are performing in a way that isn't suitable for your network, but you 
might not have admin rights on that end node, and there's no way to 
provide the end node with a hint of correct behavior.

Think of a network using "Bring Your Own Device" policy where you do not 
have any admin control e.g. no Active Directory.

There's seems to be no (effective) way of network equipment being able 
to signal to end nodes what is appropriate behavior for your particular 
network, compared to the simple existing tools like DHCPv4 options + 
extensions we are already have today. I'm sure certain SLAAC evangelists 
will tell me it's no business of mine to try to manage this at all, and 
self-configuration is the future. But never mind.
[/side track]

I've just read the RFC covering the (very interesting) mesh under / 
route over mechanism used in 6LoWPAN 
http://tools.ietf.org/html/draft-ietf-6lowpan-nd-16 . Very cool stuff.

Even there it was a requirement that all nodes taking part in the 
network behave the same way.
 > The applicability of this specification is limited to LoWPANs where 
all nodes on the subnet implement these optimizations in a homogeneous way.

So if the point of this draft is really to limit multicast, then from an 
operational perspective don't you want ALL nodes on a link to avoid 
using multicast as much as possible?

So if the point of this draft is really to avoid operational problems 
with STP thrashing, then from an operational perspective don't you want 
ALL nodes on a link to avoid timing out too fast as much as possible?

And how do the end nodes know what is appropriate operational behavior 
on this particular link? Out of scope of the draft ........ ?

If that's really true that the end nodes do not have to behave the same 
way then I do not understand why the Reachable Time and the Retransmit 
Timer are sent in an RA message.

Put it the other way around way: I don't understand then why it was 
considered so important that all nodes used the same values for 
Reachable Time and Retransmit Timer (for NUD), if it now isn't 
considered important that they even use the same retry mechanism in the 
probe state, or for how long that state can last.

That's all I'm saying. If you perform link-level coordination for one 
set of parameters used by NUD, why not this particular one?

Also for debugging, it's just one more thing to look at on that sniffer 
trace when spending a weekend / evening debugging in a data centre (not 
my favorite hobby and something I try to avoid). So is a node not 
responding because it is using exponential NUD back off, or is it not 
responding because a ND message is being dropped due to spanning tree 
thrashing around, or is it not responding because the end node 
implementation is plain broken?

Hope this helps clarify where I'm coming from. It's not in any way a 
criticism of your draft, just a potential pointer to how it could be 
"improved" from the perspective of someone operational.

regards,
RayH

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 25 May 2011 20:44:23 +0200

On May 25, 2011, at 19:58, Ray Hunter wrote:

> I've just read the RFC covering the (very interesting) mesh under / route over mechanism used in 6LoWPAN http://tools.ietf.org/html/draft-ietf-6lowpan-nd-16 . Very cool stuff.

Thanks!

> Even there it was a requirement that all nodes taking part in the network behave the same way.
> > The applicability of this specification is limited to LoWPANs where all nodes on the subnet implement these optimizations in a homogeneous way.

The idea here is that this draft updates RFC 4944 (the IP-over-foo for IEEE 802.15.4, usually referred to as 6LoWPAN) so that *all* nodes on a 6LoWPAN MUST implement 6LoWPAN-ND as opposed to ND-classic.

> So if the point of this draft is really to limit multicast, then from an operational perspective don't you want ALL nodes on a link to avoid using multicast as much as possible?

Yes, that is one of the reasons why we didn't try to tackle the theoretical possibility of interoperation between nodes implementing 6LoWPAN-ND and other nodes implementing ND-classic on 6LoWPANs, because even listening for multicast is very expensive for some nodes there.

> So if the point of this draft is really to avoid operational problems with STP thrashing, then from an operational perspective don't you want ALL nodes on a link to avoid timing out too fast as much as possible?

There is no STP in 6LoWPAN, so that isn't a consideration for this draft.
Again, 6LoWPAN-ND is currently defined for non-Ethernet networks that are homogeneously 6LoWPAN-ND, i.e. 802.15.4 (and, most likely once their IP-over-foo documents stabilize, similar constrained networks such as BT-LE or DECT-ULE), 
However if people want to go ahead and do the work of adapting 6LoWPAN-ND (or some of its elements) to other networks,
e.g., Ethernet/WiFi, then one would have to open that can of worms.

Gruesse, Carsten

---

From: Ray Hunter <[email-masked]>
Date: Wed, 25 May 2011 22:48:34 +0200

Failed context switch on my part.

I was trying to use draft-ietf-6lowpan-nd-16 as an example of modified 
ND being applied to all nodes on a certain link type for a specific 
operational reason. (Yes I understand draft-ietf-6lowpan-nd-16 is 
specifically targeted at only 6lowpan nodes, although of course your 
6LBR will have to run both flavors of ND: one special type avoiding 
multicast on the 6lowpan interface, and the other on the "classic" 
interface)

The comments I made about ALL nodes should modify their behavior were 
meant to be directed at draft-nordmark-6man-impatient-nud-00.txt. 
Apologies for any confusion on the use of "this draft"

If you modify ND behavior for one node, from a purely operational 
perspective, it makes sense IMHO to make sure the modified ND behavior 
applies to all nodes on a particular link, so that you can obtain your 
desired operational result.

/Grüße zurück aus Niederlande :-)/
RayH