Subject: Re: [v6ops] Interesting problems with using IPv6

From: Brian E Carpenter <[email-masked]>
Date: Tue, 09 Sep 2014 09:59:53 +1200

I switched to the relevant list.

On 09/09/2014 06:33, Dale W. Carder wrote:
> Thus spake Brian E Carpenter ([email-masked]) on Mon, Sep 08, 2014 at 07:50:26AM +1200:
>> If they really are interesting problems, it might be more to the
>> point to analyse them over on v6ops. Given the number of large
>> IPv6 deployments that don't have such problems, it seems like
>> this particular deployment hit an unfortunate combination of
>> implementation issues. That is worth understanding (for example,
>> how large is the layer 2 network that leads to the MLD listener
>> report overload?).
> 
> Implementing MLD snooping for Solicited-Node multicast addresses 
> is probably a bad idea.
> 
> See: draft-pashby-magma-simplify-mld-snooping-01

OK, but I would also like to understand why we require
MLD messages for a Solicited-Node multicast address to
set Router Alert.

    Brian

---

From: Mark ZZZ Smith <[email-masked]>
Date: Mon, 8 Sep 2014 18:55:35 -0700

----- Original Message -----
> From: Brian E Carpenter <[email-masked]>
> To: Dale W. Carder <[email-masked]>
> Cc: IPv6 Operations <[email-masked]>; [email-masked]
> Sent: Tuesday, 9 September 2014, 7:59
> Subject: Re: [v6ops] Interesting problems with using IPv6
> 
>>>  implementation issues.

If this email is referring to this blog post:

http://blog.bimajority.org/2014/09/05/the-network-nightmare-that-ate-my-week/

then I think this particular network was already on the verge of catastrophic failure, and some of IPv6's ND differences were just the trigger to push it over that threshold.

According to the blog post, they might have:

- hardware errors
- software errors
- 'bridge loops' causing the control plane to overload

In some of these cases they've taken actions to try to resolve them, however the actions taken seem to be based on speculating what is happening rather than finding evidence to support the hypothesised cause before taking action.

For example, there isn't any evidence that they've actually determined that 'bridge loops' are occurring, found out what is causing them, and then taking measures to prevent them or at least reduce the chances of them occurring. If a 'bridge loop' can cause the control plane to overload, then adding anything to this network like IPv6 is likely to only make the problem worse.

I'd like to see them get to the bottom of and then fix these 'bridge loop' and other problems first before any value is placed on addressing their criticisms of IPv6. Other people have deployed IPv6 without these problems, so what is different between this network and everybody else's that doesn't have these sorts of problems?

 That is worth understanding (for example,

> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 09 Sep 2014 14:14:49 +1200

Mark,

My point is that it's worth *understanding* such problems and then
perhaps writing up operational or implementation recommendations.
Exactly what the v6ops charter says we should do, in fact.

"1. Solicit input from network operators and users to identify
operational issues with the IPv4/IPv6 Internet, and
determine solutions or workarounds to those issues. These issues
will be documented in Informational or BCP RFCs, or in
Internet-Drafts."

Or recommend protocol changes if they might help, which is why I
want to understand the value of Router Alert in MLD messages for
Solicited-Node multicast addresses. Or should we revive
draft-pashby-magma-simplify-mld-snooping-01?

    Brian

---

From: Fernando Gont <[email-masked]>
Date: Tue, 09 Sep 2014 01:10:43 -0300

> MLD messages for a Solicited-Node multicast address to

Because in theory the multicast router needs to process the MLD message
to build its forwarding table....  -- or are you arguing that since the
Dst Addr "includes" the router, the router should process the MLD
message anyway?

Thanks,
-- 
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 09 Sep 2014 19:20:53 +1200

On 09/09/2014 16:10, Fernando Gont wrote:
> to build its forwarding table....

Why, for the Solicited-Node group, which is only meaningful on the link
from which the MLD message arrives?

(NBMA is different of course, but that is not the scenario in the blog.)

  -- or are you arguing that since the
> Dst Addr "includes" the router, the router should process the MLD

In its role as a node on the same link, participating in ND on that
link, yes. In its role as a forwarding device, surely it should ignore
traffic to the Solicited-Node address? So Router Alert is not
needed. Or am I missing something?

    Brian

---

From: Philip Homburg <[email-masked]>
Date: Tue, 09 Sep 2014 11:27:44 +0200

>needed. Or am I missing something?

Something else is weird as well. MLD-snooping involves a trade-off. The benefits of
MLD-snooping are reduced multicast traffic. The costs are the extra hardware needed
to do the filtering and increased complexity.

In most networks, increased complexity causes headaches and downtime, so the savings
better be significant.

Though ND is quite chatty, even in big networks it is not going to overload a
100 Mbit/s ethernet link or a host that is less than a decade old and not
significantly under powered.

One type of link that is certainly not going to be affected is backbone links between
switches.

So where does MLD-snooping make sense? On links from a switch to either a rather
slow network technology, like 2.4 GHz wifi, or particularly low-end devices, like
small ARM boards.

So the weird thing in the story is that core switches get overloaded processing MLD
even though they can just as well just forward multicast over a spanning tree.

Somehow the network architecture at CSAIL ended up with central processing of MLD and
a centralized TCAM for filtering instead of just filtering at the edges where it
makes sense.

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 09 Sep 2014 10:42:54 +0100

On 09/09/2014 02:55, Mark ZZZ Smith wrote:
> I'd like to see them get to the bottom of and then fix these 'bridge
> loop' and other problems first before any value is placed on addressing
> their criticisms of IPv6. Other people have deployed IPv6 without these
> problems, so what is different between this network and everybody else's
> that doesn't have these sorts of problems?

from my reading of the post, the author+vendor determined that the bridge 
loops happened because the STP process was starved of resources on the 
switch CPUs.

This happened because the switch CPUs were overloaded with mld report 
packets due to end hosts on the extended L2 network replying to MLD 
all-groups queries every 150 seconds.

There were large numbers of MLD reports because there was a large number of 
ipv6 privacy addresses on the network.  The network operators did not have 
admin control of the end hosts and could therefore not stop them from using 
privacy addresses.

Nick

---

From: Tore Anderson <[email-masked]>
Date: Tue, 09 Sep 2014 11:52:24 +0200

* Nick Hilliard

> The network operators did not have admin control of the end hosts and
> could therefore not stop them from using privacy addresses.

That's easily accomplished by setting Autonomous=0 in RA and using
something else for address assignment (like DHCPv6).

No SLAAC => no privacy extensions.

Tore

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 09 Sep 2014 11:12:42 +0100

> No SLAAC => no privacy extensions.

yes, this was what they decided: that SLAAC would not work on their 
configuration and that if they were going to enable ipv6 in the future, it 
would probably be on the basis of dhcpv6, with no autoconfiguration.

Nick

---

From: Lorenzo Colitti <[email-masked]>
Date: Tue, 9 Sep 2014 19:29:13 +0900

> all-groups queries every 150 seconds.

So the switch was configured to send all-groups queries to all hosts, but
did not have the CPU power to process them, and could not, or was not
configured, to rate-limit them.

News at 11: building a network beyond the capabilities of the gear that
runs it will result in failure.

That does not mean the protocol is flawed. ARP could have done the same
thing, and in fact that was a common problem many years ago... except that
these days ARP is usually processed on the fast path and it doesn't matter.

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Tue, 9 Sep 2014 10:43:07 +0000

Agreed.  Just because one vendor’s switch melted an ipv6 network is not enough justification to change protocols.

Hemant

From: v6ops [mailto:[email-masked]] On Behalf Of Lorenzo Colitti
Sent: Tuesday, September 09, 2014 6:29 AM
To: Nick Hilliard
Cc: [email-masked]; IPv6 Operations
Subject: Re: [v6ops] Interesting problems with using IPv6

On Tue, Sep 9, 2014 at 6:42 PM, Nick Hilliard <[email-masked]<mailto:[email-masked]>> wrote:
This happened because the switch CPUs were overloaded with mld report packets due to end hosts on the extended L2 network replying to MLD all-groups queries every 150 seconds.

So the switch was configured to send all-groups queries to all hosts, but did not have the CPU power to process them, and could not, or was not configured, to rate-limit them.

News at 11: building a network beyond the capabilities of the gear that runs it will result in failure.

That does not mean the protocol is flawed. ARP could have done the same thing, and in fact that was a common problem many years ago... except that these days ARP is usually processed on the fast path and it doesn't matter.

---

From: Chuck Anderson <[email-masked]>
Date: Tue, 9 Sep 2014 10:22:27 -0400

Disagree.  

In common practice, ARP doesn't need to scale to hundreds of IPv4
addresses per host.  Unforutnately, IPv6 SLAAC with Privacy Addressing
means that ND and hence MLD does need to scale much larger than ARP
ever did.  The cure of using Solicited Node multicast for ND is worse
than the disease of using broadcast for ARP.

It is not reasonable to expect a switch to scale to maintaining state
for 300 * (# of hosts connected) Solicited-Node multicast groups.
15,000 MLD groups for 48 hosts on a 48 port switch is unreasonable.
90,000 MLD groups for a 300 port switch stack is also unreasonable.
Most switches cannot handle that many IPv4 multicast groups--expecting
them to handle that many IPv6 multicast groups is unreasonable.

How many MLD reports per second have to be processed by the switch CPU
given reports every 150 seconds by each of the many Solicited Node
multicast groups on every connected host?  100 pps?  600 pps?

What are the scaling numbers for IGMP groups and MLD groups and how
many IGMP/MLD reports can be processed per second on commonly
available access-layer switches?

Having designed a ubiquitous protocol 16 years ago that can't be
implemented reasonably cheaply on current hardware is an operational
problem that can and should be dealt with by tweaking the protocol in
some cases.  I think the protocol change proposals put forward in the
article are a quite reasonable starting point for discussing how to
mitigate this issue and shouldn't be dismissed out of hand.

---

From: Fernando Gont <[email-masked]>
Date: Tue, 09 Sep 2014 11:16:47 -0300

> from which the MLD message arrives?

Then, let me change the question: Why do I need MLD for *this*?

We probably use MLD because "If you use multicast, you use MLD". Truth
is that, *unless your switch does MLD snooping* (and hence you *need*
MLD, or else your packets will not flow around), you could completely
kill MLD, and ND would still work just fine.

Not to mention that there are nodes that default t running MLDv2 *for
this* (way overkill, IMO)

Thanks,

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Tue, 9 Sep 2014 15:17:22 +0000

There are several mitigations that exist.  

a. The switch should have been designed for rate-limiting beyond a certain number of MLD reports/sec.  
b. Use DHCPv6 in the network to dispense with privacy addresses and SLAAC.  
c. ARP could be processed in the fastpath because ARP uses a different ethertype than the IP packet.  The different ethertype made the fastpath filter ARP pkts with relative ease.  One could consider defining a new ethertype for IPv6 ND and or MLD and look into moving certain IPv6 processing to the fastpath.

I do agree that a 48-port switch should not have to deal with 15k MLD groups.  ND uses multicast and multicast uses MLD.  Changing multicast and/or ND needs more discussion.  

Hemant

-----Original Message-----
From: v6ops [mailto:[email-masked]] On Behalf Of Chuck Anderson
Sent: Tuesday, September 09, 2014 10:22 AM
To: [email-masked]
Subject: Re: [v6ops] Interesting problems with using IPv6

Disagree.  

In common practice, ARP doesn't need to scale to hundreds of IPv4 addresses per host.  Unforutnately, IPv6 SLAAC with Privacy Addressing means that ND and hence MLD does need to scale much larger than ARP ever did.  The cure of using Solicited Node multicast for ND is worse than the disease of using broadcast for ARP.

It is not reasonable to expect a switch to scale to maintaining state for 300 * (# of hosts connected) Solicited-Node multicast groups.
15,000 MLD groups for 48 hosts on a 48 port switch is unreasonable.
90,000 MLD groups for a 300 port switch stack is also unreasonable.
Most switches cannot handle that many IPv4 multicast groups--expecting them to handle that many IPv6 multicast groups is unreasonable.

How many MLD reports per second have to be processed by the switch CPU given reports every 150 seconds by each of the many Solicited Node multicast groups on every connected host?  100 pps?  600 pps?

What are the scaling numbers for IGMP groups and MLD groups and how many IGMP/MLD reports can be processed per second on commonly available access-layer switches?

Having designed a ubiquitous protocol 16 years ago that can't be implemented reasonably cheaply on current hardware is an operational problem that can and should be dealt with by tweaking the protocol in some cases.  I think the protocol change proposals put forward in the article are a quite reasonable starting point for discussing how to mitigate this issue and shouldn't be dismissed out of hand.

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Tue, 9 Sep 2014 17:41:39 +0200

> 90,000 MLD groups for a 300 port switch stack is also unreasonable.

Not delving into the whole "why in today-scale network we need to
maintain separate groups per each solicited multicast" argument, let's
just assume there are some noticeable gains there and focus on the
number of groups.

1) http://tools.ietf.org/html/rfc4941#section-3.3 says:

   1.  Process the Prefix Information Option as defined in [ADDRCONF],
       either creating a new public address or adjusting the lifetimes
       of existing addresses, both public and temporary.  If a received
       option will extend the lifetime of a public address, the
       lifetimes of temporary addresses should be extended, subject to
       the overall constraint that no temporary addresses should ever
       remain "valid" or "preferred" for a time longer than
       (TEMP_VALID_LIFETIME) or (TEMP_PREFERRED_LIFETIME -
       DESYNC_FACTOR), respectively.  The configuration variables
       TEMP_VALID_LIFETIME and TEMP_PREFERRED_LIFETIME correspond to
       approximate target lifetimes for temporary addresses.

One way to interpret this text is that with the defaults, a host with
SLAAC and temporary addresses will have: 1 LL + 1 SLAAC + 7 temp
addresses = 9.

This gives 1.5 order of magnitude less groups than in the example you
cite - a not terribly unreasonable number, though it's still a 9x
amount.

The TEMP_VALID_LIFETIME and TEMP_PREFERRED_LIFETIME are host-coded,
the network (for better or worse) appears to have not much say about
them.

2) The "300 addresses" situation might look like either the defaults
were changed, or the stack is being "helpful" as described in
http://tools.ietf.org/html/rfc4941#section-6:

   An implementation might want to keep track of which addresses are
   being used by upper layers so as to be able to remove a deprecated
   temporary address from internal data structures once no upper layer
   protocols are using it (but not before).  This is in contrast to
   current approaches where addresses are removed from an interface when
   they become invalid [ADDRCONF], independent of whether or not upper
   layer protocols are still using them.  For TCP connections, such
   information is available in control blocks.  For UDP-based
   applications, it may be the case that only the applications have
   knowledge about what addresses are actually in use.  Consequently, an
   implementation generally will need to use heuristics in deciding when
   an address is no longer in use.

Would this mean the host has a connection that is lasting 300 days and
is using the same temporary address ? This does not look too
temporary, so arguably that application should not be using temporary
addresses (Also: interesting that e.g. Linux limits the default # to
16 - cf.: http://oss.sgi.com/archives/netdev/2004-01/msg01002.html
which is still true in the Ubuntu 14.04.)

BTW, talking about the applications, they should be able to select
whether they need the temporary addresses or not: rfc5014. Do they
intentionally select the temporary addresses on the long-lasting
connections ?

Now, let me argue with myself:

1) IPv6 requiring 9x time the IPv4 addresses is still quite a stretch.
If someone needs a long-lasting session, they should use a stable
address.

2) Does application have a way to say "I do not want temporary addresses" ?
(cf: https://bugs.launchpad.net/ubuntu/+source/procps/+bug/1068756)
Seems not today ?

3) Also, http://tools.ietf.org/html/rfc4941#section-3.6 clearly says:

" In addition, some applications may not behave
   robustly if temporary addresses are used and an address expires
   before the application has terminated, or if it opens multiple
   sessions, but expects them to all use the same addresses.
   Consequently, the use of temporary addresses SHOULD be disabled by
   default in order to minimize potential disruptions.  Individual
   applications, which have specific knowledge about the normal duration
   of connections, MAY override this as appropriate."

So, if the hosts were gentle with the number of addresses they try to
use, apps with long-lasting connections were to disable the temporary
addresses and the server fleet had the temporary addresses disabled
altogether, SLAAC (and, where appropriate, temporary addresses) would
not be so bad at all.

Of course, the amount of manual work this requires today means that
the A-bit is turned off and the DHCPv6 is being used. Which may be
perfectly okay - but if it is not, and we do assert that "An RA with
prefix having an A-bit set and default settings, should not cause the
10x expansion to the resource requirements compared to Legacy IP",
seems like some sort of a correction in default stacks/apps behavior
with respect to temporary addresses could be useful.

--a

---

From: "Hemant Singh (shemant)" <[email-masked]>
Date: Tue, 9 Sep 2014 15:58:42 +0000

Andrew,

Great input from you.   However, there is one catch.  A basic switch works at layer-2 and may not have its implementation communicate with upper transport layers such as TCP and UDP.  The switch is merely snooping MLD.  Then the switch should just support never letting its cpu go higher than 40% and rate limit processing in the switch punt path.

Hemant

-----Original Message-----
From: v6ops [mailto:[email-masked]] On Behalf Of Andrew ?? Yourtchenko
Sent: Tuesday, September 09, 2014 11:42 AM
To: Chuck Anderson
Cc: [email-masked]
Subject: Re: [v6ops] Interesting problems with using IPv6

Not delving into the whole "why in today-scale network we need to maintain separate groups per each solicited multicast" argument, let's just assume there are some noticeable gains there and focus on the number of groups.

One way to interpret this text is that with the defaults, a host with SLAAC and temporary addresses will have: 1 LL + 1 SLAAC + 7 temp addresses = 9.

This gives 1.5 order of magnitude less groups than in the example you cite - a not terribly unreasonable number, though it's still a 9x amount.

The TEMP_VALID_LIFETIME and TEMP_PREFERRED_LIFETIME are host-coded, the network (for better or worse) appears to have not much say about them.

2) The "300 addresses" situation might look like either the defaults were changed, or the stack is being "helpful" as described in
http://tools.ietf.org/html/rfc4941#section-6:

Would this mean the host has a connection that is lasting 300 days and is using the same temporary address ? This does not look too temporary, so arguably that application should not be using temporary addresses (Also: interesting that e.g. Linux limits the default # to
16 - cf.: http://oss.sgi.com/archives/netdev/2004-01/msg01002.html
which is still true in the Ubuntu 14.04.)

BTW, talking about the applications, they should be able to select whether they need the temporary addresses or not: rfc5014. Do they intentionally select the temporary addresses on the long-lasting connections ?

1) IPv6 requiring 9x time the IPv4 addresses is still quite a stretch.
If someone needs a long-lasting session, they should use a stable address.

So, if the hosts were gentle with the number of addresses they try to use, apps with long-lasting connections were to disable the temporary addresses and the server fleet had the temporary addresses disabled altogether, SLAAC (and, where appropriate, temporary addresses) would not be so bad at all.

Of course, the amount of manual work this requires today means that the A-bit is turned off and the DHCPv6 is being used. Which may be perfectly okay - but if it is not, and we do assert that "An RA with prefix having an A-bit set and default settings, should not cause the 10x expansion to the resource requirements compared to Legacy IP", seems like some sort of a correction in default stacks/apps behavior with respect to temporary addresses could be useful.

--a

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Tue, 9 Sep 2014 18:22:28 +0200

Hemant,

On 9/9/14, Hemant Singh (shemant) <[email-masked]> wrote:
> Andrew,
>
> Great input from you.   However, there is one catch.  A basic switch works
> at layer-2 and may not have its implementation communicate with upper
> transport layers such as TCP and UDP.  The switch is merely snooping MLD.
> Then the switch should just support never letting its cpu go higher than 40%
> and rate limit processing in the switch punt path.

"should", absolutely!

But safeguarding by other means to avoid the fireworks and explosions
if this "should" is violated due to human error (designer, programmer,
operator, or user) seems like a prudent engineering choice.

Besides, the multicast group explosion and MLD traffic is just one of
the facets related to excessive temporary addresses that just happens
to be in the spotlight today, but it is not the only one.

--a

> Not delving into the whole "why in today-scale network we need to maintain
> separate groups per each solicited multicast" argument, let's just assume
> there are some noticeable gains there and focus on the number of groups.
>
> One way to interpret this text is that with the defaults, a host with SLAAC
> and temporary addresses will have: 1 LL + 1 SLAAC + 7 temp addresses = 9.
>
> This gives 1.5 order of magnitude less groups than in the example you cite -
> a not terribly unreasonable number, though it's still a 9x amount.
>
> The TEMP_VALID_LIFETIME and TEMP_PREFERRED_LIFETIME are host-coded, the
> network (for better or worse) appears to have not much say about them.
>
> 2) The "300 addresses" situation might look like either the defaults were
> changed, or the stack is being "helpful" as described in
> Would this mean the host has a connection that is lasting 300 days and is
> using the same temporary address ? This does not look too temporary, so
> arguably that application should not be using temporary addresses (Also:
> interesting that e.g. Linux limits the default # to
> BTW, talking about the applications, they should be able to select whether
> they need the temporary addresses or not: rfc5014. Do they intentionally
> select the temporary addresses on the long-lasting connections ?
>
> So, if the hosts were gentle with the number of addresses they try to use,
> apps with long-lasting connections were to disable the temporary addresses
> and the server fleet had the temporary addresses disabled altogether, SLAAC
> (and, where appropriate, temporary addresses) would not be so bad at all.
>
> Of course, the amount of manual work this requires today means that the
> A-bit is turned off and the DHCPv6 is being used. Which may be perfectly
> okay - but if it is not, and we do assert that "An RA with prefix having an
> A-bit set and default settings, should not cause the 10x expansion to the
> resource requirements compared to Legacy IP", seems like some sort of a
> correction in default stacks/apps behavior with respect to temporary
> addresses could be useful.
>
> --a
>

---

From: Brian Haberman <[email-masked]>
Date: Tue, 09 Sep 2014 13:09:06 -0400

> Then, let me change the question: Why do I need MLD for *this*?

The MLD specs say that an MLD Report is sent for every multicast group
joined except the All-Nodes multicast address.

The use of MLD Reports for essentially all multicast addresses was done
to facilitate this very type of snooping functionality.  The use of
Router Alerts in MLD messages is due to MLDv1 (and IGMPv2) using the
group address as the IP destination rather than the All-Routers
multicast address.

> kill MLD, and ND would still work just fine.

Sure, since NDP is link-local.  The drawback is what happens if your
network is using RFC 4541 snooping that relies on seeing those MLD
messages to build forwarding/filtering tables?

> this* (way overkill, IMO)

Why is MLDv2 overkill?

Brian

---

From: 神明達哉 <[email-masked]>
Date: Tue, 9 Sep 2014 11:57:44 -0700

At Tue, 09 Sep 2014 09:59:53 +1200,

> MLD messages for a Solicited-Node multicast address to

I don't know the answer, but my guess is that when the original MLD
was designed in RFC 2710 they simply didn't have that question.  In
fact, we were not aware of various implications with MLD messages for
solicited-node multicast addresses for quite some time, which made us
include some specific discussion on this topic in RFC 4862 (Section
5.4.2).

--
JINMEI, Tatuya

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 10 Sep 2014 08:40:36 +1200

>> Then, let me change the question: Why do I need MLD for *this*?

I think Brian Haberman's reply shows why that is the wrong question.
You need MLD for every multicast group, including a solicited-node
group, and if you insist on MLD snooping in the bridges (let's not
obfuscate by calling them switches) then you need to snoop every
solicited-node group.

My question is orthogonal to MLD snooping: why do we require router-alert
for MLD messages referring to a solicited-node group, since it by
definition is limited to a single L2 link (even if that link is
split up by bridges)?

I tend to think this requirement is an error in the MLD spec.

    Brian C

> ------------------------------------------------------------------------
>

---

From: Fernando Gont <[email-masked]>
Date: Tue, 09 Sep 2014 21:31:28 -0300

> with respect to temporary addresses could be useful.

FWIW:
<https://tools.ietf.org/id/draft-gont-6man-managing-privacy-extensions-01.txt>

Thanks,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Fernando Gont <[email-masked]>
Date: Tue, 09 Sep 2014 21:40:17 -0300

Hi, Brian,

> messages to build forwarding/filtering tables?

Exactly. That's why I said "unless...".

>> this* (way overkill, IMO)

If you're just going to use MLD for ND, MLDv2 is unnecessarily complex
-- whole different game if you're using non-link-local multicast, of course.

That said, I seem to recall the node reqs RFC requires MLDv1, rather
than MLDv2?

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Tue, 09 Sep 2014 21:47:37 -0300

> definition is limited to a single L2 link (even if that link is

.. to avoid them being a special case?  -- i.e., all MLD packets carry a
Router Alert option. And, actually, an MLD report could potentially
include groups other than solicited-node (so you probably wouldn't want
to turn the "include a router-alert" on and off).

Thanks,

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 10 Sep 2014 14:16:15 +1200

On 10/09/2014 12:47, Fernando Gont wrote:
> Router Alert option. 

Yes, it would be an exception. But we know that HbH options in general
and Router Alert in particular are a serious performance issue, and
that all links carry this particular kind of MLD traffic, so an
exception seems like something to be discussed.

> And, actually, an MLD report could potentially

Sure, the exception would have to be for messages containing only
solicited-nodes data.

   Brian

---

From: Fernando Gont <[email-masked]>
Date: Wed, 10 Sep 2014 03:52:46 -0300

Hi, Brian,

> exception seems like something to be discussed.

Agreed.

Cheers,

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Wed, 10 Sep 2014 12:18:29 +0200

> exception seems like something to be discussed.

How does this interact with the case where a "pure L2" switch within
the link would want to snoop MLD ? Presumably, today the L2 switch
would see the "router alert" and know that it needs to process this
traffic.

If this were to be removed, then it would be much trickier to direct
the traffic to the slow path in that scenario, wouldn't it ?

--a

---

From: Brian Haberman <[email-masked]>
Date: Wed, 10 Sep 2014 08:15:29 -0400

Hi Brian,

> I tend to think this requirement is an error in the MLD spec.

This is an interesting point.  The use of Router Alert was carried
through the various versions of IGMP/MLD from the original IGMPv1 spec.
 Since IGMPv1, IGMPv2, and MLDv1 sent Reports to the group address, the
Router Alert was meant to signal to routers that they needed to process
the packet.  There are two things that I think have changed.

1) Routers tend to trigger processing based on the Protocol or Next
Header field.  I think it would be worthwhile to verify with router
vendors that they do not use the Router Alert to trigger processing.

2) IGMPv3 and MLDv2 now send Reports are sent to an
all-multicast-routers multicast address.

Regards,
Brian H

---

From: Brian Haberman <[email-masked]>
Date: Wed, 10 Sep 2014 08:19:17 -0400

> -- whole different game if you're using non-link-local multicast, of course.

I disagree.  When sending MLD Reports for NDP, you don't need to involve
the complexity of MLDv2 source filters.  You simply send a group-only
Report message.

> That said, I seem to recall the node reqs RFC requires MLDv1, rather

Seems orthogonal to the discussion since MLDv2 can act in MLDv1 mode for
these link-local multicast groups.  Additionally, RFC 5790 has
simplified MLDv2 to avoid some complexities.

Regards,
Brian

---

From: Brian Haberman <[email-masked]>
Date: Wed, 10 Sep 2014 08:26:38 -0400

Hi Andrew,

> the traffic to the slow path in that scenario, wouldn't it ?

I think that depends.  As I mentioned in an earlier e-mail, some routers
simply use the IPv4 Protocol field or the IPv6 Next Header field to see
if they need to process the packet.  It is unclear (to me) how many
implementations still use the Router Alert to determine if the packet
needs local processing.  Maybe some vendors could chime in?

Regards,
Brian

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Wed, 10 Sep 2014 15:40:47 +0200

> <https://tools.ietf.org/id/draft-gont-6man-managing-privacy-extensions-01.txt>

Yes, we even discussed this draft with you at some point :-).

My opinion (while agreeing with the spirit of this draft, somewhat) is
still - it is too late.

All it would do today is split the hosts into two groups - who
understand this bit, and those who do not, adding more difficulty to
the life of network administrator, who, having turned the knob "no
temp address please" would expect one behavior, and get a mix of two
in the best case.

(This all puts aside any philosophical arguments of whether the
network should dictate the nodes whether they can use the temporary
"privacy" addresses or not: temporary addresses go against the
incentives of basically *any* sufficiently large network
administrator, so this bit would be almost universally on).

If the host OSes/apps want to be sensible with the usage of temporary
addresses, they can do so without any flag.

--a

---

From: Fernando Gont <[email-masked]>
Date: Wed, 10 Sep 2014 18:34:15 -0300

Hi, Brian,

> the complexity of MLDv2 source filters.  You simply send a group-only

That would still means that you'd have to implement it. For small
devices, that might be an issue. And, given how many implementations got
ND (which is rather way simpler) wrong one way or another, I wouldn't be
surprised to see some interesting behaviors if one were to play with
(crafted) MLDv3 packets.

Cheers,

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 11 Sep 2014 13:33:15 +1200

> needs local processing.  Maybe some vendors could chime in?

I have the impression that many boxes will look at *any* ICMPv6
packet, because in their firewall role they filter ICMPv6.
In that case the Router Alert is certainly redundant. However,
we do need vendor feedback, as you say.

   Brian C

---

From: joel jaeggli <[email-masked]>
Date: Wed, 10 Sep 2014 19:41:30 -0700

> we do need vendor feedback, as you say.

Beware equating filtering with with inspection by the control plane,
many asic based forwarding platforms ingress acl matches are never slow
path. they wouldn't be useful for dos protection if they had to hit a cpu.

---

From: Brian Haberman <[email-masked]>
Date: Thu, 11 Sep 2014 08:35:59 -0400

Hi Joel,

> path. they wouldn't be useful for dos protection if they had to hit a cpu.

Agreed.  What I am looking for is vendor input on whether Router Alerts
are still needed for IGMP/MLD messages.

Regards,
Brian

---

From: Ray Hunter <[email-masked]>
Date: Fri, 12 Sep 2014 10:07:55 +0200

> than the disease of using broadcast for ARP.
The obvious history here is that Ethernet was a big yellow cable, 
multicast was "for free", an (ARP) broadcast disturbed all end nodes 
causing them to grind to a halt on large L2 nets, whereas a multicast 
only interrupted a subset of the end nodes.
Now Ethernet is effectively a point to point protocol (wired), multicast 
is certainly not for free (silicon) nor reliable (wireless).
Agreed.
> mitigate this issue and shouldn't be dismissed out of hand.

I read the article but did not see the specific recommended tweaks.

Can you enumerate?

One thing that might be an operational workaround (via OS patch) would 
be to update 4941 RID generation Section 3.2.12 to add Step 4a so that 
the candidate privacy addresses all hash to a limited subset of MLD 
group(s) per host that are used for solicited-node multicast address 
(RFC 4291 section 2.7.1) e.g. max of 16 MLD active groups per host out 
of the range FF02:0:0:0:0:1:FFXX:XXXX. As privacy addresses expire, new 
solicited-node multicast address groups could be activated.

That would avoid the large number of multicast update messages per host. 
It would also put an upper limit of multicast groups a L2 switch would 
have to support to n * number of hosts in the L2 domain.

Downside is that there is obviously less privacy address space to brute 
force track at any one time.

The more fundamental question is whether multicast is the correct way to 
resolve addresses in modern L2 networks.
Or whether on certain link technologies, a protocol that registers hosts 
over unicast to one or more L3<->L2 nameservers might be more 
appropriate (as per 6lo), but that is beyond v6ops.

>> On Tue, Sep 9, 2014 at 6:42 PM, Nick Hilliard<[email-masked]<mailto:[email-masked]>>  wrote:

-- 
Regards,
RayH

---

From: Erik Nordmark <[email-masked]>
Date: Fri, 12 Sep 2014 08:49:15 -0700

> from which the MLD message arrives?

Correct. That is true for all the link-local groups. The only reason to 
report link-local groups is to enable MLD snooping switches. Routers 
should only care about groups scope > link-local. Thus we can remove 
router-alert from MLD reports for link-local groups.

Another sub-optimal aspect is that when a router sends a MLD query, it 
gets responses for all groups including link-ocal. Only an MLD query 
from a snooping switch would need to elicit responses for link-local 
groups. (That would imply adding some flag to the MLD query which might 
not be worth-while at this point in time.)

    Erik

---

From: Owen DeLong <[email-masked]>
Date: Fri, 12 Sep 2014 15:33:52 -0700

> than the disease of using broadcast for ARP.

Privacy addressing is the true disease here, not ND. Solicited Node multicast actually isn’t all that bad in any sane implementation. ARP has certainly created its share of problems over the years.

However, I’ve never seen privacy addressing result in hundreds of addresses per host. That would have to be a pretty broken implementation or some really strange values for preferred/valid.

If you use the (relatively standard) 3600/86400, then any addresses that are more than a day old should get removed from the interface.
Assuming you generate a new privacy address every hour, that shouldn’t be more than ~25 addresses per interface at any given time.

> for 300 * (# of hosts connected) Solicited-Node multicast groups.

Nor does any rational use of SLAAC+Privacy addresses result in anything near that.

> 15,000 MLD groups for 48 hosts on a 48 port switch is unreasonable.

25*48 = 1,200, not 15,000. Even if we double that, it’s still 2,400 max.

A potential improvement could be if privacy addressing created a persistent random lower 24 bits and only rehashed the upper 40 bits of the suffix with each address rotation. In that way, you’d only have ~2 ND multicast groups per host.

> them to handle that many IPv6 multicast groups is unreasonable.

Again, this is not anything close to a real world scale for the situation. Using hyperbole derived numbers to make things sound bad enough that you can blame the protocol for a situation that is much more directly the result of bad network design is absurd.

> multicast groups on every connected host?  100 pps?  600 pps?

Let’s assume a 300 node subnet (rather large by most standards). Let’s further assume that you have 50 SNMGs per host with zero overlap.
That’s a total of 15,000 SNMGs network wide which, divided by 150 seconds results in a PPS rate of 100. Note, this is double the actual anticipated number of SNMGs per host and that zero overlap is a worst case scenario. A switch stack supporting 300 nodes shouldn’t have any problem at all forwarding 100PPS.

MLD snooping of ND traffic is a pretty bad idea and there’s really no good reason for the switch to not simply flood ND traffic.

> available access-layer switches?

You probably aren’t running a 300-node subnet on lower end common access-layer switch hardware. It’s going to require something higher end than that.

> mitigate this issue and shouldn't be dismissed out of hand.

I think that the protocol can be implemented reasonably cheaply on current hardware if you don’t design your network so poorly that the fact it hasn’t collapsed in on itself in IPv4 is a minor miracle. Lots of people have very large IPv6 networks running just fine on a  pretty wide variety of commodity hardware at this point.

The OP makes clear a number of places where his network design is an utter failure waiting to happen, then goes on to blame IPv6 based on little more than speculation about the cause.

Smaller layer 2 zones would greatly improve his situation. Turning off privacy addressing would also help. (This is, as has been pointed out, under the control of the network operator as he can set the M bit, turn off the A bit, and give out DHCP addresses as an example.)

I suspect there are a number of other viable solutions.  OTOH, modifying ipv6 to support such an environment seems a fools errand to me.

Owen

---

From: Chuck Anderson <[email-masked]>
Date: Fri, 12 Sep 2014 19:12:55 -0400

> Nor does any rational use of SLAAC+Privacy addresses result in anything near that.

Actually deployed networks are encountering 300+ privacy addresses on
some machines.  Is that a bug or by design of the OS vendor?  I don't
know.

> A potential improvement could be if privacy addressing created a persistent random lower 24 bits and only rehashed the upper 40 bits of the suffix with each address rotation. In that way, you’d onl…

Agreed.  Good idea.

> Again, this is not anything close to a real world scale for the situation. Using hyperbole derived numbers to make things sound bad enough that you can blame the protocol for a situation that is mu…

It isn't hyperbole.  It has been observed multiple times by different
people:

http://blog.bimajority.org/2014/07/16/ipv6-privacy-addresses-windows-just-say-no/

"it was a Windows machine, [...] I saw that it was responsible for the
vast majority of those multicast memberships: 350 different multicast
groups in all, nearly all of which were “solicited node” groups for
different IPv6 “privacy” addresses"

"I used Ubuntu as an example, but it is hardly the worst offender. We
have seen Windows machines with more than 300 IPv6 addresses"

And a thorough description of the problem and possible solutions:

http://inconcepts.biz/~jsw/ipv6_nd_problems_with_l2_mcast.pdf

> I think that the protocol can be implemented reasonably cheaply on current hardware if you don’t design your network so poorly that the fact it hasn’t collapsed in on itself in IPv4 is a minor mira…

The only reasonable way to avoid this problem today via network
design, is to design you network to use DHCPv6 instead of SLAAC or
turn off privacy addressing on all hosts.  Unfortunately for less
managed networks (the exact use case that SLAAC is designed for--hands
off) it isn't feasible to go to every host and manually turn off
privacy addressing.  That means the only practical way to solve this
problem is to only ever use DHCPv6.  This is a somewhat surprising
result, perhaps unanticipated by those who worked on ND & MLD, and
probably not desireable for those who champion SLAAC.

> Smaller layer 2 zones would greatly improve his situation. Turning off privacy addressing would also help. (This is, as has been pointed out, under the control of the network operator as he can set…

He doesn't have large Layer 2 zones.

> I suspect there are a number of other viable solutions.  OTOH, modifying ipv6 to support such an environment seems a fools errand to me.

Modifying ND is not without precedent.

---

From: Fernando Gont <[email-masked]>
Date: Fri, 12 Sep 2014 22:56:24 -0300

On 09/12/2014 07:33 PM, Owen DeLong wrote:
> A potential improvement could be if privacy addressing created a
> persistent random lower 24 bits and only rehashed the upper 40 bits
> of the suffix with each address rotation. In that way, you’d only
> have ~2 ND multicast groups per host.

That would break the sole purpose of privacy addresses. If the lower
24-bits are kept the same, an attacker can correlate the activities of a
node along time.

Thanks,

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 13 Sep 2014 14:59:53 +1200

Excuse front posting, but telling somebody "your network breaks
our specs" just invites the response "your specs broke my
network." I think the obvious next step in this matter is to
write down operational and implementation guidance to prevent
this sort of problem occurring. That seems to be part of v6ops'
job. (Sorry, I'm not volunteering.)

Regards
   Brian

---

From: Owen DeLong <[email-masked]>
Date: Fri, 12 Sep 2014 20:17:56 -0700

> some machines.  Is that a bug or by design of the OS vendor?  I don't

It’s definitely a bug. I’ve never seen that many, so I’d love to know the specifics where this has occurred.

> have seen Windows machines with more than 300 IPv6 addresses”

Ah, so the grotesque violations of network etiquette come from windows boxes… What a surprise. Explains why I haven’t seen them. I gave up on Windows over a decade ago.

On most well behaved OS, btw, it’s a maximum of ~7.

> http://inconcepts.biz/~jsw/ipv6_nd_problems_with_l2_mcast.pdf

Right… Again, it seems like large layer 2 zones+mld snooping+privacy addressing are the “fire triangle” here and that if you remove any leg, you eliminate the problem.

> probably not desireable for those who champion SLAAC.

No, you can also avoid it by network design by limiting the size of your link local domains. If your switch has a 1000 multicast group limit, then you shouldn’t have more than about 120 hosts on that network. If you’re running Windows, turn off privacy addressing in group policy or demand that Micr0$0ft fix their broken shit.

> He doesn't have large Layer 2 zones.

Yes he does… He stated that he had a campus wide L2 zone.

> Modifying ND is not without precedent.

Sure, and I’d welcome modifying ND for legitimate need, such as RFC6106.

In this case, ND doesn’t strike me as the place where ire is best targeted. MLD snooping could be improved to ignore solicited node addresses. This wouldn’t affect anything else and would clearly solve the problem. Micr0$0ft could fix their broken shit and not generate so many extra addresses for no reason.

Owen

---

From: Owen DeLong <[email-masked]>
Date: Fri, 12 Sep 2014 20:23:39 -0700

> 24-bits are kept the same, an attacker can correlate the activities of a

The purpose of privacy addresses is broken by so many things as to render them an absurd waste in general.

We’ve had this argument before. We’ve agreed to disagree before.

I suppose another viable solution would be to require all privacy addresses to use a common lower 24 bit string. In this way, you’d still have anonymity within the subnet which is all you really get from privacy addresses to begin with.

Owen

---

From: Sander Steffann <[email-masked]>
Date: Sat, 13 Sep 2014 13:20:33 +0200

Hi Owen,

> I suppose another viable solution would be to require all privacy addresses to use a common lower 24 bit string.

That would defeat the purpose of the solicited node multicast address. All of them would use the same one, so all hosts have to listen to the same one, and we're back to ARP-like efficiency again...

Cheers,
Sander

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Sat, 13 Sep 2014 12:11:18 +0000

I would think he meant for a single node, not across all nodes. Thus each node only has one multicast group (for privacy addresses).

- Bernie (from iPad)

> On Sep 13, 2014, at 7:21 AM, "Sander Steffann" <[email-masked]> wrote:
> 
> Hi Owen,
>

---

From: Sander Steffann <[email-masked]>
Date: Sat, 13 Sep 2014 14:14:51 +0200

Hi Bernie,

> I would think he meant for a single node, not across all nodes. Thus each node only has one multicast group (for privacy addresses).

I would hope so as well but I got the impression he didn't...

Cheers :)
Sander

---

From: Owen DeLong <[email-masked]>
Date: Sat, 13 Sep 2014 12:59:47 -0700

No... It would only have that impact on hosts that choose to use privacy addresses. I'd much rather see the impact of privacy addressing transferred to the hosts using it than to the network infrastructure. 

> On Sep 13, 2014, at 4:20, Sander Steffann <[email-masked]> wrote:
> 
> Hi Owen,
>

---

From: Owen DeLong <[email-masked]>
Date: Sat, 13 Sep 2014 13:03:36 -0700

It was in response to someone who felt that would defeat privacy addressing (which it likely would). However, having all hosts use the same last 24 for all privacy addresses would preserve privacy while transferring the negative impact from them onto the hosts in a much less impactful way than the current impacts in network infrastructure. 

I do like the idea of having a limited number of low-24 bit patterns that privacy addresses can rotate through. Seems like a potential best of both worlds. 

Owen

> On Sep 13, 2014, at 5:14, Sander Steffann <[email-masked]> wrote:
> 
> Hi Bernie,
>

---

From: Mark ZZZ Smith <[email-masked]>
Date: Sun, 14 Sep 2014 18:03:20 -0700

Hi Brian,

----- Original Message -----
> To: Mark ZZZ Smith <[email-masked]>
> Cc: Dale W. Carder <[email-masked]>; IPv6 Operations <[email-masked]>; "[email-masked]" <[email-masked]>
> Sent: Tuesday, 9 September 2014, 12:14

Sure.

The trouble with that blog post is that it doesn't provide conclusive evidence that the Router Alert option in MLD messages is the only cause of their networking problems.

The actual blog post doesn't really provide much technical evidence of anything. The evidence is does provide however is that they haven't been performing troubleshooting well, nor maintaining their network very well:

- they took a 'suck-it-and-see' approach to trying to fix the first issue, and in particular, changed multiple things at once. If the problem then disappears, which change made resolved it? Would they now perform disruptive firmware upgrades even though that change may not have been the one that fixed the problem?

- they suspect they have 'bridge loops' forming, and that is overloading the control plane during these 'bridge loops'. Yet they don't say they have spent any time confirming their speculation, or any effort in identifying the cause of these 'bridge loops'. They also speculate about what the consequences of these 'bridge loops' are (STP messages being dropped because of control plane overload). They're actively working on taking actions to remedy this, without any evidence their speculation is true. They're jumping to conclusions.

With so little apparent proper evidence collecting and proper troubleshooting and diagnosis, how can the blog author be sure IPv6 MLD with the RA option is the specific cause of their problems? Consequently, how can this group then accept that as conclusive and therefore spend time speculating on changes that could be made to MLD that may also be completely ineffective on that network - because it may not be the cause at all? There isn't even enough detail in that blog post to be able to reproduce the scenario independently - what hosts/host OSes are they using, how many are there, are they running other multicast applications etc., etc.,

I think it is possible that RA option in MLD messages might be one of the causes of or contributors to their problems. However, there are many other possible causes that don't seem to have been investigated.

One reason I'm sceptical about MLD being the cause is because versions of Windows since Vista (released January 2007) have been IPv6 enabled by default, and have been issuing solicited-node MLDv2 reports for its link-local addresses, and if global or ULA prefixes are available, has been generating privacy addresses and corresponding solicited-node MLDv2 reports for those too. So since at least 2007, many MLD messages have been sent onto many different switched networks for many years. If RA option in MLDv2 reports were such a problem to switched networks, we should have heard about it by now and have also seen other networks suffer from this problem. This problem seems to be unique to this network, and therefore I think that is a sign that something else is going on, in addition to the other signs like this 'bridge loops' and unexplained software and perhaps hardware faults.

I've proposed using these solicited-node MLD reports to further help mitigate the ND cache DoS (draft-smith-v6ops-mitigate-rtr-dos-mld-slctd-node). People were concerned that hosts weren't sending them, so I've collected some packet captures of an individual host booting on a (virtual) network with a single IPv6 router (OpenWRT 14.07 rc1) announcing a single ULA prefix. If people want to see what is typical, here are the Windows captures (Windows XP after IPv6 was enabled). Note the captures were taken at the router's interface rather than the host's, so they're showing what the router receives from the hosts or sends towards them, rather than what the hosts were sending or receiving.

http://www.users.on.net/~markachy/windows-xp-sp3-boot-rtr-cap.pcap

http://www.users.on.net/~markachy/windows-vista-boot-rtr-cap.pcap

http://www.users.on.net/~markachy/windows-7-boot-rtr-cap.pcap

http://www.users.on.net/~markachy/windows-8.1-boot-rtr-cap.pcap

Regards,
Mark.

>>>>   Thus spake Brian E Carpenter ([email-masked]) on Mon, 
> Sep 08, 
>>>  2014 at 07:50:26AM +1200:
>>  then I think this particular network was already on the verge of 
> catastrophic failure, and some of IPv6's ND differences were just the 
> trigger to push it over that threshold.
>> 
>> 
>>  In some of these cases they've taken actions to try to resolve them, 
> however the actions taken seem to be based on speculating what is happening 
> rather than finding evidence to support the hypothesised cause before taking 
> action.
>> 
>>  For example, there isn't any evidence that they've actually 
> determined that 'bridge loops' are occurring, found out what is causing 
> them, and then taking measures to prevent them or at least reduce the chances of 
> them occurring. If a 'bridge loop' can cause the control plane to 
> overload, then adding anything to this network like IPv6 is likely to only make 
> the problem worse.
>> 
>>  I'd like to see them get to the bottom of and then fix these 
> 'bridge loop' and other problems first before any value is placed on 
> addressing their criticisms of IPv6. Other people have deployed IPv6 without 
> these problems, so what is different between this network and everybody 
> else's that doesn't have these sorts of problems?
>> 
>> 
>>

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 15 Sep 2014 16:45:25 +1200

> With so little apparent proper evidence collecting and proper troubleshooting and diagnosis, how can the blog author be sure IPv6 MLD with the RA option is the specific cause of their problems?

I'm pretty sure that the Router Alert is a red herring and the real problem
was MLD snooping (i.e. a layer violation by L2 switches). Even so,
the MLD Router Alert issue seems to be a real one, if not in this scenario.

Regards
   Brian

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 15 Sep 2014 07:04:26 +0200 (CEST)

> was MLD snooping (i.e. a layer violation by L2 switches). Even so,

This is the first time I have seen anyone calling MLD/IGMP snooping a 
"layer violation by L2 switches" (I'm interpreting this as you thinking 
it's negative, do tell if this is not the case).

How else is L2 equipment going to make intelligent decisions about 
multicast forwarding within an L2?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 16 Sep 2014 08:20:54 +1200

> it's negative, do tell if this is not the case).

Yes, layer violations have this nasty habit of coming back and
biting you, and this is an example.

> multicast forwarding within an L2?

They're not. That's why DEC Gigaswitches had ARP throttling, for example,
to control broadcast storms. It's unavoidable if you build big L2
networks. I learnt not to do that.

    Brian

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 15 Sep 2014 15:08:53 -0700

> biting you, and this is an example.

Brian,

I don't think we have sufficient data on the contributions of the 
different factors that resulted in triggering the implementation 
problems in this case.

The overloading of IGMP for to both feed L3 multicast routing 
information into routers and to feed L2 multicast information into IGMP 
snooping switches (which was faithfully carries forward into MLD) 
certainly is part of the picture. That results in the routers seeing MLD 
reports for link-local groups, which the routers do not care about.

But even if we had a non-overloaded approach (with L2 multicast 
membership being propagated by IEEE *Multiple Registration Protocol* 
(*MRP*) instead of IGMP/MLD snooping), the L2 switches would still be 
overloaded with so many temporary addresses and associated 
solicited-node multicast addresses.

Of course, I've never seen a host which implements MRP, but that is 
separate from the implied architectural question about which part of the 
blame should be placed on overloading (or layer violations if you prefer 
that term) in this case.

Regards,
    Erik

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 16 Sep 2014 00:40:40 +0100

> networks. I learnt not to do that.

let's be careful what we're talking about here.  The problems of yesteryear
were related to large broadcast domains.  The problems we're discussing
here relate to modest-sized broadcast domains, but potentially with a large
number of them traversing a single switch.  These are different problem
spaces and it is not helpful to confuse the two.

This original issue brings up a number of awkward questions about
scalability, which at heart is a protocol design issue and not - as stated
by others - either a vendor implementation issue or a protocol issue.  The
IETF created a protocol dependency mechanism to assist scalability by
allowing large numbers of v6 addresses to exist on the same l2 network
(both single broadcast domain and multiple broadcast domains).

The mechanism operates along the lines of:

- ND depends on multicast for basic functionality

- ND multicast addressing uses multicast groups to assist scalability

- MLD is implemented to help l2 devices decide how and where to prune v6
multicast transmission

- this mechanism pushes state into the l2 forwarding control plane (SP =
switch processor)

- privacy addresses increase the number of v6 addresses on any single/multi
broadcast domains by an order of magnitude, give or take.

- anecdotally, it seems that the continued existence of large-ish but
segmented multiple vlan l2 networks and the advent of privacy addresses
means that switches are seeing a problematic quantity of state being pushed
to the SP on some networks.

Stepping back a little to broadcast vs multicast, the v6 protocol was
designed this way to work around the rampant broadcast storm problems of
the 1990s where large broadcast domains were all the rage.  Those who were
around at the time will remember windows 3.1 cpus pegging due to 10 megs of
broadcast traffic on the campus /16.  In this sort of situation, mld
snooping might well have worked nicely to stop problems at the network edge

The problem set has changed - modern networks rarely use large flat
networks but instead segment networks into large numbers of vlans on a
shared physical infrastructure.  This means that today, the protocol that
was designed to help at the network edge is hurting at the network core and
we are applying a solution to a problem which largely no longer exists in
its original context.

The irony is that the solution which was created to allow ipv6 to scale on
large layer 2 domains is restricting scalability on large layer 2 domains.
 We have merely swapped one scaling problem for another.

Nick

(*) there is an exception in large scale virtualised infrastructure, where
VM networking people have not learned the lessons of the past and there is
a massive push to create underlay protocols to allow enormous l2 domains
which span multiple locations.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 16 Sep 2014 12:03:23 +1200

> spaces and it is not helpful to confuse the two.

Well, that's true, and the definition of "large" has changed
as microelectronics has continued to improve. But the point is
that for any particular scenario and generation of technology,
there is scaling limit for L2. What you say below is also
very true.

    Brian

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 16 Sep 2014 07:16:41 +0200 (CEST)

> - MLD is implemented to help l2 devices decide how and where to prune v6

Well, one could argue that MLD is there to tell the NIC what multicast 
packets to actually forward into the computer, and which ones are safe to 
drop on the floor at the NIC level. So one doesn't actually have to have 
L2 switches look at MLD and use it for forwarding decisions to actually 
get some benefit out of the multicast/broadcast distinction in IPv6.

So if bandwidth isn't a problem, but CPU IRQs are (they're usually not), 
then it's still a benefit. But I generally agree with your assessment of 
the situation, I just wanted to add this one point.