Subject: [v6ops] new draft: draft-elkins-v6ops-multicast-virtual-nodes

From: [email-masked]
Date: Fri, 19 Sep 2014 04:47:01 -0700

A new draft has been posted, at http://tools.ietf.org/html/draft-elkins-v6ops-multicast-virtual-nodes. Please take a look at it and comment.

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Fri, 19 Sep 2014 19:14:17 +0200

A directed broadcast ping on IPv4 gives pretty much the same result.
Did you test the effects of that ?

Of course, private VLANs or (if we are talking VMs) or just using p2p
links with /128s would help this in the environments where the hosts
can not be trusted - and this of course is not virtual/physical
specific.

If we're talking specifically virtual environment, here's an approach
on how to use ebtables to isolate the hosts:

ebtables -P FORWARD DROP
ebtables -F FORWARD
ebtables -A FORWARD -i $uplinkPort -j ACCEPT # let the traffic flow
from uplink to any ports
ebtables -A FORWARD -o $uplinkPort -j ACCEPT # let the traffic flow
from any ports to uplink

(source: http://serverfault.com/questions/388544/is-it-possible-to-enable-port-isolation-on-linux-bridges)

So looks like the question at hand is:

"Should IPv6 nodes respond to Ping to FF0x::1?"

Which can be rephrased differently to ease the start of the discussion:

"What are the legitimate uses of a ping to ff0x::1 ?"

Right ?

--a

On 9/19/14, [email-masked] <[email-masked]> wrote:
> A new draft has been posted, at
> http://tools.ietf.org/html/draft-elkins-v6ops-multicast-virtual-nodes.
> Please take a look at it and comment.
>
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: Nalini Elkins <[email-masked]>
Date: Fri, 19 Sep 2014 12:24:31 -0700

>Did you test the effects of that ?

I had not.  But, since you mentioned it, I did it on two different Windows machines.  The one that is the server in question had the following results:

C:\Users\Administrator>ping x.x.x.255

Pinging x.x.x.255 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for x.x.x.255:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

Arp cache was not updated.   I did a packet trace in the background and indeed no ICMP replies were seen.   

I did the same ping x.x.x.255 on one of my client PCs and saw:

Pinging x.x.x.255 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

But this time the packet trace showed that actually ICMP replies were sent.

>can not be trusted - and this of course is not virtual/physical

Yes.  We wanted to bring up the topic of isolation of nodes for discussion.

>(source:http://serverfault.com/questions/388544/is-it-possible-to-enable-port-isolation-on-linux-bridges)

I think this is very good.  But, unfortunately not very well known.   Also, is this possible for all platforms or just Linux?

>Right ?

Yes.

--a

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Fri, 19 Sep 2014 22:27:14 +0200

On 9/19/14, Nalini Elkins <[email-masked]> wrote:
> I had not.  But, since you mentioned it, I did it on two different Windows
> machines.  The one that is the server in question had the following results:
>

So, besides this cosmetic difference the behavior is identical for the
case of IPv4 ping ?

> Yes.  We wanted to bring up the topic of isolation of nodes for discussion.

This is a useful topic in general - especially in light of the
MLD-related thread, but the draft seems to focus on a very corner case
- there are many much better reasons not to do large L2 networks.

> I think this is very good.  But, unfortunately not very well known.   Also,
> is this possible for all platforms or just Linux?

This was just my first google.com search result when searching for
"private vlans linux".

The first google.com search result on "private vlans windows" brings
up http://blogs.technet.com/b/scvmm/archive/2013/06/04/logical-networks-part-iv-pvlan-isolation.aspx
which seems to document similar technique.

"private vlan vmware" results in
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010691
as its first hit.

Would an IETF publication be easier to find than the above ?

>>"What are the legitimate uses of a ping to ff0x::1 ?"

Allright, so I'll mention my use of ping6 ff0x::1:

* quick check to find "a few hosts that are alive on this link"
(obviously not to be done on large segments).

* A way to trigger the remote side's ND process without having to do
one myself (I know the multicast packet to all-hosts *will* usually
get received regardless of the underlying issues with the L2.5
infrastructure (snooping, etc.)

* A way to further debug malfunctioning L2.5 infra, by comparing the
reaction to pings to ff02::1, solicited node multicast, other
multicast groups, etc.

Summary: the ff02::1 current ping behavior does help in a non-trivial
amount of cases.

The security aspects the draft mentions indeed do exist, but in a
properly configured network can be easily mitigated as I've shown
above. So I'd be against changing the functionality in the existing
stacks.

On the education side: the existing search results
https://www.google.com/?gws_rd=ssl#q=ping6+ff02::1 show
8000-something results - so indeed probably this behavior may be
highlighted more during the IPv6 education. However: every IPv6 course
I've heard says "there is no broadcast in IPv6, but there is all-hosts
multicast" which should not make the behavior the draft describes a
major surprise.

--a

---

From: "Ackermann, Michael" <[email-masked]>
Date: Fri, 19 Sep 2014 20:55:37 +0000

HI Andrew

Thanks for the great responses and related information.   

My comment on the inclusion of MLD,  is that we talked a lot about MLD during the creation of this draft and it certainly is consistent with the theme and adds to the issues.   However, we chose to leave MLD out of this draft, in an effort to keep the draft smaller and more focused.   We do mention MLD as being a related factor but state it is outside of the scope of the draft.     Given that, a question for you is if you feel this draft SHOULD include MLD?    

One other comment is that although there is education and information about these types of IPv6 subjects/issues, most large enterprises are unaware of any of this.   Our hope was to spare them from having an unpleasant initial experience with IPv6, by pointing out some issues  and pitfalls we have run into.   There is enough reluctance to pursue IPv6, in the corporate and large enterprise environments today and we want to eliminate as many negatives from that decision process as possible.  

Hope that makes sense. 

Thanks again

Mike

-----Original Message-----
From: v6ops [mailto:[email-masked]] On Behalf Of Andrew ?? Yourtchenko
Sent: Friday, September 19, 2014 4:27 PM
To: Nalini Elkins
Cc: [email-masked]; [email-masked]
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-multicast-virtual-nodes

On 9/19/14, Nalini Elkins <[email-masked]> wrote:
> I had not.  But, since you mentioned it, I did it on two different 
> Windows machines.  The one that is the server in question had the following results:
>

So, besides this cosmetic difference the behavior is identical for the case of IPv4 ping ?

> Yes.  We wanted to bring up the topic of isolation of nodes for discussion.

This is a useful topic in general - especially in light of the MLD-related thread, but the draft seems to focus on a very corner case
- there are many much better reasons not to do large L2 networks.

>>from uplink to any ports ebtables -A FORWARD -o $uplinkPort -j ACCEPT 
>># let the traffic flow from any ports to uplink
>

This was just my first google.com search result when searching for "private vlans linux".

The first google.com search result on "private vlans windows" brings up http://blogs.technet.com/b/scvmm/archive/2013/06/04/logical-networks-part-iv-pvlan-isolation.aspx
which seems to document similar technique.

* A way to trigger the remote side's ND process without having to do one myself (I know the multicast packet to all-hosts *will* usually get received regardless of the underlying issues with the L2.5 infrastructure (snooping, etc.)

* A way to further debug malfunctioning L2.5 infra, by comparing the reaction to pings to ff02::1, solicited node multicast, other multicast groups, etc.

Summary: the ff02::1 current ping behavior does help in a non-trivial amount of cases.

The security aspects the draft mentions indeed do exist, but in a properly configured network can be easily mitigated as I've shown above. So I'd be against changing the functionality in the existing stacks.

On the education side: the existing search results
https://www.google.com/?gws_rd=ssl#q=ping6+ff02::1 show 8000-something results - so indeed probably this behavior may be highlighted more during the IPv6 education. However: every IPv6 course I've heard says "there is no broadcast in IPv6, but there is all-hosts multicast" which should not make the behavior the draft describes a major surprise.

--a

The information contained in this communication is highly confidential and is intended solely for the use of the individual(s) to whom this communication is directed. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information is prohibited. Please notify the sender, by electronic mail or telephone, of any unintended receipt and delete the original message without making any copies.
 
 Blue Cross Blue Shield of Michigan and Blue Care Network of Michigan are nonprofit corporations and independent licensees of the Blue Cross and Blue Shield Association.

---

From: Nalini Elkins <[email-masked]>
Date: Fri, 19 Sep 2014 15:01:58 -0700

>So, besides this cosmetic difference the behavior is identical for the

Andrew, I think you may have misunderstand me.  The behavior is very definitely not
identical in IPv4.   In IPv4, many people block or disable directed broadcast PING.  Such
PINGs can be used for amplification in Smurf attacks.

http://www.techrepublic.com/article/understanding-a-smurf-attack-is-the-first-step-toward-thwarting-one/

Ping to FF02::1 is definitely amplification when 10 echo requests can create 2,000+ 
echo replies.   When you do a Ping on Linux, it is continuous until you stop it.
So, you may very easily do amplification without even meaning to.

>- there are many much better reasons not to do large L2 networks.

We wanted to pick one specific instance.  Maybe we went TOO small!

>Would an IETF publication be easier to find than the above ?

Maybe not but the hosting company who provided us the service where we did this real life test does not seem to have read any of the documents you reference.

What to do is a good question.  IPv6 implementation at end user sites is new to many & mistakes are being made.  I do not know what is the answer.

>above. So I'd be against changing the functionality in the existing

I don't know, Andrew.  I worry about the broadcast nature of the ping.
I feel like it is trouble waiting to happen.

--a

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Sat, 20 Sep 2014 00:24:32 +0200

Hi Michael,

On 9/19/14, Ackermann, Michael <[email-masked]> wrote:
> My comment on the inclusion of MLD,  is that we talked a lot about MLD
> during the creation of this draft and it certainly is consistent with the
> theme and adds to the issues.   However, we chose to leave MLD out of this
> draft, in an effort to keep the draft smaller and more focused.   We do
> mention MLD as being a related factor but state it is outside of the scope
> of the draft.     Given that, a question for you is if you feel this draft
> SHOULD include MLD?

No, my point was about the overall approach of the draft - there
seemed to be two aims:

1) the "TBD" placed into the section aiming at general design of subnets,

2) a question "should the hosts respond to ff0x::1?" following after
the description of a link-local equivalent of a smurf attack.

If the draft had just the (2) above (even limit it to just
observations, maybe? Could someone else from the community chime in
who thinks it is a problem?), and also discussed whether one can use
larger scopes (what happens if ff05::1?), and the *operational*
measures that can be taken today to prevent that threat, this would
make it self-contained. As of now, it reads a bit unclear, and (1) is
a rather broad topic to cover adequately while keeping it a short
read, if you wanted to do so.

>
> One other comment is that although there is education and information about
> these types of IPv6 subjects/issues, most large enterprises are unaware of
> any of this.   Our hope was to spare them from having an unpleasant initial
> experience with IPv6, by pointing out some issues  and pitfalls we have run
> into.   There is enough reluctance to pursue IPv6, in the corporate and
> large enterprise environments today and we want to eliminate as many
> negatives from that decision process as possible.
>
> Hope that makes sense.

My general feeling: if a network is designed/operated in such a way
that ping6 ff02::1 is an unpleasant experience, there should be a big
warning sign "this is the least of your worries, you did not do your
homework, so buckle up for the real fun to come".

At the same time,  if you indeed saw someone operationally messing up
the network due to this issue - a short focused document discussing
the observations and mitigations would be useful to point those people
to.

--a

> This is a useful topic in general - especially in light of the MLD-related
> thread, but the draft seems to focus on a very corner case
> The first google.com search result on "private vlans windows" brings up
> http://blogs.technet.com/b/scvmm/archive/2013/06/04/logical-networks-part-iv-pvlan-isolation.aspx
> * A way to trigger the remote side's ND process without having to do one
> myself (I know the multicast packet to all-hosts *will* usually get received
> regardless of the underlying issues with the L2.5 infrastructure (snooping,
> etc.)
>
> reaction to pings to ff02::1, solicited node multicast, other multicast
> groups, etc.
>
> The security aspects the draft mentions indeed do exist, but in a properly
> configured network can be easily mitigated as I've shown above. So I'd be
> against changing the functionality in the existing stacks.
>
> https://www.google.com/?gws_rd=ssl#q=ping6+ff02::1 show 8000-something
> results - so indeed probably this behavior may be highlighted more during
> the IPv6 education. However: every IPv6 course I've heard says "there is no
> broadcast in IPv6, but there is all-hosts multicast" which should not make
> the behavior the draft describes a major surprise.
>
> --a
>
>>
>> --a
>>
>>
> The information contained in this communication is highly confidential and
> is intended solely for the use of the individual(s) to whom this
> communication is directed. If you are not the intended recipient, you are
> hereby notified that any viewing, copying, disclosure or distribution of
> this information is prohibited. Please notify the sender, by electronic mail
> or telephone, of any unintended receipt and delete the original message
> without making any copies.
>
>  Blue Cross Blue Shield of Michigan and Blue Care Network of Michigan are
> nonprofit corporations and independent licensees of the Blue Cross and Blue
> Shield Association.
>

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 19 Sep 2014 23:54:55 +0100

> So, you may very easily do amplification without even meaning to.

directed broadcast ping on ipv4 is harmful because it uses global
addresses, and the directed broadcast ping can be initiated remotely and
can be used to target third party addresses.  ff02::1 is link-local only,
which means that directed broadcast ping can only be initiated locally.  If
the operator follows BCP38, the replies will stay local so the scope for
damage is minimal.  The two aren't comparable.

Nick

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Sat, 20 Sep 2014 00:56:38 +0200

> identical in IPv4.   In IPv4, many people block or disable directed

On the same subnet ? You wrote that the replies were indeed amplified
and sent, or did I misunderstand ?

> So, you may very easily do amplification without even meaning to.

This is all link-local. The reason Smurf is dangerous is that one can
send the request from across the internet. Here it's on the same
segment - so it's a customer-provider relationship, and there are ton
of the existing operational mechanisms that can be used to take care
of this if this is a concern - documenting them might be useful.
Saying that the hosts should not reply altogether - not; for the
reasons outlined in the previous mail.

(Of course this is just my experience, would be interesting to hear
others' opinions).

> We wanted to pick one specific instance.  Maybe we went TOO small!

No, the more focused the doc is, the better.

>>up
> Maybe not but the hosting company who provided us the service where we did
> this real life test does not seem to have read any of the documents you
> reference.

If you were not spoofing the source, then the only DoS target was
yourself, and the inter-customer resource isolation should take care
of the rest. This is relates to "Oops I don't know what I'm doing"
part.

Now, if you start to spoof your source and send the traffic at high
rates, we're into the ToS violation territory, and the hosting company
probably assumes a certain degree of good citizenship from its clients
towards each other.

If you were to try this kind of link-local smurf, why not
try stealing other VM's IPv4 addresses and running MITM on them ? Or
try spoofing the default router ? Or try sweep-scanning their entire
/64 from Internet at high rate ? Or get into legacy and try
ARP-flooding with broadcast ARP requests/replies at line rate ?

There usually is a very simple mitigation to all of the above: first
warn the hosting customer that is being naughty, then terminate the
contract. I know for sure it is being practiced.

--a

>
> What to do is a good question.  IPv6 implementation at end user sites is new
> to many & mistakes are being made.  I do not know what is the answer.
>
>>

---

From: Nick Hilliard <[email-masked]>
Date: Sat, 20 Sep 2014 00:34:25 +0100

On 19/09/2014 23:24, Andrew 👽  Yourtchenko wrote:
> who thinks it is a problem?),and also discussed whether one can use
> make it self-contained. 

the draft describes that when pinging the mcast all-hosts address, a bunch
of replies are received from third party hosts which populated the nd cache
on the author's machine with a bunch of entries.  The authors have not
provided enough detail in the email to show the cause of the amplification;
 anecdotally on my home network, pinging ff02::1%en0 causes some ND packets
and gets a single echo reply per node, which is nothing like what the
authors of this draft are seeing.

It would be more useful at this stage for the authors to upload a pcap file
for analysis rather than assuming there is a protocol level problem.  It
seems unusual that 10 icmp requests would result in 2840 replies but
without details, it's not possible to tell what's going on and on that
basis, an ID seems premature.

As a side note, the ND cache entries look like they're derived from Xen MAC
adddresses (00:16:3E:*).  If the virtual hosting provider in question has
made ipv6 end-user access available on a shared broadcast domain on Xen,
then large ND caches are probably the least of their worries: neither Xen
nor openvswitch support RA guard at the time of writing, which means that
if the authors were to inject fake RAs, they could trivially take over the
entire ipv6 network.

Nick

---

From: Nalini Elkins <[email-masked]>
Date: Fri, 19 Sep 2014 16:37:07 -0700

>damage is minimal.  The two aren't comparable.

I think the similarity has to do with the potential for amplification.

The other point we were trying to make is that "replies staying local" is not really harmless.

In IPv6 subnets, there may be many other nodes on link with you.   We were using the
Ping to FF02::1 as a graphic example of how it is possible to impact both yourself and
other nodes without even meaning to.

Nick

---

From: Nalini Elkins <[email-masked]>
Date: Fri, 19 Sep 2014 16:49:23 -0700

>and sent, or did I misunderstand ?

The replies on the same subnet appear to be blocked in one case and not blocked in another case.  
So, one cannot make a blanket statement (at least based on two data points!).

>(Of course this is just my experience, would be interesting to hear

As I said in another response, in IPv6 subnets, there may be many other nodes
on link with you.   We were using the
 Ping to FF02::1 as a graphic example of how
it is possible to impact both yourself and other nodes without even meaning to. 

We discovered this by accident.   We were doing testing for another
purpose and were quite surprised to see that both servers were seeing all the
multicast packets and even more surprised when we did the PING to FF02::1.

>of the rest. This is relates to "Oops I don't know what I'm doing"

Actually, not.  In this case, there was no inter-customer resource
isolation.  So, we were impacting everyone.   As I say, we discovered
this by accident.

I am thinking that the bigger question that we should address is
node isolation.  I think that people implementing IPv6 have got the
/64 for subnet in their heads pretty well.  But, not the other part
about how multicast - and other packets impact nodes on link with you.
And that they should do node isolation.

What do you think?

>contract. I know for sure it is being practiced.

We discovered this by accident & were using the example of Ping FF02::1 to
illustrate what can happen. 

--a

---

From: Nalini Elkins <[email-masked]>
Date: Fri, 19 Sep 2014 16:52:18 -0700

Nick,

Try doing :

PING FF02::1 -n 10

How many devices have you on link with you?

Am happy to share the PCAP with you privately.   Pls let me know.
 
Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

----- Original Message -----
From: Nick Hilliard <[email-masked]>
To: Andrew 👽 Yourtchenko <[email-masked]>
Cc: "[email-masked]" <[email-masked]>; "[email-masked]" <[email-masked]>
Sent: Friday, September 19, 2014 4:34 PM
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-multicast-virtual-nodes

Nick

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Sat, 20 Sep 2014 02:36:24 +0200

On 9/20/14, Nalini Elkins <[email-masked]> wrote:
>>> Windows

With the test I did within my home network, I saw amplified replies on
both, so one can not make a blanket statement indeed and it depends on
the OS and the setup.

>> Ping to FF02::1 is definitely amplification when 10 echo requests can
>> create
>> 2,000+
>> echo replies.   When you do a Ping on Linux, it is continuous until you
>> stop
>> it.

So you ping ff02::1 on a network with X hosts, where X is large. So
each of these X hosts has to process 1 packet and generate 1 packet,
you have to process X responses, you populate ND cache on your hosts
with X entries, each of the X hosts populates their ND cache with your
address.

Your host is quite obviously impacted. But the others process a 1/X
share of the load compared to you, an equivalent to a regular 1 host
exchange. So I don't see how they are impacted.

> multicast packets and even more surprised when we did the PING to FF02::1.

Nick has a good point this whole situation would benefit of some more
debugging.

> isolation.  So, we were impacting everyone.   As I say, we discovered

Again, it's obvious to see how you're impacting yourself but it's
unclear to me how were you impacting the others.

I did a quick test in my lab, and the only node that significantly
grew the ND cache in my test was the one sending the ping. The rest
just added only the sender's address. Sure, the sender also triggered
NUD on the addresses, but again it is only the sender that was
impacted - the rest had just 1 packet to handle, which is not much.

Please help me understand where is the problem ?

> And that they should do node isolation.

Give each vhost a routed /64 and use link-local next-hops - this will
give the same "broadcast profile" as for IPv4, and reduce the hoster's
resources needed to manage the allocated addresses, and still allows
to sell 65536 vhosts, assuming the said hoster has only /48.

Then on the shared segment that carries the routed traffic,
intra-subnet apply the same security policy to the ff02::1 IPv6
traffic as you do to IPv4 x.x.x.255 originated within and destined for
that subnet - private VLANs, p2p links, or nothing.

--a

---

From: Nalini Elkins <[email-masked]>
Date: Fri, 19 Sep 2014 20:54:26 -0700

>exchange. So I don't see how they are impacted.

On Windows, I did a :  ping ff02::1 -n 10

where the -n is the count.

So for each 1 ping request I send out, each of my neighbors sends me 10 responses.
So, I have made them do 10 times as much work as me.  Also, since in this case, I
had about 200 neighbors, the link had a bunch of traffic on it (the ping requests 
and replies from everyone) (as well as the neighbor discovery packets) which were 
in the way of anyone wanting to actual productive work.

>Nick has a good point this whole situation would benefit of some more

Such as what?  BTW, I have sent Nick the trace & can send to you as well.

>Please help me understand where is the problem ?

See above.  Also, I believe it is not a question of the neighbor cache.
We just showed that to prove that other nodes actually were responding.

Also, our topic actually was a bit more broad.  We were not talking only
of the PING to FF02::1, but of multicast.   Multicast is seen by all the 
neighbors and can be quite a bit of the link activity.   Again, getting in
the way of productive traffic.

We may be spending a bit too much time focused solely on the Ping.
I stand by my statement that the Ping can be an amplification.   
However, we used the example of the Ping to show that if you have a 
number of neighbors on link, as shown by the Ping, then they will see
all the multicast packets generated as well.

>that subnet - private VLANs, p2p links, or nothing.

I think this focuses too narrowly on the Ping.    I need to think some
more on how to talk about isolation.

--a

---

From: Andrew 👽 Yourtchenko <[email-masked]>
Date: Sat, 20 Sep 2014 09:30:10 +0200

On 9/20/14, Nalini Elkins <[email-masked]> wrote:
>> it is possible to impact both yourself and other nodes without even
>> meaning
>> to.
>

No, the "-n 10" means you send 10 requests, each of them triggers a
response from each of the neighbors, at least according to Microsoft
documentation at
http://technet.microsoft.com/en-us/library/cc737478%28v=ws.10%29.aspx

> So, I have made them do 10 times as much work as me.

No, you did not. You made them do the same amount of work as you did
sending the packets - you sent 10 requests, they sent 10 replies.

> Also, since in this
> case, I

ICMP echo requests in the syntax you described are sent ~1 per second,
so the replies is about 200 per second. It's a very small amount of
traffic.

> Such as what?  BTW, I have sent Nick the trace & can send to you as well.

Sure, feel free to send it.

> See above.
> Also, I believe it is not a question of the neighbor cache.

The address is named all-hosts multicast, so it is logical for the
hosts to see it and react.

> more on how to talk about isolation.

Subnets is the industry-standard way of isolating the broadcast domains.

--a

---

From: Nick Hilliard <[email-masked]>
Date: Sat, 20 Sep 2014 10:16:11 +0100

> So for each 1 ping request I send out, each of my neighbors sends me 10

The "-n" parameter specifies how many icmp echo requests are sent out, not 
how many replies are solicited.

> Such as what?  BTW, I have sent Nick the trace & can send to you as well.

The pcap file shows 10 icmp echo requests and 2840 echo replies, and that 
the echo replies used 284 unique link-local ipv6 addresses associated with 
284 unique MAC addresses.  This looks both correct and consistent with the 
figures in the draft.  The lesson is that if you have a lot of ipv6 hosts 
on the same broadcast domain, you will get lots of echo replies from a ping 
to ff02::1.

Nalini, if you feel that there is an issue here, you should talk to your 
hosting provider and suggest that they don't put as many ipv6 hosts on the 
same broadcast domain.  This isn't a protocol level problem: it's expected 
and correct behaviour.

Nick

---

From: Nalini Elkins <[email-masked]>
Date: Sat, 20 Sep 2014 07:15:48 -0700

>on the same broadcast domain, you will get lots of echo replies from a ping 

This is the point we were trying to make.  But, we were using Ping to illustrate
that much layer 2 communication will occur - such as multicast.

Also, I was wrong in my explanation of multicast ping amplification.   How it
works is this:

Situation:  node A is on a subnet with 25 other nodes (B-Z)

1.  Node A sends 10 ICMP ping requests to FF02::1
2.  Nodes B through Z send Node A back ICMP 10 ping replies each

Impact: with very little work by Node A, he has made B - Z do work &
created network congestion with 250+ packets.

>same broadcast domain.  This isn't a protocol level problem: it's expected 

I think people will be quite surprised at this behavior.   We think that the
issues of how many hosts are on a broadcast domain and node isolation
should be addressed.   We will add this to our draft.

Because It is not just ICMP but all link local and multicast communication.  I tell 
you, I do not think that people are not considering this in their network design.  
We are using the hosting company as an example of what really is happening in 
the wild.

Nick

---

From: Nalini Elkins <[email-masked]>
Date: Sat, 20 Sep 2014 07:22:38 -0700

>sending the packets - you sent 10 requests, they sent 10 replies.

You are correct. I was wrong in my explanation of multicast ping amplification. 
How I believe it works is this:

>so the replies is about 200 per second. It's a very small amount of

In this case.  The surprise (at least to us) was that it happens at all.  Also,
it highlighted the point of node isolation - that one node can impact others
and the network with both link-local and multicast traffic.

BTW, this is also a security issue.

>Sure, feel free to send it.

Doing so privately.

>Subnets is the industry-standard way of isolating the broadcast domains.

Then, IMHO, there needs to be a "Best Practices" document for how to create
subnets.  I am not of the opinion that creating a subnet that spans multiple
virtual machines, which could really expect to have separation from each
other, is a great practice.

I believe that in the wild, /64 appears to be the standard for a subnet no matter
what the circumstances are.

--a

---

From: Nick Hilliard <[email-masked]>
Date: Sat, 20 Sep 2014 18:23:48 +0100

> created network congestion with 250+ packets.

If the hosting provider has any sense, this will count towards node A's 
bandwidth allocation and if they persist in doing this, they will end up 
with a large bill as well as having trashed their connectivity.

The lesson is that if you shoot yourself in the foot, it will hurt.  If 
your network design is such that shooting yourself in the foot causes other 
hosts to be shot in the foot, then your network design may have a problem.

If you are a commercial hosting provider who provides third party shared 
tenancy networks which use this design, then you should expect problems of 
this sort.

Most hosting providers know these things and there is no need for the IETF 
to remind them.

Nick

---

From: Nalini Elkins <[email-masked]>
Date: Sat, 20 Sep 2014 16:25:04 -0700

>Most hosting providers know these things and there is no need for the IETF 

Really?

So then, are you saying that the hosting company example we gave was for a 
particularly ignorant or incompetent firm?   That most hosting providers of IPv6
services and most end user enterprises (because our draft is for them as well)
are quite well aware of the potential drawbacks and implications of IPv6 link-local 
and multicast traffic in the design of their subnets or broadcast domains.
And, furthermore take such traffic and / or the need for isolation of nodes for
security or regulatory purposes into account when doing their network design?

And, basically, the issues that we are trying to bring up are quite well known 
to everyone.   You could be right (but I am betting against it!).    

Maybe my co-author, who works for a large end-user enterprise, and whose 
organization is trying to implement IPv6, can chime in and let us know how 
well known these issues are in his own organization and those he speaks to
regularly.

Maybe other enterprise customers on the list can speak as well.  Would love
to hear from them.  Having said that, probably if they are on this email list,
they are probably orders of magnitude more sophisticated than most.

I can also take an informal poll.   I can call 2 -3 other hosting companies 
to see what their IPv6 addressing structure is & let you all know.   I can also
call 5 - 10 large enterprise companies in the US (Fortune 500 & large federal 
agencies & the like) to tell you if they take such things into consideration.   

A few years ago, I compiled IPv6 addressing plans (subnet structure,
naming conventions, etc) from 6 - 7 universities (commercial entities will
generally not speak publicly about how they do things).   I can tell you there
were no such considerations in their plans.  I can go back and ask them
if they will allow me to distribute their plans publicly.

All this will take me a few days, so maybe we can pick up the discussion
on Wed / Thurs of next week.

I believe that guidance on network design is needed for end-user sites -
including hosting companies.   That was the reason for our draft.  If I am 
wrong and no such guidance is needed, then I will be quiet and withdraw
the draft.

Nalini

---

From: Bill Cerveny <[email-masked]>
Date: Sun, 21 Sep 2014 10:29:39 -0400

I personally think that the question "4.2 Should nodes respond to Ping to FF0x::1" in the recommendations section is an interesting one. Just as I would expect there is guidance (but don't know if there is) regarding responses to IPv4 broadcast ICMP echo requests, it seems it could be helpful to provide guidance to OS or OS firewall manufacturers regarding if or how nodes should respond to ICMPv6 echo requests via multicast addresses (if this guidance doesn't already exist).

Bill Cerveny

---

From: "Ackermann, Michael" <[email-masked]>
Date: Sun, 21 Sep 2014 14:59:47 +0000

In response to how much relative knowledge there is on IPV6 and the understanding that issues such as this may exist……..
In most corporate and enterprise worlds that I deal with, which is primarily Financial, insurance and Health Care,   the answer is nearly zero.   I find this unfortunate and frustrating,  but true.   I and a few others are trying to change this and evangelize IPv6, but we are pushing ropes up hills in these environments.    Most large enterprises want to avoid IPv6 as long as possible and look for excuses to do so.
I fully recognize that this type of information contained in this draft,  may be considered elementary,  trivial,  or even unnecessary in IETF circles, but I can guarantee you that the preponderance of network technicians in  the corporate world do not even know these protocols exist, much less understand the potential damage they can inflict.
My suspicion regarding “Hosting Companies”,  is that their IPv6 knowledge levels lie somewhere above the dearth of corporate America, but well below the knowledge level of those involved in IETF.

Therefore, I believe that information and advice, such as contained in this draft, will be very helpful to those not proficient in IPv6 and hence positive to the cause of promoting IPv6 deployment.

From: v6ops [mailto:[email-masked]] On Behalf Of Nalini Elkins
Sent: Saturday, September 20, 2014 7:25 PM
To: Nick Hilliard; Andrew 👽 Yourtchenko
Cc: [email-masked]; [email-masked]
Subject: Re: [v6ops] new draft: draft-elkins-v6ops-multicast-virtual-nodes

Really?

Nalini

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 22 Sep 2014 10:09:00 +0200 (CEST)

> particularly ignorant or incompetent firm?   That most hosting providers of IPv6

No, I think the general thinking is that they're of average ignorance and 
competency, which doesn't say much.

The general advice both for IPv4 and IPv6 is not to put a huge amount of 
nodes in the same L2 domain. This has been true for 30 years, that's why 
IP replaced the L2 bridged protocols or lots-of-years-ago.

If you're putting 284 nodes in the same broadcast domain, you're asking 
for trouble. My advice is to separate customers into separate L2 domains, 
or make sure your L2 infrastructure has some L3 functionality, for 
instance private-vlan or other filtering.

So while your draft describes a problem that might be there, asking for 
nodes to not respond to link-local FF0x::1 ICMP ECHO REQUEST isn't the 
correct solution.

Btw, how does the data center you've done testing on implement BCP38 so 
that the customers can't originate packets for addresses they do not 
"own"?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Nalini Elkins <[email-masked]>
Date: Mon, 22 Sep 2014 10:55:50 -0700

>instance private-vlan or other filtering.

It is an interesting question of whether 284 is too many or not.  I
think I prefer to start using a term "administrative domain" (or some such)
where those are the nodes that are under a single control for business
or security reasons.

So, "administrative domain" (AD) and "broadcast domain" (BD) should be the same.
For example, if I have three virtual servers (or nodes), then that is my 
administrative and broadcast domain.

If I have a 1,000 nodes under my control, then that is how many that
should be in my AD and BD.   IPv6 allows for large subnets so why not?

But, if I have only 1 node, then having more than one node in my BD
is too much.

>nodes to not respond to link-local FF0x::1 ICMP ECHO REQUEST isn't the 

Possibly not.  We wanted to raise the issue.  BTW, Pings, as you know,
are the precursor to much malicious activity.  It is something that
someone with ill intent is liable to do readily.  And, it provides
a huge amount of data on other nodes in a broadcast domain.

>that the customers can't originate packets for addresses they do not "own"?

No idea.  I can try spoofing packets for other prefixes and for my servers.

Interestingly, enough, from BCP38 is:

"The one MAC address that all devices share in common in normal operation is the media broadcast, or FF:FF:FF:FF:FF:FF.  In this case, a device will take the packet and send an interrupt for processing. Thus, a flood of these broadcast frames will consume all available resources on an end-system [9]. It is perhaps prudent that system administrators should consider ensuring that their border routers do not allow directed broadcast packets to be forwarded through their routers as a default."

On Linux, I believe when you do a PING, it is continuous until intentionally stopped.  So, doing a Ping to FF02::1 from a
Linux machine will create havoc without having ill intent.  That is, accidentally.

---

From: [email-masked]
Date: Mon, 22 Sep 2014 21:21:51 +0200 (CEST)

> should be in my AD and BD.   IPv6 allows for large subnets so why not?

If you are putting 1000 nodes in the same broadcast domain, you are
deliberately ignoring painful lessons which have been learned by many
network engineers.

> Linux machine will create havoc without having ill intent.  That is, accidentally.

On all the Linux boxes I just tried this on, I get:

sthaug@server% ping6 ff02::1
connect: Invalid argument

Steinar Haug, AS 2116

---

From: Bill Cerveny <[email-masked]>
Date: Mon, 22 Sep 2014 15:43:07 -0400

> connect: Invalid argument

Try specifying the interface, such as: ping6 -I eth0 ff02::1

Bill Cerveny

---

From: [email-masked]
Date: Mon, 22 Sep 2014 21:56:16 +0200 (CEST)

> Try specifying the interface, such as: ping6 -I eth0 ff02::1

Or in my case, using the -I option to specify source address:

 -I interface address
        Set  source address to specified interface address. Argument may be
        numeric IP address or name of device. When pinging IPv6  link-local
        address this option is required.

Since an extra option is required, the risk of somebody doing this by
accident is significantly reduced.

Steinar Haug, AS 2116

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 23 Sep 2014 08:38:14 +0200 (CEST)

> should be in my AD and BD.  IPv6 allows for large subnets so why not?

Because it's a bad idea. IPv4 allows for subnetting at /22 for your 1000 
nodes, that doesn't mean it's a good idea.

> are the precursor to much malicious activity.  It is something that

No, I don't know that. ICMP PINGs are an essential diagnostic tool.

> a huge amount of data on other nodes in a broadcast domain.

I don't believe in security by obscurity. I believe in real security.

> "The one MAC address that all devices share in common in normal 
> operation is the media broadcast, or FF:FF:FF:FF:FF:FF.  In this case, a 
> device will take the packet and send an interrupt for processing. Thus, 
> a flood of these broadcast frames will consume all available resources 
> on an end-system [9]. It is perhaps prudent that system administrators 
> should consider ensuring that their border routers do not allow directed 
> broadcast packets to be forwarded through their routers as a default."

... and this is exactly what happened in 1997 when smurf attacks were 
stopped by this exact feature. "no ip directed-broadcast" on Cisco 
routers. You can still ping the broadcast address from the router itself, 
it just won't allow remote pings to work as it won't forward packets to 
the network broadcast IP address. Exactly the same way it works in IPv6 
because Link local addresses aren't reachable from another interface and 
aren't forwarded between interfaces.

> On Linux, I believe when you do a PING, it is continuous until 
> intentionally stopped.  So, doing a Ping to FF02::1 from a Linux machine 
> will create havoc without having ill intent.  That is, accidentally.

Absolutely, and it's all due to the decision to put a lot of nodes in the 
same L2 domain which has been known for 20-30 years to be bad design and 
can cause the exact problems you're describing. If you ping the network 
broadcast address on IPv4 you get the exact same thing, I don't see why 
you blame IPv6 for this?

---

From: Nalini Elkins <[email-masked]>
Date: Wed, 24 Sep 2014 14:41:59 -0700

>nodes, that doesn't mean it's a good idea.

This needs a lot more discussion.  It is a much more complex topic than
can be handled in a few sentences.  We will do another draft (will have to be 
for the Dallas IETF as time is getting short now for Hawaii) on node
isolation and subnetting.   We can discuss at much longer length then.

If you would be willing to, we would certainly appreciate some private
feedback in advance of our posting.  We can send you our draft as we
have it.

>broadcast address on IPv4 you get the exact same thing, I don't see why 

I am not blaming IPv6.  In fact, I am an IPv6 bigot and have been preaching
conversion to IPv6 for quite a few years.  (I might add, much to the dismay 
of those who have been cornered by me at various events!)

What we want to do is to point out where the pitfalls might be in the migration
to IPv6 so that people who implement will have as smooth a path as possible so 
that adoption becomes widespread.

Nalini

---

From: Randy Bush <[email-masked]>
Date: Thu, 25 Sep 2014 14:50:33 +0900

> This needs a lot more discussion.

discussion is not what is lacking.  in fact, isn't this a repeat of the
same discussion you had last time?  we have decades of too much tragic
experience of layer two not scaling well.  as mikael said, it is a bad
idea.  do not do it.  do not design for it.

randy