Subject: [v6ops] Multiple IPv6 prefixes issues-//FW: New Version Notification for draft-liu-v6ops-running-multiple-prefixes-01.txt

From: "Liubing (Leo)" <[email-masked]>
Date: Thu, 3 Jul 2014 07:48:34 +0000

Hi all,

We uploaded a new version of the draft. We once had some discussion on this topic.
This new version clearly separated the scope from MIF. And there are lots of texts revision to make it pay more attention on operational considerations and problems. 

Particularly, there is a newly identified operational issue in real product network as described in Section 3.2, the " ND table space shortage in big L2 networks", I'd like to hear comments from you on this specific issue. 
Comments on other content are of course welcomed as well.

Best regards,
Bing

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, July 03, 2014 3:31 PM
To: Boyang; Sheng Jiang; Liubing (Leo); Sheng Jiang; Boyang; Liubing (Leo)
Subject: New Version Notification for draft-liu-v6ops-running-multiple-prefixes-01.txt

A new version of I-D, draft-liu-v6ops-running-multiple-prefixes-01.txt
has been successfully submitted by Bing Liu and posted to the IETF repository.

Name:		draft-liu-v6ops-running-multiple-prefixes
Revision:	01
Title:		Running Multiple IPv6 Prefixes
Document date:	2014-07-03
Group:		Individual Submission
Pages:		11
URL:            http://www.ietf.org/internet-drafts/draft-liu-v6ops-running-multiple-prefixes-01.txt
Status:         https://datatracker.ietf.org/doc/draft-liu-v6ops-running-multiple-prefixes/
Htmlized:       http://tools.ietf.org/html/draft-liu-v6ops-running-multiple-prefixes-01
Diff:           http://www.ietf.org/rfcdiff?url2=draft-liu-v6ops-running-multiple-prefixes-01

Abstract:
   This document discusses that multiple prefixes in one network/host
   might be common in IPv6 deployment, and describes several typical
   multiple prefixes use cases. Then some operational considerations and
   current problems of running multiple prefixes are described.

Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: "Liubing (Leo)" <[email-masked]>
Date: Wed, 9 Jul 2014 08:38:13 +0000

Hi All,

Please allow me post the below mentioned "ND table space shortage in big L2 networks" issue (which was documented in draft-liu-v6ops-running-multiple-prefixes-01 as Section 3.2) in the mailing list as below. 
In summary, the issue is that in Dual-Stack L2 networks, the hosts would cost much more MAC table space in the switch than the IPv4-only hosts. Thus in big L2 networks, there probably comes the MAC table shortage problem.

This issue was found in real deployment experience. The most straightforward solution is to just increase the MAC table size. However, relevant hardware resource such as TCAM is very expensive and high power consumption that in some cost-sensitive scenarios it might not be a good solution. 
So I was thinking whether this could be considered as an operational issue, and whether there could be some operational mitigation approach.

Your comments would be appreciated very much.

"In some scenarios such as campus networks and enterprise networks,
   the "big L2 network" architecture is often used to reduce the cost
   and ease the management. In a big L2 network, a large amount of hosts
   (e.g. 10K users) are aggregated to the core at layer 2.

   The top-level core switch needs to record the MAC and IP addresses of
   all the hosts in the aggregation domain as entries in the ARP/ND
   table, so that incoming packets to the hosts could be forwarded
   directly by the line cards in a line speed. When the ARP/ND table is
   full, normally the network would not allow new hosts to access.
   Because in this situation, the switch need to instantly look up the
   destination through ARP/ND broadcast/multicast, thus the CPU needs to
   be involved in this processing. This would significantly cost the
   performance of the switch and packet loss might happen.

According to current state of the art, the maximum amount of ARP/ND
   entries in a switch is normally under 16K (the high-end ones could
   reach to 64K, measured by per line card). In IPv4, each host only
   occupies one entry, so normally the table space is enough. However,
   when the network is in an IPv6 transition, the table would be in a
   significant shortage. In implementation, one IPv6 ND entry needs 2-4
   times of an IPv4 ARP entry; and one IPv6-enabled host might configure
   2-4 IPv6 addresses or even more. For example, an IPv6-enabled Window
   7 host would have at least three IPv6 addresses (one link-local
   address, one permanent global address and one temporary global
   address) when it connects to an IPv6 network. Finally, a dual-stack
   network would cost 5-17 times table space than the IPv4 does, so that
   the table space shortage is likely to happen."

Best regards,
Bing

> From: v6ops [mailto:[email-masked]] On Behalf Of Liubing (Leo)
> Sent: Thursday, July 03, 2014 3:49 PM
> To: [email-masked]
> Subject: [v6ops] Multiple IPv6 prefixes issues-//FW: New Version
> Notification for draft-liu-v6ops-running-multiple-prefixes-01.txt
> 
> Hi all,
> 
> This new version clearly separated the scope from MIF. And there are lots of
> texts revision to make it pay more attention on operational considerations
> and problems.
> 
> Particularly, there is a newly identified operational issue in real product
> network as described in Section 3.2, the " ND table space shortage in big L2
> networks", I'd like to hear comments from you on this specific issue.
> Subject: New Version Notification for
> draft-liu-v6ops-running-multiple-prefixes-01.txt
> 
> 
> URL:
> http://www.ietf.org/internet-drafts/draft-liu-v6ops-running-multiple-prefixe
> s-01.txt
> Status:
> https://datatracker.ietf.org/doc/draft-liu-v6ops-running-multiple-prefixes/
> Htmlized:
> http://tools.ietf.org/html/draft-liu-v6ops-running-multiple-prefixes-01
> Diff:
> http://www.ietf.org/rfcdiff?url2=draft-liu-v6ops-running-multiple-prefixes-0
> 1
> 
> Abstract:
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> 
> The IETF Secretariat
> 
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops

---

From: Gert Doering <[email-masked]>
Date: Wed, 9 Jul 2014 11:46:43 +0200

Hi,

On Wed, Jul 09, 2014 at 08:38:13AM +0000, Liubing (Leo) wrote:
> In summary, the issue is that in Dual-Stack L2 networks, the hosts
> would cost much more MAC table space in the switch than the IPv4-only
> hosts. Thus in big L2 networks, there probably comes the MAC table
> shortage problem.

Uh, what?  I can see ND table overflows, but since when did hosts grew
extra MAC addresses for IPv6?

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444           USt-IdNr.: DE813185279

---

From: joel jaeggli <[email-masked]>
Date: Wed, 09 Jul 2014 03:24:27 -0700

> In summary, the issue is that in Dual-Stack L2 networks, the hosts would cost much more MAC table space in the switch than the IPv4-only hosts. Thus in big L2 networks, there probably comes the MAC…

As one of the contributors to rfc6583 I tend to be a believer that no
hosts at all are required on a subnet to result in ND table exhaustion.

With respect to the question of how much fib memory is required for l2
nexhops, that kinda falls into the design space, just as it does for
example layer 3 routes. if you can't scale it to the level necessary
then you adopt design that addresses those constraints. whether multiple
prefix are in use in one vlan or multiple vlans makes very little
differnce to a traditional l2 core, if the l2 networks are overlays then
it may make a lot of difference because there may be no core per say
that needs to carry these entries.

I have some concern  as product of direct experience about notion that
particularly large L2 networks are actually good practice, ARMD for
example kind of ran into trouble in this area.
>    reach to 64K, measured by per line card).
For the reasons above I don't think the actual number is very relevant
except in the context of a particular design problem. but you're off
about at least a factor of 4 of what a reasonably stout switch complex
looks like.
>  In IPv4, each host only

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 9 Jul 2014 12:30:25 +0200 (CEST)

On Wed, 9 Jul 2014, Liubing (Leo) wrote:

> This issue was found in real deployment experience. The most 
> straightforward solution is to just increase the MAC table size. 
> However, relevant hardware resource such as TCAM is very expensive and 
> high power consumption that in some cost-sensitive scenarios it might 
> not be a good solution. So I was thinking whether this could be 
> considered as an operational issue, and whether there could be some 
> operational mitigation approach.
>

Yeah, this was a bad idea for IPv4, and it's a bad idea for IPv6. I 
propose making smaller L2 domains by means of L3 switches closer to the 
devices, and then do routing. This solves several issues when it comes to 
scalability, while it might be seen as causing problems in other aspects.

I am a firm believer in creating a distributed L3 network and that large 
L2 domains cause problems. The issue you mentioned is just one of them...

Only way I can imagine to mitigate this operationally without doing 
anything else, is to turn off SLACC and just use single DHCPv6_IA. You 
still use 3 times more space (2 IPv6 addresses and 1 IPv4 address) for 
dual stacked host than you do for a single stacked host, but that's hard 
to avoid.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Gert Doering <[email-masked]>
Date: Wed, 9 Jul 2014 12:53:27 +0200

Hi,

On Wed, Jul 09, 2014 at 10:02:51AM +0000, Liubing (Leo) wrote:
> > From: Gert Doering [mailto:[email-masked]]
> > 
> The scenarios is regarding L3 switches, which record "IP-MAC" pairs. 

In that case, please make this clear: this is about ND/ARP cache, not 
about "MAC table space".  The latter has a very well defined meaning, and
is very much independent from IPv4/IPv6.

> So, since hosts have multiple IPv6 addresses, they'll consume multiple "IP-MAC" records in the L3 switch. And one IPv6 record would cost 2-4 times than the IPv4 record. Thus, in dual-stack scenario, the MAC table consumption is much higher than the IPv4 scenario.

This is *not* "the MAC table".

Gert Doering
        -- NetMaster

---

From: Randy Bush <[email-masked]>
Date: Wed, 09 Jul 2014 21:50:13 +0900

>  and ease the management.

operational, not vendor, experience is that "big l2 network"
architecture increases cost, management, and many kinds of
pain.

randy

---

From: "Dale W. Carder" <[email-masked]>
Date: Wed, 09 Jul 2014 09:29:20 -0500

>From our experience "big L2 network" does not reduce cost nor ease management.
We have found quite the opposite.  (We are now getting close to hitting the 
128k mac addr limit on one of our switches... <sigh>).

Any large scale design has to deal with FIB scale issues.  A recommended
approach is to "divide & conquer" by pushing the L2/L3 boundary towards the
edge.  Otherwise you must buy things with bigger FIB's, and that is not
a cost saving endeavor.

Dale

---

From: "Eric Vyncke (evyncke)" <[email-masked]>
Date: Wed, 9 Jul 2014 14:51:48 +0000

Mikael

I can only second your layer-3 architecture for the same reasons as yours:
especially with IPv6 where we can get numerous IPv6 prefixes up to one /64
per host if required.

OTOH, let's face reality and a lot of 'resiliency' or 'VM mobility' or ...
"DC solutions" requires a L2 spanning multiple kilometers. So, ending in
having the same <MAC, Ipv*> bindings in several layer-3 nodes :-(

Another corner case is the WLAN used at large large conferences where it
is easy to end up with thousands of WiFi stations on the same layer-2.
And, if those WiFi stations have privacy extension addresses (99.9% of
them), then the NDP cache can grow and grow...

-éric

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 9 Jul 2014 17:13:18 +0200 (CEST)

On Wed, 9 Jul 2014, Eric Vyncke (evyncke) wrote:

> OTOH, let's face reality and a lot of 'resiliency' or 'VM mobility' or 
> ... "DC solutions" requires a L2 spanning multiple kilometers. So, 
> ending in having the same <MAC, Ipv*> bindings in several layer-3 nodes 
> :-(

"Doctor, when I do this, it hurts." Well, tell them to stop, and find 
other solutions instead.

> them), then the NDP cache can grow and grow...

Absolutely, so make the vendors invent mechanisms that work around this, 
for instance just because you enter a certain SSID, doesn't mean you have 
to enter the same L2 domain as everybody else. I could imagine solutions 
where this was hash:ed based on MAC address or something, so that nodes 
would end up in different L2 domains. For the ND/MAC bindings, well, there 
is no way around that, you just have to make sure you have a platform that 
allows for around 4-8 bindings per device you expect to connect to your 
network.

---

From: Richard Hartmann <[email-masked]>
Date: Wed, 9 Jul 2014 17:42:55 +0200

On Wed, Jul 9, 2014 at 5:13 PM, Mikael Abrahamsson <[email-masked]> wrote:
> Absolutely, so make the vendors invent mechanisms that work around this, for
> instance just because you enter a certain SSID, doesn't mean you have to
> enter the same L2 domain as everybody else. I could imagine solutions where
> this was hash:ed based on MAC address or something, so that nodes would end
> up in different L2 domains. For the ND/MAC bindings, well, there is no way
> around that, you just have to make sure you have a platform that allows for
> around 4-8 bindings per device you expect to connect to your network.

And segment the v4 prefix you have for that automagically, how? All of
a sudden, you will also need a lot more VLANs. With some APs limited
to 16 VLANs and requirements to keep university network, eduroam, etc
pp up at the same time, this is an actual real life scenario where the
answer is not immediately obvious.

Richard

PS: Yes, this is v6ops, but Eric's argument of "large conference WiFi"
hits home a bit, for me.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 9 Jul 2014 18:06:16 +0200 (CEST)

On Wed, 9 Jul 2014, Richard Hartmann wrote:

> And segment the v4 prefix you have for that automagically, how? All of a 
> sudden, you will also need a lot more VLANs. With some APs limited to 16 
> VLANs and requirements to keep university network, eduroam, etc pp up at 
> the same time, this is an actual real life scenario where the answer is 
> not immediately obvious.

I agree that with the current platforms that are available, there is no 
obvious solution. This usually means some vendor will come up with a 
solution, and others will have to follow of lose business.

If you don't want to buy new hardware (or have vendor fix current 
scalability in software if they can) and limit the impact of this, then I 
suggest using DHCPv6_IA and disallow SLAAC, that at least cuts down on 
potential IP/MAC bindings that you have to handle to an architectural 
minimum (that I can see anyway).

Make sure to raise the issues you see with your vendor so they at least 
fix it in next generation hardware and software.

---

From: Andrew Yourtchenko <[email-masked]>
Date: Wed, 9 Jul 2014 18:41:20 +0200

On Wed, 9 Jul 2014, Mikael Abrahamsson wrote:

> "Doctor, when I do this, it hurts." Well, tell them to stop, and find other 
> solutions instead.
>
>> Another corner case is the WLAN used at large large conferences where it is 
>> easy to end up with thousands of WiFi stations on the same layer-2. And, if 
>> those WiFi stations have privacy extension addresses (99.9% of them), then 
>> the NDP cache can grow and grow...
>

Here's some data, taken from CiscoLive2014 in Milan half a year ago:

http://2014.ciscolive-ipv6.com/munin/ipv6noc/ipv6noc/net_neighbors.html
^ 
the sum of all entries in "show arp" and "show ipv6 neighbors", with the 
split by global/link-local for IPv6. We can see the total count for IPv6 
is about 3x that of IPv4.

Now, this network ran SLAAC, with about 50% clients being recent Apple 
gear, so understandably the globally routable IPv6 addresses spike up.

http://2014.ciscolive-ipv6.com/munin/ipv6noc/ipv6noc/neighbors_unique_per_vlan.html
^
This graph shows duplicates within address family per MAC address
collapsed, per VLAN.

VLAN4, as obvious, was the VLAN with pretty much all the wireless users.

I'll take the numbers below from the maximums of the graphs, so it is not 
scientific at all but gives the ballpark:

IPv4: 9.46k -> 9.35k
IPv6: (16.60k global + 7.73k link-local ) -> (6.76k global + 7.55k link-local)

So, yeah there is a ~1.7x bloat in the neighbor table due to privacy 
addresses, but that does not look too dramatic, does it ?

> enter the same L2 domain as everybody else. I could imagine solutions where

You can only do a subnet-per-area if you have non-overlapping areas 
of coverage, and are happy to have the clients restart the apps on reconnect.
Most of the time it's out of the question.

> up in different L2 domains.

Then you would need to roam the client within its VLAN across the entire 
area of coverage. So since there are N equal VLANs, do they share the same 
physical box as the default gateway ?

If they do, there's no win on NDP cache space.

I re-read the original mail starting this thread and the problem 
description there to me boils down to two issues:

1) one IP node has more IP addresses in version 6 than in version 4.
2) an IP address takes more lookup space in version 6 than in version 4.

Do I grok it right ?

> For the ND/MAC bindings, well, there is no way 

+1.

--a

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 9 Jul 2014 20:33:48 +0200 (CEST)

On Wed, 9 Jul 2014, Andrew Yourtchenko wrote:

> Then you would need to roam the client within its VLAN across the entire area 
> of coverage. So since there are N equal VLANs, do they share the same 

Yes, the only thing it would solve is to make the broadcast domains 
contain fewer devices per broadcast domain, the broadcast domains would 
still span the same APs.

> If they do, there's no win on NDP cache space.

Correct, but potentially you could have several smaller boxes, each 
serving one or more vlans, instead of a bigger box, serving all clients in 
a single vlan.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 10 Jul 2014 08:29:10 +1200

> architecture increases cost, management, and many kinds of

Yes, this is something that (to name but three that I know
of personally) Boeing, Microsoft and CERN discovered in the
late 1980s, and although the value of "big" has increased
somewhat since then, it remains true: you can't scale an L2
architecture beyond the limit set by the cost and speed of
CAM silicon.

    Brian

---

From: George Michaelson <[email-masked]>
Date: Thu, 10 Jul 2014 08:43:46 +1200

whats a rule-of-thumb size? 500? 1000? 10,000?

do we have enough thumbs?

---

From: Randy Bush <[email-masked]>
Date: Thu, 10 Jul 2014 10:38:00 +0900

> For the ND/MAC bindings, well, there is no way around that, you just
> have to make sure you have a platform that allows for around 4-8
> bindings per device you expect to connect to your network.

and having all those addresses per interface is soooo useful, one of the
wonders of ipv6

randy

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 10 Jul 2014 14:08:46 +1200

> whats a rule-of-thumb size? 500? 1000? 10,000?

I think that there were early L2 bridges with a 1024
limit. I suspect that some modern switches can survive
with 10000, but then you'll have other practical problems.

The point is that there's always a limit, and (unlike
L3 routing) you can't just fix the problem by recursing
the solution.

I have to say that Randy is correct - the multiaddressing
model for IPv6 makes the ND cache significantly larger,
but that shouldn't affect switches unless they are snooping
L3 stuff. <sarcasm>Nobody would do that, would they?</sarcasm>

     Brian

---

From: Randy Bush <[email-masked]>
Date: Thu, 10 Jul 2014 11:12:54 +0900

> the multiaddressing model for IPv6 makes the ND cache significantly
> larger

and it is such a major win.  will the police surveillance state mandate
that your govt id number be in your ip address?

randy

---

From: "Liubing (Leo)" <[email-masked]>
Date: Thu, 10 Jul 2014 10:02:07 +0000

Hi Andrew,

> So, yeah there is a ~1.7x bloat in the neighbor table due to privacy addresses,
> but that does not look too dramatic, does it ?
[Bing] Would you like to share more details of the network? E.g. it is a dual-stack or something else.
It is quite often that a host would have at least 3 IPv6 addresses: a link-local, a global and a privacy address. If ULAs are enabled, DHCPv6/SLAAC co-existing, there might be more IPv6 addresses.
So I was just guessing, maybe the 1.7x only represent that part of the nodes is enabling IPv6?
Plus, an IPv6-MAC binding would cost more cache space than an IPv4-MAC binding (2-4 times due to implementation).

> I re-read the original mail starting this thread and the problem description
> there to me boils down to two issues:
> 
[Bing] Basically yes. This is one of the problems cause by multiple IPv6 addresses in draft-liu-v6ops-running-multiple-prefixes.

B.R.
Bing

---

From: "Liubing (Leo)" <[email-masked]>
Date: Thu, 10 Jul 2014 10:08:45 +0000

Hi Eric,

> easy to end up with thousands of WiFi stations on the same layer-2.
> And, if those WiFi stations have privacy extension addresses (99.9% of them),
> then the NDP cache can grow and grow...

[Bing] We also had experience of the problem in a conference, but I think maybe it would not need to be a "large large" conference to encounter the problem.
Say, the administrators use a medium-end switch which could handle traffic of hundreds of WiFi stations but doesn't have enough ND/ARP cache space to support the full table of each host.

B.R.
Bing

> >> However, relevant hardware resource such as TCAM is very expensive
> >> and high power consumption that in some cost-sensitive scenarios it
> >> might not be a good solution. So I was thinking whether this could be
> >devices, and then do routing. This solves several issues when it comes
> >to scalability, while it might be seen as causing problems in other aspects.
> >
> >dual stacked host than you do for a single stacked host, but that's
> >hard to avoid.
> >
> >--

---

From: Andrew Yourtchenko <[email-masked]>
Date: Thu, 10 Jul 2014 12:54:20 +0200

Hi Leo,

> [Bing] Would you like to share more details of the network? E.g. it is a dual-stack or something else.

It's dualstack, the vast majority of the clients are on a single wireless 
segment (the total number of wired devices is ~300-400, so I treat them as 
noise.

The wireless clients are addressed with SLAAC, and are outside of 
my control (conference attendees).

The segment has no officially designated servers.

The configuration from the first hop router is below:

interface Vlan4
  description !!! WIRELESS CLIENTS !!! CiscoLive2014 !!!
  .. IPv4 configuration omitted ..
  ipv6 address FE80::1 link-local
  ipv6 address 2001:4D38:A:400::1/64
  ipv6 nd reachable-time 1800000
  ipv6 nd prefix default 86400 9000 off-link no-autoconfig
  ipv6 nd prefix 2001:4D38:A:400::/64 604800 86400 off-link
  ipv6 nd prefix 2001:4D38:A:6800::/64 604800 0 off-link
  ipv6 nd router-preference High
  ipv6 nd ra lifetime 9000
  no ipv6 redirects
  no ipv6 unreachables
  ipv6 verify unicast source reachable-via rx

ipv6 route 2001:4D38:A:400::/64 vlan4

Note a few bits that are specific for the conference setup, and some are 
specific to the topology:

1) maximally high lifetimes (RAs are throttled on their way to the 
clients down to 1 RA in 10 minutes).

2) clearing on-link bit: results in clients never trying to resolve the 
other clients' addresses => less memory usage on the portable devices, 
smaller ND caches there.

3) default no-autoconfig: to force the necessity to add the prefixes 
manually, to force the operator to always have the "ipv6 nd prefix ..." 
with the correct config.

4) two prefixes, one with zero preferred lifetime: there was a second VLAN 
for disaster recovery purposes, where the clients would have ended up if 
there was a catastrophic failure no the main pair of the 
wireless controllers. That VLAN had a mirrored configuration. 
Experimentally, I've found such a configuration to cause a rapid 
deprecation of the "wrong" prefix on a iOS and OSX,
resulting in minimal disruption to them in case of switchover.

5) interface route for the prefix, to send the traffic to it (configuring 
the prefix "off-link" had removed the directly connected route).

> It is quite often that a host would have at least 3 IPv6 addresses: a 
>link-local, a global and a privacy address. If ULAs are enabled, 
>DHCPv6/SLAAC co-existing, there might be more IPv6 addresses.

Right. I was well within the constraints for the first hop, so opted to 
not have DHCPv6. If I wanted a tighter control, I'd have turned off SLAAC 
and used stateful DHCPv6. Maybe I'll experiment with it this year.

> So I was just guessing, maybe the 1.7x only represent that part of the nodes is enabling IPv6?

The %% of the nodes running IPv6 was about 80-85% 
(http://2014.ciscolive-ipv6.com/munin/ipv6noc/ipv6noc/neighbors_dualstack_percentage.html 
- the dips are the the night times, a lot of internal apps were IPv4 so 
the neighbor entries went away)

I calculated this %% by taking the "show arp" and "show ipv6 neigh" 
outputs and correlating the hosts by mac address.

As I said Apple devices (they are the most aggressive today wrt the 
privacy addresses) are ~50% of the network, so maybe in another setup the 
figures would be different.

Do you have some numbers you could share ?

> [Bing] Basically yes. This is one of the problems cause by multiple 
> IPv6 addresses in draft-liu-v6ops-running-multiple-prefixes.

Ok, then I think that's a tricky one - we can not decrease the number of 
bits in IPv6 address, nor get the end hosts to go back to ARP for address 
resolution :-)

Also, if iOS 8.0 indeed changes the MAC address upon the wireless 
connection, the problem is just about to get much worse and not 
IPv6-specific ;-)

So, I think there are several means to combat the lookup space churn:

1) IPv6-only networks. We can't do that today and IPv4 is relatively not 
*so* much but it's something.

2) Table maintenance/cleanup mechanisms. These appear to work today 
reasonably.

3) provisioning hardware according to anticipated scale - and either using 
larger router boxes, or splitting large L3 domains into several smaller 
segments, like Mikael mentioned.

4) using DHCPv6-only allocation of addresses, with one address per host 
assignment.

Do you think there is something more that is needed besides the above 
approaches to steer the addresses on the host ?

--a

---

From: "Liubing (Leo)" <[email-masked]>
Date: Fri, 11 Jul 2014 08:11:25 +0000

Hi Andrew,

> The wireless clients are addressed with SLAAC, and are outside of my control
> (conference attendees).
> 
> 1) maximally high lifetimes (RAs are throttled on their way to the clients
> down to 1 RA in 10 minutes).
> 
> 2) clearing on-link bit: results in clients never trying to resolve the other
> clients' addresses => less memory usage on the portable devices, smaller ND
> caches there.
> 
> 3) default no-autoconfig: to force the necessity to add the prefixes manually,
> to force the operator to always have the "ipv6 nd prefix ..."
> with the correct config.
> 
> there was a catastrophic failure no the main pair of the wireless controllers.
> That VLAN had a mirrored configuration.
> Experimentally, I've found such a configuration to cause a rapid deprecation
> of the "wrong" prefix on a iOS and OSX, resulting in minimal disruption to
> them in case of switchover.
> 
> 5) interface route for the prefix, to send the traffic to it (configuring the
> prefix "off-link" had removed the directly connected route).

[Bing] Andrew, thanks much for sharing the details. 

> Right. I was well within the constraints for the first hop, so opted to not have
> DHCPv6. If I wanted a tighter control, I'd have turned off SLAAC and used
> stateful DHCPv6. Maybe I'll experiment with it this year.
> 
> - the dips are the the night times, a lot of internal apps were IPv4 so the
> neighbor entries went away)
> 
> As I said Apple devices (they are the most aggressive today wrt the privacy
> addresses) are ~50% of the network, so maybe in another setup the figures
> would be different.
> 

[Bing] The problem was encountered by the co-author (Yang Bo) when doing projects in some campus/enterprise networks. I'm sorry we don't have the specific statistics data like above, since we don't own the networks. 

Now there are some enterprise/campus networks under real use or considering using L2 networks. Some are aiming at better user isolation through VLANs (some even consider QinQ mechanism); while some are aiming less configuration/management than the traditional L3 networks. So there would be thousands of hosts aggregated to the core switch (normally there are two core switches stacked together, but only share one cache space). As IPv6 is beginning real use, for example, some of the campus networks are already dual-stack, and the majority of the hosts are Win 7, we once observed in one campus that DHCPv6/SLAAC are both enabled, each Win 7 host had 4 IPv6 addr (SLAAC+DHCPv6+Privacy+link-local)+1 IPv4 addr. 

Current high-end switch can certainly fulfill the thousands of users' IP-MAC entries, however, the forwarding performance would be a significant redundancy, which would cost more budget than expected. This is something that won't happen in IPv4, so I thought maybe this could be considered as an IPv6 networks operational issue, and took the issue into the draft and gain some opinions in the list.

> > Plus, an IPv6-MAC binding would cost more cache space than an IPv4-MAC
> binding (2-4 times due to implementation).
> >
> Ok, then I think that's a tricky one - we can not decrease the number of bits
> in IPv6 address, nor get the end hosts to go back to ARP for address
> resolution :-)
>

[Bing] Thanks for the proposed approaches. It seems that's what we can do so far in operation perspective.
As the opinions raised in the mailing list, people tend to consider it as a network design issue rather than operational issue. I will reflect the opinions in the next version of the draft.

Best regards,
Bing

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 11 Jul 2014 10:40:36 +0200

Hi Leo,

On Fri, 11 Jul 2014, Liubing (Leo) wrote:

>
> Now there are some enterprise/campus networks under real use or 
>considering using L2 networks. Some are aiming at better user isolation
>through VLANs (some even consider QinQ mechanism); while some are aiming 
>less configuration/management than the traditional L3 networks. So there 
>would be thousands of hosts aggregated to the core switch (normally there 
>are two core switches stacked together, but only share one cache space). 
>As IPv6 is beginning real use, for example, some of the campus networks 
>are already dual-stack, and the majority of the hosts are Win 7, we once 
>observed in one campus that DHCPv6/SLAAC are both enabled, each Win 7 
>host had 4 IPv6 addr (SLAAC+DHCPv6+Privacy+link-local)+1 IPv4 addr.

If the majority of the hosts are Win 7, and are under the control of the 
administrator, this looks more like a misconfiguration rather than 
anything else: clear the "A" bit on the prefix, and they'll half the
address usage - down to just link-local and DHCPv6-based.

>
> Current high-end switch can certainly fulfill the thousands of users' 
>IP-MAC entries, however, the forwarding performance would be a 
>significant redundancy, which would cost more budget than expected. This 
>is something that won't happen in IPv4, so I thought maybe this could be 
>considered as an IPv6 networks operational issue, and took the issue into 
>the draft and gain some opinions in the list.

I'd say it's a question of different scaling requirements, depending on 
the network design.

--a

---

From: "Liubing (Leo)" <[email-masked]>
Date: Fri, 11 Jul 2014 09:40:04 +0000

Hi Andrew,

> >through VLANs (some even consider QinQ mechanism); while some are
> >aiming less configuration/management than the traditional L3 networks.
> >So there would be thousands of hosts aggregated to the core switch
> >(normally there are two core switches stacked together, but only share one
> cache space).
> >are already dual-stack, and the majority of the hosts are Win 7, we
> >once observed in one campus that DHCPv6/SLAAC are both enabled, each
> >Win 7 host had 4 IPv6 addr (SLAAC+DHCPv6+Privacy+link-local)+1 IPv4 addr.
> 
> administrator, this looks more like a misconfiguration rather than anything
> else: clear the "A" bit on the prefix, and they'll half the address usage - down
> to just link-local and DHCPv6-based.

[Bing] I can hardly say SLAAC and DHCPv6 co-existing is a misconfiguration, but I agree DHCPv6-only deployment can partly relieve the problem.
However, even DHCPv6-only would have 2 IPv6 addr+1 IPv4 addr, which would cause approximately 5~8 times cache space than IPv4-only.

Best regards,
Bing

---

From: "Eric Vyncke (evyncke)" <[email-masked]>
Date: Fri, 11 Jul 2014 10:28:22 +0000

Regarding cache space (and no cache entry count), please note that the IP
address is only part of the entry, there are many other bytes (counters
for example) which also reside in this entry. So, it is not comparing 4
bytes vs. 16 bytes but rather comparing n+4 bytes vs. n+16 bytes (where n
> 16 in most implementations I would guess)

-éric

On 11/07/14 11:40, "Liubing (Leo)" <[email-masked]> wrote:

>>anything
>[Bing] I can hardly say SLAAC and DHCPv6 co-existing is a
>misconfiguration, but I agree DHCPv6-only deployment can partly relieve
>the problem.
>However, even DHCPv6-only would have 2 IPv6 addr+1 IPv4 addr, which would
>cause approximately 5~8 times cache space than IPv4-only.
>
>Best regards,
>Bing
>

---

From: Jared Mauch <[email-masked]>
Date: Fri, 11 Jul 2014 06:47:56 -0400

Apple believes the coexistence of slaac and dhcp6 are a problem and close defects as "3rd party problem won't fix" when raised with them. 

Jared Mauch

> On Jul 11, 2014, at 5:40 AM, "Liubing (Leo)" <[email-masked]> wrote:
> 
> Hi Andrew,
>

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 11 Jul 2014 14:00:54 +0200

Hi Leo,

On Fri, 11 Jul 2014, Liubing (Leo) wrote:

> [Bing] I can hardly say SLAAC and DHCPv6 co-existing is a misconfiguration,

Note, I said "for this specific case". If you control your devices *and* 
devices are DHCPv6-capable *and* you want to minimize the number of 
addresses using SLAAC is not ideal.

> but I agree DHCPv6-only deployment can partly relieve the problem.
> However, even DHCPv6-only would have 2 IPv6 addr+1 IPv4 addr, which 
> would cause approximately 5~8 times cache space than IPv4-only.

You can work towards turning off IPv4.

Past that - it's just protocol's properties.

--a

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 11 Jul 2014 15:06:59 +0200

> Apple believes the coexistence of slaac and dhcp6 are a problem and close defects as "3rd party problem won't fix" when raised with them.

Looks like this might be a useful addition to the 
http://tools.ietf.org/html/draft-liu-v6ops-dhcpv6-slaac-guidance-01#section-3.4, 
or, rather, one more problem in section 2 and reference it from section 
3.4.

--a

---

From: "Liubing (Leo)" <[email-masked]>
Date: Mon, 14 Jul 2014 01:26:38 +0000

Hi Jared,

> Apple believes the coexistence of slaac and dhcp6 are a problem and close
> defects as "3rd party problem won't fix" when raised with them.

[Bing] This is interesting. Would you mind providing a pointer for more detailed information? Does Apple consider dhcpv6/slaac co-existence as a problem in general, or just in some specific cases? 

Best regards,
Bing

---

From: "Liubing (Leo)" <[email-masked]>
Date: Mon, 14 Jul 2014 01:33:52 +0000

Hi Andrew,

> > misconfiguration,
> Note, I said "for this specific case". If you control your devices *and* devices
> are DHCPv6-capable *and* you want to minimize the number of addresses
> using SLAAC is not ideal.
[Bing] For this case, I think you're right. It might be good to add some texts regarding to the cache table problem in http://tools.ietf.org/html/draft-liu-v6ops-dhcpv6-slaac-guidance-01

Thanks.

Best regards,
Bing