Subject: Re: IPv6 deployment [was Re: Recent Internet governance events]

From: [email-masked] (Noel Chiappa)
Date: Fri, 22 Nov 2013 11:46:27 -0500 (EST)

> From: Ted Lemon <[email-masked]>

    > CGNs are expensive. Why would people prefer to maintain them if the
    > IPv6 infrastructure was working? 

Cheaper than customer service calls.

Even if the ISP's system does something clever like allocate the customer
only an IPv6 address if it only sees IPv6 traffic from them, if the customer
tries to reach an IPv4 only asset, the ISP is going to have to have a NAT
device of some kind anyway (else customer service calls will ensue). Once you
have to have a NAT anyway...

	Noel

---

From: Ted Lemon <[email-masked]>
Date: Fri, 22 Nov 2013 11:59:16 -0500

> Cheaper than customer service calls.

That's almost certainly true now.   I'm not talking about now.   Really, the big problem with CGNs is that they don't scale—this is why lw4over6 and MAP represent a significant and useful innovation.

What I expect to see happen in the market is that in the next decade, CGN will start to seem old-fashioned, and more and more what will be deployed will be stateless port-sharing NAT hardware in the core, supported by stateful NATs at the CP, just like we were doing before the CGN idea got popular.

The nice thing about stateless NATs in the core is that they can be scaled up and down according to need.   So as more traffic is carried over the native IPv6 network, you can start switching off boxes in the core.   You can track the demand, so you don't switch off boxes that are needed.   You can even goose the switchover by running your IPv4 NATs closer to capacity, forcing traffic that could go over either transport to go over v6, because it's working better, and Happy Eyeballs can tell.

---

From: John C Klensin <[email-masked]>
Date: Fri, 22 Nov 2013 13:02:29 -0500

--On Friday, November 22, 2013 11:46 -0500 Noel Chiappa
<[email-masked]> wrote:

>     > CGNs are expensive. Why would people prefer to maintain
> them if the     > IPv6 infrastructure was working? 
> 
> Even if the ISP's system does something clever like allocate
> the customer only an IPv6 address if it only sees IPv6 traffic
> from them, if the customer tries to reach an IPv4 only asset,
> the ISP is going to have to have a NAT device of some kind
> anyway (else customer service calls will ensue). Once you have
> to have a NAT anyway...

In addition, an important part of the customer service cost
story is the training and skill level required of the support
people.  Having customer service support for two network
technologies (or, of you prefer, two networks) is vastly more
expensive than supporting one.  And, once one figures out how to
train and organize people to support CGNs, much of that cost is
sunk: while there is turnover and one therefore has to keep
training new people, the costs are entirely different from
retraining everyone again to support a pure IPv6 network.

Looked at purely from the standpoint of economic incentives,
forklift upgrades to an end state are an expensive and risky
step.  The right time to do one is as early as possible after
the target technology seems stable and the transition strategy
seems clear.  And the right transition strategies to promote
that are actual transition strategies, not "keep bad things from
happening for a while longer while we make a plan" ones.  As
soon as one goes significantly down the path of delaying making
real changes, those strategies tend to take on their own inertia
because, on any given day, it will almost certainly be cheaper
to put up with the problems they cause and continue tolerating
the pain (both of which one has presumably gotten used to) than
make a major change.  One can argue that those delay mechanisms
won't scale or won't last forever for other reasons and probably
be right, but scenarios based on those assumptions strongly
resemble frog-boiling and the outcomes when there are no choices
other than to change something may not be the one that one likes.

I'd be more optimistic right now if a highly fragmented set of
networks, with duplicate or overlapped address spaces,
interconnected by application-layer gateways, were beyond the
state of the art.  But, unfortunately, we know perfectly well
how to do those, even down to the level of running code.

For the record, I've never been an IPv6 hater.  Because of the
economic and incentive arguments, I think we would have been
better off with protocol design elements that would either have
provided an easier and more obvious transition strategy or more
compelling advantages for conversion than a "running out"
scenario.  I said so at the time, but deferred to people who I
thought understood the issues better than I did.  I still hope
that it all works out, that people make the right decisions when
faced with the scaling and other problems and that we end up
with an IPv6-based network with good end to end properties.   I
just think it is important that we be as realistic and
clear-headed as possible about where we are now as a basis for
what we do, or encourage others to do, next.  That state, to
paraphrase Randy, is not a happy one, nor one that will be
significantly improved by cheerleading.

    john

---

From: Richard Barnes <[email-masked]>
Date: Fri, 22 Nov 2013 13:32:21 -0500

For a feel for some of reasons your question sounds naïve, see Geoff
Huston's presentation on IPv6 and CGNs at APNIC 32 all the way back in 2011
(trust me, it's entertaining, as Geoff usually is):
Slides: <http://labs.apnic.net/presentations/store/2011-08-30-exhaustion.pdf
Recording: <http://webcast.apnic.net/meetings/32/opening-hinted.mov>

On Fri, Nov 22, 2013 at 11:59 AM, Ted Lemon <[email-masked]> wrote:

> That's almost certainly true now.   I'm not talking about now.   Really,
> the big problem with CGNs is that they don't scale—this is why lw4over6 and
> MAP represent a significant and useful innovation.
>
> What I expect to see happen in the market is that in the next decade, CGN
> will start to seem old-fashioned, and more and more what will be deployed
> will be stateless port-sharing NAT hardware in the core, supported by
> stateful NATs at the CP, just like we were doing before the CGN idea got
> popular.
>
> The nice thing about stateless NATs in the core is that they can be scaled
> up and down according to need.   So as more traffic is carried over the
> native IPv6 network, you can start switching off boxes in the core.   You
> can track the demand, so you don't switch off boxes that are needed.   You
> can even goose the switchover by running your IPv4 NATs closer to capacity,
> forcing traffic that could go over either transport to go over v6, because
> it's working better, and Happy Eyeballs can tell.
>
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 22 Nov 2013 14:29:55 -0500

> MAP represent a significant and useful innovation.

They will scale fine just like the dialup pools have scaled fine.

If the switchover is managed right, there will be no difference between the
functionality of an IPv6+CGN connection and an IPv4 connection. At some
point in the future, maybe in 5 years time, the IPv6+CGN connection will be
on the modern hardware and have the better performance than the legacy IPv4
connection and that is what will create demand to switch.

As the transition continues the percentage of the net that can only be
reached through IPv4 will decline from the current 95% to more like 5%. As
that happens, the load on the CGNs will gradually fall as less traffic
needs to be routed through them.

There was a time when the Internet was serviced mostly though 48Kb modems.
providers had rack upon rack of the thing and were doubling capacity every
12 months back in '95. But then the demand started to taper off and these
days it is legacy technology.

Or to look at it another way, the entire PSTN system plus SIP is nothing
more than a giant legacy transition infrastructure right now. Eventually
even the telephone numbers will go away.

Building carrier grade NAT and carrier grade gateways is essentially the
same problem. People might build boxes that only do one but anyone buying
them is taking a risk that they are not going to be forced into upgrading
them to support transition to IPv6.

One of the consequences of what happened yesterday in the US Senate is that
the Republican party just lost their ability to block appointment of FCC
commissioners who support and judges who recognize the right of the FCC to
implement policies to promote net neutrality and technical standards to
promote competition.

-- 
Website: http://hallambaker.com/

---

From: Ted Lemon <[email-masked]>
Date: Fri, 22 Nov 2013 16:11:11 -0500

> They will scale fine just like the dialup pools have scaled fine.

Dialup pools are stateless.   CGNs are stateful.   They do not scale the same way.   This is not to say that you _can't_ scale CGNs—it's just easier to scale stateless-core NATs.   I say this based on zero operational experience, of course... :)

---

From: Warren Kumari <[email-masked]>
Date: Fri, 22 Nov 2013 19:12:57 -0300

On Nov 22, 2013, at 6:11 PM, Ted Lemon <[email-masked]> wrote:

> Dialup pools are stateless.   CGNs are stateful.   They do not scale the same way.   This is not to say that you _can't_ scale CGNs—

So, something that has always confused me abut the CGN deployment discussions and scaling is the number of customers (victims?!) that people want to put behind an IP…

If you are an operating ISP with e.g a /18 you can have ~16,000 customers[0]. Great, you are still growing, and want to add another 10,000 users, good for you.
For some reason at this point many ISPs start talking about putting on the order of 100s of users behind an IP, then the discussion turns into port starvation and scaling and such… 

What's wrong with putting 2 users behind each IP? Are you really planning on doubling your size *before* significant advances in v6 deployment and CGN scaling come about? Yes? Ok, so put 4 users behind one IP (note, I did not say "device") -- are you really planning on quadrupling in the next few years? And if so, are you hiring? :-P

Seriously, I don't get the "If we deploy CGN's we have to cram as many users behind one address as possible…" bit -- can anyone enlighten me?

I've run some big NATs (for example, for AOL's corporate network) and yes it sucks, but you can minimize your (and your customer's) pain by overloading as little as possible…. 

> it's just easier to scale stateless-core NATs.   I say this based on zero operational experience, of course... :)

Sure, fair 'nuff, no argument here….

W
[0]: Yes, yes, handwave, handwave, packing issues, infrastructure space, etc. Been there, run a network… Skipping all that for easy of discussion. 

--
There were such things as dwarf gods. Dwarfs were not a naturally religious species, but in a world where pit props could crack without warning and pockets of fire damp could suddenly explode they'd seen the need for gods as the sort of supernatural equivalent of a hard hat. Besides, when you hit your thumb with an eight-pound hammer it's nice to be able to blaspheme. It takes a very special and straong-minded kind of atheist to jump up and down with their hand clasped under their other armpit and shout, "Oh, random-fluctuations-in-the-space-time-continuum!" or "Aaargh, primitive-and-outmoded-concept on a crutch!"
  -- Terry Pratchett

---

From: Ralph Droms <[email-masked]>
Date: Fri, 22 Nov 2013 17:24:50 -0500

> On Nov 22, 2013, at 5:12 PM, Warren Kumari <[email-masked]> wrote:
> 
> 

Warren - thank you for asking this question.  I asked it for several years, never got a satisfactory answer (never got much of any answer, really) and stopped asking...

- Ralph

---

From: "cb.list6" <[email-masked]>
Date: Fri, 22 Nov 2013 14:50:49 -0800

> Warren - thank you for asking this question.  I asked it for several years, never got a satisfactory answer (never got much of any answer, really) and stopped asking...

Here is a hint http://www.flickr.com/photos/ericsson_images/10795593603/in/set-72157637442787564

Oh, and, here is a usecase:  Network operator wants to start are
regional mobile network in Europe or APAC:  APNIC or RIPE will give
you a /22 for your entire business.

CB

CB

---

From: Warren Kumari <[email-masked]>
Date: Fri, 22 Nov 2013 20:22:12 -0300

> Here is a hint http://www.flickr.com/photos/ericsson_images/10795593603/in/set-72157637442787564

Ok, interesting… I hadn't been thinking of the mobile folk. 

So, if you happened to have *all* the 1.9 billion widgets at the moment (and were able to IP them now), putting 4x as many thingies behind an address would take you to 7.6B, or "good till ~2021ish"? 

> you a /22 for your entire business.

Ok, that's more of an issue. Presumably if you are a startup regional mobile network in Eu or APAC you'd also purchase some additional addresses though?

But yup,  I can see how the world is different for mobile folk, and now have some more clue. Thanks.
The folk I had been talking to were primarily large cable type players, who already have significant space and network.

W

-- 
"Does Emacs have the Buddha nature? Why not? It has bloody well everything else..."

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 22 Nov 2013 19:39:59 -0500

On Fri, Nov 22, 2013 at 5:12 PM, Warren Kumari <[email-masked]> wrote:

> > Dialup pools are stateless.   CGNs are stateful.   They do not scale the
> same way.   This is not to say that you _can't_ scale CGNs—
>
> So, something that has always confused me abut the CGN deployment
> discussions and scaling is the number of customers (victims?!) that people
> want to put behind an IP…
>
> If you are an operating ISP with e.g a /18 you can have ~16,000
> customers[0]. Great, you are still growing, and want to add another 10,000
> users, good for you.
> For some reason at this point many ISPs start talking about putting on the
> order of 100s of users behind an IP, then the discussion turns into port
> starvation and scaling and such…
>
>
> What's wrong with putting 2 users behind each IP? Are you really planning
> on doubling your size *before* significant advances in v6 deployment and
> CGN scaling come about? Yes? Ok, so put 4 users behind one IP (note, I did
> not say "device") -- are you really planning on quadrupling in the next few
> years? And if so, are you hiring? :-P
>

Sharing a single IP among a group of users is clearly not optimal as the
probability of contention is much higher than sharing a pool of IP among a
group of users. But assuming you are saying an average of n users per IP, I
don't think that is going to be much of a problem. A 16 IP address pool
should be able to support 2048 customers without anyone noticing much. That
would almost certainly be enough to stretch the remaining IPv4 addresses to
cover the transition needs.

> Seriously, I don't get the "If we deploy CGN's we have to cram as many
> users behind one address as possible…" bit -- can anyone enlighten me?
>
> I've run some big NATs (for example, for AOL's corporate network) and yes
> it sucks, but you can minimize your (and your customer's) pain by
> overloading as little as possible….
>
>
> > it's just easier to scale stateless-core NATs.   I say this based on
> zero operational experience, of course... :)
>
> [0]: Yes, yes, handwave, handwave, packing issues, infrastructure space,
> etc. Been there, run a network… Skipping all that for easy of discussion.
>
>
> --
> There were such things as dwarf gods. Dwarfs were not a naturally
> religious species, but in a world where pit props could crack without
> warning and pockets of fire damp could suddenly explode they'd seen the
> need for gods as the sort of supernatural equivalent of a hard hat.
> Besides, when you hit your thumb with an eight-pound hammer it's nice to be
> able to blaspheme. It takes a very special and straong-minded kind of
> atheist to jump up and down with their hand clasped under their other
> armpit and shout, "Oh, random-fluctuations-in-the-space-time-continuum!" or
> "Aaargh, primitive-and-outmoded-concept on a crutch!"
>   -- Terry Pratchett
>
>
>

---

From: Ted Lemon <[email-masked]>
Date: Fri, 22 Nov 2013 22:26:14 -0500

> For some reason at this point many ISPs start talking about putting on the order of 100s of users behind an IP, then the discussion turns into port starvation and scaling and such… 

Yes, it does.   However, if all the really grabby services your users are running work over IPv6, that becomes a non-problem.   You only need IPv4 for the long tail stuff, and that generally doesn't consume a bazillion ports at a time.

> What's wrong with putting 2 users behind each IP? Are you really planning on doubling your size *before* significant advances in v6 deployment and CGN scaling come about? Yes? Ok, so put 4 users be…

If indeed your market is static, this is a non-problem, since you already have enough IP addresses to support it.   Apparently this is not true of all ISPs.

> Seriously, I don't get the "If we deploy CGN's we have to cram as many users behind one address as possible…" bit -- can anyone enlighten me?

You cram as many as you have to.   Have to may be dictated by current growth, by planned growth, or by planned non-growth, and you will choose your port allocations accordingly.   Bear in mind that not all the world is flush with IP addresses and fully built out.

> I've run some big NATs (for example, for AOL's corporate network) and yes it sucks, but you can minimize your (and your customer's) pain by overloading as little as possible…. 

Of course.   Nobody is going to deploy any of this stuff if they don't have to!   If you have enough IP addresses, dual stack is fine, and you can tunnel IPv4 if you get to the point where you want to avoid having to manage a dual-stack infrastructure outside of the core.

---

From: John Curran <[email-masked]>
Date: Tue, 26 Nov 2013 12:48:59 -0500

> The folk I had been talking to were primarily large cable type players, who already have significant space and network.

Note - there's also significant progress on the front (at least from some of the providers):

26 Nov 2013 - <http://corporate.comcast.com/comcast-voices/comcasts-xfinity-internet-now-the-worlds-largest-native-ipv6-deployment>

"Today, over 25% (and growing) of Comcast’s Xfinity Internet customers are actively provisioned with native dual stack broadband Internet service.  Native IPv6 support has been deployed to over 75% of our broadband network, and our goal is 100% in early 2014."

FYI,
/John