Subject: Re: [v6ops] new draft: draft-carpenter-v6ops-icp-guidance

From: Tore Anderson <[email-masked]>
Date: Tue, 01 Nov 2011 10:10:05 +0100

* Brian E Carpenter

> That is a very different deployment model than
> draft-carpenter-v6ops-icp-guidance advocates. I can understand that
> there might be unusual circumstances in which it would be chosen, but
> it is so much simpler to dual-stack all servers.

In a purely technical sense, you might be right. However, I've come to
realize that dual-stack is not really a viable transition approach for
us, due to organizational and human considerations. I think the draft is
intended exactly for organizations such as ours btw, as we're a
small/medium-sized ICP.

We have a team of people that works on the network infrastructure (of
which I'm part), and several teams with sysadmins that operate the
customers' servers, operating systems, and the applications running on
them. While my colleagues and I are happy to promote IPv6 by always
assigning an IPv6 prefix to all LANs that are requested by the server
guys, their priorities are very different - they don't want to get
involved more networking than absolutely necessary. So if I give them
two IP stacks to work with, they'll choose one and leave the other one
unused, and not surprisingly, the one they choose to work with is going
to be the most familiar one - IPv4.

I can understand it, too - dual stack is in a sense dual work and dual
complexity. They'd need to configure ACLs twice, monitor everything
twice, accept lots of new failure scenarios and so on. It's real
operational overhead, and nobody really wants that.

If there is an explicit demand from the customer to have IPv6
availability, that changes things somewhat - but usually not more than
causing a dual-stacking of the public internet-facing frontends. Most
servers remain IPv4-only, while only the load balancers/web
servers/whatever gets an additional IPv6 address that can be published
in DNS. In any case, customer requests for IPv6 are few and far between,
and to this date no customer has ever asked us for a dual-stacking of
all of his servers.

After all, dual-stack as a transition mechanism has gained approx. zero
momentum in the decade it's been the recommended IPv6 transition
mechanism. I don't see any reason why that would change any time soon.

I'm instead pursuing a path where I'll give the sysadmins single-stack
IPv6 to work with, and handle IPv4 connectivity as a translation
function in the core network (e.g. using SIIT). We'll see how that works
out in the end, but it has some advantages over dual-stack, in my
opinion: First, it forces the sysadmins to actually make use of IPv6.
Second, it doesn't require them to deal with more than 1 IP protocol.
Third, and perhaps most importantly, it helps a lot with IPv4 depletion.

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 02 Nov 2011 09:27:08 +1300

Grumble. I don't like that conclusion, because it's clear that
the desired end point is native v6 and I don't see how your
approach gets us there.

OK, if the draft becomes a WG draft, I can see that we would
need to add the draft-sunq-v6ops-contents-transition model
to the outside-in scenario, as an alternative to the dual
stack proxy. However, although the proxy may be a bit slower,
it strikes me as a much cleaner solution than NAT64. And
of course it's off-the-shelf running code.

Regards
   Brian

---

From: Mark Andrews <[email-masked]>
Date: Wed, 02 Nov 2011 18:13:23 +1100

> operational overhead, and nobody really wants that.

Our ops people would say you are exagerating the costs.  They have
7+ years of dual stack experience.  It's not zero but is close to
that.
 
Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Tore Anderson <[email-masked]>
Date: Wed, 02 Nov 2011 10:05:33 +0100

* Brian E Carpenter

> the desired end point is native v6 and I don't see how your

Native IPv6 on all servers and used by all applications running on those
servers doesn't get us to native IPv6? Huh? :-)

---

From: Tore Anderson <[email-masked]>
Date: Wed, 02 Nov 2011 10:22:34 +0100

* Mark Andrews

> Our ops people would say you are exagerating the costs.

Considering I didn't mention costs at all, I'm not sure how you came to
that conclusion. I don't believe that dual stack mean you have to double
your operations budget. The added technical complexity is the main
problem for us, not costs.

Not even an hour after I sent my previous e-mail yesterday, a colleague
came over to me asking if I could look into why some accounting cron
jobs had stopped working since last month. Turns out the problem was
that a server had been recently dual-stacked, which caused a download
job from another server (which had been dual-stacked for a long time) to
fail, because the ACLs that allowed the download to happen over IPv4
wasn't properly duplicated in IPv6. Easy enough to fix, and fortunately
not critical at all, but it's an example of a failure scenario you are
open to only when running dual-stack.

Best regards,

---

From: Mark Andrews <[email-masked]>
Date: Wed, 02 Nov 2011 22:11:44 +1100

> open to only when running dual-stack.

I would suggest that your tools are not RFC 1123 compliant as they
didn't cope with a multi-homed servers.  This failure could have
happened with IPv4 only.  It's only been been 22 years since support
for multi-homed servers was recommended.

---

From: Gert Doering <[email-masked]>
Date: Wed, 2 Nov 2011 15:08:56 +0100

Hi,

> 7+ years of dual stack experience.  It's not zero but is close to

Speaking as an Ops person, I really, *really* want to get rid of 
dual-stack.  Double address management, double monitoring, double
testing for internal connections between machines (will it use v4 or
v6 today?), double ACLing things, twice as many opportunities for 
people to make mistakes.  Worse, if one of the protocols isn't set
up right, you might not even notice it right away if the applications
are using the other one today.

I very much like Tore's approach.

(I don't see us actually doing this any time soon - the network is 
not currently set up properly for that - but the idea of running
everything IPv6-only with an IPv4 front-end proxy has lots of merits)

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (89) 32356-444            USt-IdNr.: DE813185279

---

From: Gert Doering <[email-masked]>
Date: Wed, 2 Nov 2011 15:10:02 +0100

Hi,

> for multi-homed servers was recommended.

Cheap shot.

"If you don't like the extra work something causes, it must be because
you / your tools are not up for the job"

Gert Doering
        -- NetMaster

---

From: Qiong <[email-masked]>
Date: Wed, 2 Nov 2011 22:37:04 +0800

Hi, Gert,

On Wed, Nov 2, 2011 at 10:08 PM, Gert Doering <[email-masked]> wrote:

> Fully agree. This is also what we are discussing in
draft-sunq-v6ops-contents-transition-02, in which native IPv4/IPv6 server
models are introduced. Looking forward to your comments as well.

> For IPv6-only server, stateless IVI can be deployed with good scalablity
and application-independent feature.

Best wishes

Qiong

Gert Doering
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: Cameron Byrne <[email-masked]>
Date: Wed, 2 Nov 2011 08:13:32 -0700

> open to only when running dual-stack.

In case folks want to know the hard cold cash cost of using ipv4, ask
Microsoft

http://www.bbc.co.uk/news/technology-12859585

Consider that the floor price, now that it went through, the model is
proven and prices will rise.

Ipv6 is essentially free, especially on a per address basis :-)

Not only that, the ROI timescale for ipv6 payback on initial investments is
as long as you think ipv6 will be around. Ipv4 payback ...not quite as long.

Cb

---

From: Behcet Sarikaya <[email-masked]>
Date: Wed, 2 Nov 2011 14:38:44 -0500

This part of the news is scary:

Registries that oversee the allocation of net addresses are also working on
plans for a re-circulation system that takes IPv4 addresses from firms that
are using IPv6 and releases them for use by others.

On Wed, Nov 2, 2011 at 10:13 AM, Cameron Byrne <[email-masked]> wrote:

> Not only that, the ROI timescale for ipv6 payback on initial investments
> is as long as you think ipv6 will be around. Ipv4 payback ...not quite as
> long.
>
> Cb
>
> > Best regards,
> > --
> > Tore Anderson

---

From: Lorenzo Colitti <[email-masked]>
Date: Wed, 2 Nov 2011 12:41:55 -0700

> for multi-homed servers was recommended.

And yet, arguing with reality is not going to help you change reality. It
will only make you out of touch.

---

From: Gert Doering <[email-masked]>
Date: Wed, 2 Nov 2011 20:44:46 +0100

Hi,

> are using IPv6 and releases them for use by others.

Uh, what exactly is scary about that?  If company A doesn't need their
addresses anymore, and company B does, and the RIRs keep track of 
"who is using what", this sounds like a reasonable thing to me.

The nitty details of individual transfer policies can indeed be scary,
but that's certainly not what you were referring to.

Gert Doering
        -- NetMaster

---

From: Joel jaeggli <[email-masked]>
Date: Wed, 02 Nov 2011 12:45:43 -0700

On 11/2/11 12:38 , Behcet Sarikaya wrote:
> Registries that oversee the allocation of net addresses are also working
> on plans for a re-circulation system that takes IPv4 addresses from
> firms that are using IPv6 and releases them for use by others.

If you return addresses to an RIR  or transfer them to a third party,
you can expect them to be reused... I don't expect that you'd want to
use the BBC as your authoritative source of ip addresses policy news
however.

> On Wed, Nov 2, 2011 at 10:13 AM, Cameron Byrne <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
> 
>     On Nov 2, 2011 2:22 AM, "Tore Anderson"
>     <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     >
>     > * Mark Andrews
>     >
>     > > In message <[email-masked]
>     <mailto:[email-masked]>>, Tore Anderson writes:
>     > >>
>     In case folks want to know the hard cold cash cost of using ipv4,
>     ask Microsoft
> 
>     Consider that the floor price, now that it went through, the model
>     is proven and prices will rise.
> 
>     Not only that, the ROI timescale for ipv6 payback on initial
>     investments is as long as you think ipv6 will be around. Ipv4
>     payback ...not quite as long.
> 
>     Cb
> 
>     > Best regards,
>     > --
>     > Tore Anderson
>     > [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 03 Nov 2011 09:17:47 +1300

It was this that triggered my comment:

>> and not surprisingly, the one they choose to work with is going

I agree that what you describe is a good approach. Sorry that wasn't
clear (I misphrased it; yesterday was a busy day).

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 03 Nov 2011 09:26:49 +1300

Hi Gert,

> everything IPv6-only with an IPv4 front-end proxy has lots of merits)

However, that would be a hard sell to a conservative ICP in 2011.
I certainly agree that this is a desirable end state, but my guess
is that it's at least ten years away as a mainstream approach.

Maybe we should add a new phrase to go with 'early adopter',
such as 'early rejecter".

   Brian

---

From: Mark Andrews <[email-masked]>
Date: Thu, 03 Nov 2011 11:01:13 +1100

> you / your tools are not up for the job"

The entire 6to4 fiasco with web clients is because they did not
handle multi-homed servers.  If they handled multi-homed servers
(a bug that was reported long before there was much if any dual
stacked servers) they would have handled 6to4 issues.  They would
have connected / fallen back to IPv4 when the IPv6 path was blocked
due to stupid firewall blocking the encapulated packets.

Dual stack just means you clients need to work well with multi-homed
servers.

Mark

---

From: Philip Homburg <[email-masked]>
Date: Thu, 03 Nov 2011 10:02:01 +0100

>Dual stack just means you clients need to work well with multi-homed

I think this is a bit silly. 

The best thing we have today to let a host deal with 6to4 brokenness is HE
and even that is not a fully general solution.

In all the time there where multi-homed services (I think multi-homed servers
is the wrong term given how redundancy is typically implemented) there was
almost no progress in destination selection.

The only games in town were DNS, which is also rather limited in this context
(where IPv6 does not work at all) and anycast routing (same story).

If we ignore specialized solutions and consider what is shipped with major
operating systems as the state of the art, then the conclusion is that we
simply do not know how to do multi-homing.

So until major operating systems ship with libraries or other services that
make multi-homing just work, I'd suggest the we consider any kind of 
multi-homing as a potential reliability problem.

And in that model, if you can drop IPv4 internally and replace it with just an
external IPv4-to-IPv6 traslator then things probably get better.

---

From: Mark Andrews <[email-masked]>
Date: Thu, 03 Nov 2011 20:55:24 +1100

> almost no progress in destination selection.

You are looking for optimal.  It would be nice if applications just
tried alterative addresses.  Just because one is unreachable it
says nothing about the other.

> (where IPv6 does not work at all) and anycast routing (same story).

DNS works fine for IPv6 both at the transport and data level.

> external IPv4-to-IPv6 traslator then things probably get better.

OS's have shipped with libraries that have returned multiple addresses
for 2+ decades.  When the application only tries the first you can
blame the application not the the OS.

The complaint here was the data didn't transfer when there were two
addresses and the acl path blocked one of the transports.  The other
transport was wide open.  The application should have tried it.

Mark

---

From: Philip Homburg <[email-masked]>
Date: Thu, 03 Nov 2011 11:40:24 +0100

>DNS works fine for IPv6 both at the transport and data level.

Just trying alternative addresses is nowhere near enough. For anything
interactive, users expect real time response. 

Just add an unreachable DNS resolver to the head of the list of resolvers in
/etc/resolv.conf and see how quickly you want to remove it again.

>blame the application not the the OS.

I agree that application should do that at some basic level.

>transport was wide open.  The application should have tried it.

There are ACLs at many levels. Retrying for errors at the network level is
common. But if you get an access denied at the application level do you retry?

Suppose a web browser get a 404, do you expect it to try all other addresses to
see if one would work?

---

From: Gert Doering <[email-masked]>
Date: Thu, 3 Nov 2011 11:58:41 +0100

Hi

> due to stupid firewall blocking the encapulated packets.

One of us seems to be a bit decoupled from reality.

All the clients (that have been adapted to IPv6 in the first place) *do* 
fall over if IPv6 connection fails, and they used to do so by following 
the sample implementation in the RFC that explained that you should use 
getaddrinfo() and then loop on the results, trying to connect to each 
address in sequence.

And as a result, they fall over *after timeouting* on connect(), again
following the RFCs on how long a TCP connect should try.

Which works fine if you either have connectivity, or have an ICMP error
or TCP reset coming back, but "firewalls blocking the encapsulated packets"
exactly do not do so - so it's blackholing, and 30 seconds to 1.5 minutes
wait time before trying the next address in the list  (which is completely
unacceptable for a content provider trying to serve web stuff).

So where exactly did "the web clients" fail to implement the authoritative
RFCs on this?

Gert Doering
        -- NetMaster

---

From: Mark Andrews <[email-masked]>
Date: Thu, 03 Nov 2011 23:18:10 +1100

In message <[email-masked]>, Gert Doering writes:
> All the clients (that have been adapted to IPv6 in the first place) *do*=20
> fall over if IPv6 connection fails, and they used to do so by following=20
> the sample implementation in the RFC that explained that you should use=20
> getaddrinfo() and then loop on the results, trying to connect to each=20
> address in sequence.

It's sample code.  Not the "Holy Writ".  If you needed to fall over
faster you were free to do so.  There have been application that
have done so.

> following the RFCs on how long a TCP connect should try.

Yet these browser vendors have years old bug reports saying that
their performance sucked, especially when a IPv6 address was
unreachable, and didn't do anything about it.

> So where exactly did "the web clients" fail to implement the authoritative

So did the RFC's also say they couldn't remember which address
worked and using for the next element of the page?  The browser
developers just didn't think about this.

> --=20
> --IfZ+tgy+ooJOsAAy
> Content-Type: application/pgp-signature
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (FreeBSD)
> 
> iQCVAwUBTrJz4akuBuNlUUl1AQLnWwP/akivmkkzuE+8V0Qe3qQUHS1KZNmROh/o
> fXK2JKcX57QVy1DDk3cDbCQrWpHh2WsEVIk0QgUgqlJgi8MpsFIf4PLO/TKt/v5o
> DOah40fcB5945QwztImd4cQ1Ew64WlK++BH+gLP7ekBOTAQPD/E4qnONQKsn3r7k
> SOom4o4Kfwc=
> =4IF8
> -----END PGP SIGNATURE-----
> 
> --IfZ+tgy+ooJOsAAy--

---

From: Philip Homburg <[email-masked]>
Date: Thu, 03 Nov 2011 13:52:56 +0100

>faster you were free to do so.  There have been application that

Yes, I can see how that works. First we encourage application writers to just
come up with their own solutions, and when they have rolled out something
that happens to be broken we can complain about it and write BCPs about how
it should be done.

We would never, like, write a HE specification before it actually became an
issue.

>unreachable, and didn't do anything about it.

And in all that time, nobody bothered to write a BCP about how it should be
done.

In my experience, implementing HE was hard work. 

So far we have a set of requirements, which is good, but nothing telling
implementors how it should be done. 

And if it was trivial, Apple would not have gotten it wrong.

>developers just didn't think about this.

The days that the contents of a website came form just a single server are 
long gone.

The 'nice' thing about applications caching name to address mappings is that
they don't know the DNS TTLs. Creating all kinds of other problems. 
I think that for the same origin policy, browsers already have to use the same
IP address as they used earlier, but I don't know if they do that in practice.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 04 Nov 2011 10:29:32 +1300

I'm failing to see anything specifically relevant to advice for ICPs
in this sub-thread.

In the document under discussion, we refer to AAAA whitelisting
as a possible tactic and state that:

   There is little an ICP can do to deal with client-side or remote ISP
   deficiencies in IPv6 support, but it is hoped that the "happy
   eyeballs" [I-D.ietf-v6ops-happy-eyeballs] approach will improve the
   ability for clients to deal with such problems.

We probably need to add NAT64 (draft-sunq-v6ops-contents-transition)
as an additional scenario. Is there anything else in this sub-thread
that constitutes useful advice for ICPs?

If not, please change the Subject header ;-)

Regards
   Brian

---

From: Mark Andrews <[email-masked]>
Date: Fri, 04 Nov 2011 09:08:04 +1100

> that happens to be broken we can complain about it and write BCPs about how

Web browers were making parallel connections to web servers to speed up
performance yet they waited 30+ seconds for a initial connect to succeed
when they know that 99.9999% of connections won't succeed if they don't
succeed in the first 2 seconds and users are hitting stop after 5 seconds.

They just weren't meeting their users needs.

> We would never, like, write a HE specification before it actually became an

Its been a issue for years.  People have been saying we can't deploy
IPv6 for years because browsers don't failover properly.  Browsers
have had toggles to turn off IPv6 for years.  Those toggles were
added because it was a issue.  It's becoming a critical issue because
most of the net will be dual stacked rsn and people are beging to
see they can't delay going dual stack any longer and are there for
kicking up a bigger stink to get the browsers fixed.

The browser vendors choose not to fix the issue, instead they hid
the issue for as long as they could.

HE pushes the wait done to the 100's of milliseconds.  I order of
magnitude faster than application like ssh have been doing for years
now.

> And in all that time, nobody bothered to write a BCP about how it should be

Because it shouldn't have needed a BCP.  Non blocking connect was
introduces in BSD 4.3 if my memory is correct.  I know I've been
using non blocking connects for about as long as they have existed
as I've had applications that just can't afford to block.

> And if it was trivial, Apple would not have gotten it wrong.

Apple tried to be over smart and didn't listen to the advice given
to them.  Apple have IPv6 aware applications that don't work in a
IPv6 only environment.  Apple have TCP based applications, that
they have written, that don't fallback to IPv4 if they can't reach
the server over IPv6.  They just don't try the other address.

> IP address as they used earlier, but I don't know if they do that in practice

And applications are free to retrieve the DNS TTL if they want.  The tools
for doing this are decades old.  Most applications don't need to know the
DNS TTL.  They make a single connection.

For many applications waiting 30 seconds for connect to fail is not a
issue especially if they are doing other things at the same time.

---

From: Philip Homburg <[email-masked]>
Date: Fri, 04 Nov 2011 10:42:46 +0100

>kicking up a bigger stink to get the browsers fixed.

I don't like this line of reasoning. If this is something we care about, we
should write a RFC detailing how it should be done. If we can't bothered to
write an RFC, why would anyone else care?

For browser vendors it worked for years, and it still works. Switching off IPv6
solves the problem. No need to invent complicated trickery.

As for you suggesting to reduce timeouts to 2 seconds instead of 30. That may
help during a short outage, but if my browser would always wait 2 second due
to IPv6 failing then I would quickly disable IPv6 (or fix IPv6).

So, changing that timeout doesn't fix anything. And it may actually hurt,
because there maybe situations where users rely on the 30 second timeout.

>magnitude faster than application like ssh have been doing for years

And this is the point where it actuaslly becomes useful. Certainly with
stateful HE, it will just work around broken connections.

>as I've had applications that just can't afford to block.

Browsers have been using non blocking connects as long as the first versions of
Netscape. That's not the issue.

HE is a new game in town. It would have been nice if HE style connecting was
suggested in the IPv6 socket RFCs. But it wasn't. 

>the server over IPv6.  They just don't try the other address.

That's obviously a bug. But I was referring to case where OSX will connect over
IPv4 when a perfectly good IPv6 link is availabe as well.

>DNS TTL.  They make a single connection.

And yet, getaddrinfo doesn't return the TTL. Another example where what's in
the RFC is just incomplete.

---

From: Philip Homburg <[email-masked]>
Date: Fri, 04 Nov 2011 10:48:59 +0100

>I'm failing to see anything specifically relevant to advice for ICPs

Yes, quite right.

I'm not an ICP, but what seems to be missing from this draft is explict 
operator experience. Though IPv6 day was mostly smooth and boring there were
some interesting gotchas. It may be worth collecting those.

This draft seems to have the obvious stuff. But the stuff you would not think
of yourself and that you normally only find out the hard way seems to be
missing.