Subject: Re: [DNSOP] FYI: DNSOPS presentation

From: Igor Gashinsky <[email-masked]>
Date: Fri, 2 Apr 2010 00:22:11 -0400 (EDT)

On Wed, 31 Mar 2010, Jason Livingood wrote:

:: Igor - How do you define broken?  And what technical issues do you believe
:: underlie this condition?

This is actually very subjective, and I suspect would differ from provider 
to provider, so, these are *my* own shot at this definition, not my 
employer's..

"Broken" = any time a user has a worse experience in the presense of ipv6 
(AAAA) then over ipv4 (A) in any "measurable" way. 

Measurable ways:

1) connectivity: ie user can't connect to the site if the site has 
   ipv6, compared to a site that only has ipv4 (this one is very simple)

2) latency: A page takes noticebly longer to load in the presense of ipv6, 
   compared to an ipv4-only site. So, to measure this, take the total page 
   load time of ipv4, and compare it to the load times of ipv6.

   So, what is "noticebly longer"?  (this is a lot more subjective)

        a) studies have shown that users easily notice (and are 
	   impacted by) extra latency of 100-200ms... see 
	http://en.oreilly.com/velocity2009/public/schedule/detail/8523

	b) for those users that connect over low-bandwidth or far away 
	links, let's set another upper bound of 10% performance penalty 
	(so, basically, extra latecy of 100ms is the cut-of for total 
	page render of 1s, and 500ms is the cut-off for total page render 
	of 5s)

We can debate if those cut-offs are right, or if 2x of those is where the 
line should be drawn, but we'd still be in the same order of magnitude, so 
let's just stick to that entire range, with the cut-off being somewhere in 
a 10 to 20% or 100-200ms penalty (whichever is higher).

This means that the default retry/fallback to ipv4 time-outs of most 
browsers are completely inadequate, and any user that believes to have
working ipv6 connectivity, but doesn't actually have it, or who's ipv6 
connectivity is worse then ipv4 connectivity will be "broken" by AAAA's 
being added for content.

As to what can break, here is what I've seen or heard about:

1) home gateways announcing RA's into the LAN w/o working external ipv6 
	connectivity of any type
2) home gateways that when configured for IPv4 VPN, will try to send IPv6 
	packets over it
3) Personal firewall apps that will block tcp/udp ipv6
4) IPv4-only vpn clients that turn-off split-tunneling, thereby breaking 
	ipv6
5) Users who have configured proxies in their browser which are not 
	ipv6-capable (either manually or via PAC for vpn/whatnot reasons)

The above breaks ipv6 completely, and causes users into a 21 - 186 sec 
time-outs... I've also seen 6to4/Teredo injecting really bad latency into 
the path that is *way* outside of what we'd concider to be acceptable, and 
would concider that to be "broken" as well..

Does this help?

-igor

---

From: Mark Andrews <[email-masked]>
Date: Fri, 02 Apr 2010 15:46:29 +1100

In message <[email-masked]>, Igor G
> would concider that to be "broken" as well..

How many of those clients are actually using ISP's nameservers when
the breakage occurs?

How many would be better off taking the breakage hit so that they
know something is broken and then fixing it?

> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Igor Gashinsky <[email-masked]>
Date: Fri, 2 Apr 2010 01:54:04 -0400 (EDT)

On Fri, 2 Apr 2010, Mark Andrews wrote:

:: How many of those clients are actually using ISP's nameservers when
:: the breakage occurs?

I'll be able to answer that "somewhat" when we have our data collected. 
The "somewhat" is because I don't know of a way to identify if the user's 
request goes through a midleware box to get to the ISP resolver or not, 
but, I should at least be able to tell if the request went to the ISP 
resolver vs over VPN somewhere off-net.. 

The VPN case is pretty easy to work around -- those will mostly be 
corporate resolvers, and those will simply not be whitelisted... the 
middleware boxes however, are a whole other problem, and, honestly, I 
don't know of a way to detect them.. Do you have any ideas?

:: How many would be better off taking the breakage hit so that they
:: know something is broken and then fixing it?

Pretty much none of them. Yes, I'm serious. 

Why is it that some people think it's acceptable to break perfectly 
working ipv4 users just because you are going to roll out ipv6 services? 

I, for one, get pretty damn pissed when my vendors roll out new features 
(most of which I could care less about) while breaking existing things 
that I use -- I tend to not deploy those things into production. So, why 
is it that we think that our users are any different? Remember, to them 
IPv6 is meaningless, they just want "the web" to work. 

Given today's stats (with 0.3% ipv6 adoption, and 0.074% ipv6 breakage) 
I can see the conversation with businesspeople going something like 
this:

"Let me get this straight.. So, if we start to hand out AAAA's, for every 
4-5 users who will then get to us over IPv6 (and could reach us just fine 
over ipv4 today), you are going to break 1 user that works just fine 
today?!?! Are you out of your mind?!?!"

The funny thing is, I'm what some might term an "ipv6 chearleader", and I 
honestly can't disagree with the above reaction.

And people wonder why more content isn't serving AAAA's by default... 

So, we are absolutely going to need to fix the underlying problems to 
make sure that things are not going to get worse and to make sure that 
they can get better over time (some of that is under way now). But, given 
that those causes of breakage are already deployed today, and, even if 
everything was magicaly fixed right now, it is going to be *years* 
before those fixes are deployed, we are still going to need to find some 
way of mitigating their damage now.

This is why we are going to have whitelists, and, on top of whitelists, we 
are going to need other measures to drop those numbers significantly (like 
filter-aaaa, or anything better that people come up with) in order to get 
more ipv6 adoption (by several orders of magnitude).. 

So, other then saying "fixing broken OS/apps does not belong in DNS", what 
do you think we can do? Where can we (operators) try to fix it, because 
counting on users to do it simply isn't going to work in time..

Thanks,
-igor

---

From: Rémi Després <[email-masked]>
Date: Fri, 2 Apr 2010 11:54:34 +0200

Le 2 avr. 2010 à 07:54, Igor Gashinsky a écrit :

> that I use -- I tend to not deploy those things into production.

That's precisely the reason why it *must not* become a rule that DNS servers would everywhere refuse to respond AAAAs when queried in IPv4.
This is in use today and works well.

> So, why 

Fortunately, there is already at least one large ISP whose users who have IPv6 enabled ignore in general whether they work in IPv4 or IPv6, and don't suffer from IPv6 being enabled.

There should therefore be a solution to your problem.

> they can get better over time (some of that is under way now).

The first thing to do is to understand why some operators have problems with IPv6, or anticipate to have such problems, while some others haven't.

> more ipv6 adoption (by several orders of magnitude).. 

A suggestion could be that ISPs start with sending AAAAs *only* to sites to which they provide native IPv6.
These sites shouldn't have problems, as the experience acquired by Free has shown. 

If these sites can be recognized with some specific IPv4 prefixes, distinguishing them would be much easier than with a  white list containing individual addresses.
Note that Free didn't need to recognize these sites, because in its case *all* customers have Free's CPEs, and these CPEs offer native IPv6 prefixes if IPv6 is activated. (In Free's case, these prefixes are statelessly derived from IPv4 addresses with 6rd, which makes it particularly easy to deploy). 

Thought?

Regards,
RD