Subject: Re: [DNSOP] Ugly DNS ack

From: Tore Anderson <[email-masked]>
Date: Thu, 01 Apr 2010 15:42:32 +0200

Hi Rémi,

* Rémi Després

> => If there are indeed OS bugs that break connectivity, they should 
> justify quick patches like those that concern security.

I think this is wishful thinking, unfortunately.  For example, the 30th
of October last year I reported an IPv6-related bug in the Opera web
browser.  After a considerable amount of nagging and pestering, even on
their public user forums, a fixed version was finally released on the
2nd of March this year.  That was a new major version.  IPv6-related
issues did not appear to be a priority for them, and it certainly seemed
out of the question to back-port the fix to the older maintenance-only
branches.  I don't expect other vendors to behave much differently.

And even if vendors would fix bugs rapidly in new releases, there's
still the issue of making end users upgrade to them.  Now a month has
passed since the Opera bug was fixed, but it continues to be a major
cause of client loss.

> "Gashinsky adds that Yahoo is conducting its own analysis of broken 
> IPv6 connectivity, which it will share with the Internet engineering 
> community in June."
> => As a minimum, what the problem really is should be documented
> before proposing to adopt any solution to solve it, in particular if
> it is "ugly".

It would certainly be useful to see the conclusions from Yahoo's
measurements - if we know what actually causes the client loss, we might
try to fix the actual problems instead of working around it with DNS magic.

To that end, I've been running my own measurement for quite some time
now.  It's very regional in scope (Norwegian end users mostly), so it is
very likely that Yahoo, with their global audience, are observing
problems that I do not, but I'll try to make a summary of my conclusions
from March:

- Total client loss to the dualstacked host is at 0.074%.
- (At least) 95% of the client loss is due to clients choosing to use
  inherently unreliable transitional IPv6 (6to4/Teredo) instead of IPv4.
- I've identifed three groups of clients that behave in this way:
  1) Users of Opera <10.50, which did not use the RFC 3484-compliant
     getaddrinfo() call, at least on Windows.  It would prefer any form
     of IPv6 connectivity above IPv4.  This is especially a problem on
     Windows Vista and 7, where 6to4 and/or Teredo are enabled by
     default.  This accounts for about half of the 95%.
  2) Dualstacked Mac OS X users with RFC 1918 IPv4 addresses (using
     NAT) and transitional IPv6 addresses.  In this case, getaddrinfo()
     will sort the IPv6 address above the IPv4 address, causing the
     transitional connectivity to be used.  This accounts for the other
     half of the 95%.
  3) Dualstacked Linux users with RFC 1918 IPv4 addresses and
     transitional IPv6 addresses, as GNU libc's getaddrinfo()
     implementation behaves exactly like the one in Mac OS X.  However,
     the overall client loss caused by this is miniscule compared to
     #1 and #2 (I have problems measuring any at all).
- When disregarding all hits from users in problem groups #1 and #2,
  the total client loss is at 0.003% - this is, in my opinion, low
  enough to accept.  (Of course, other content providers might feel
  differently.)

I've informed the vendors in question about the problems (and I
understand others have too, before me).  While the Opera issue is
already fixed, the other two reports are here:

http://lists.apple.com/archives/Ipv6-dev/2010/Mar/msg00003.html
http://sourceware.org/bugzilla/show_bug.cgi?id=11438

In the first case, there's no responses from anyone posting from
@apple.com, in the second case, there's no response at all.  If the
vendors are considering IPv6 issues a priority, they're not telling.

You can read my entire March report here:

http://lists.cluenet.de/pipermail/ipv6-ops/2010-April/003272.html

> "Only way of knowing the user has working IPv6 connectivity, is if 
> the AAAA query came over IPv6!"
> => This DOESN'T WORK :

I agree, not in all cases at least.  Your example of IPv6-capable
clients using IPv4 for its DNS queries is one valid case, another would
be when the clients' end stations are not querying the ISPs recursive
resolvers directly, but instead a (caching) DNS forwarder embedded in
their CPE/SOHO device.  This kind of setup is very common here in
Norway.  There's no guarantee that the DNS forwarder will use the same
IP version as the client did when forwarding the query to the ISP's
recursive resolver - which could either cause a broken end station to
see the AAAA records, or vice verca, that a IPv6 capable end station
will only be presented with A records.

The best way to know if the user has working IPv6 connectivity is to
actually test it as he's visiting a IPv4-only web site.  A few days ago
I made a attempt of making such a test, inspired by the so-called
«Norwegian IE6 Spring Cleaning», you can try it here:

http://ajaxtest.fud.no/test.php

It's only partially working, but then again I have no real skills when
it comes to web developement - I'm sure someone would be able to
implement the idea it in a much better way.  Even though certainly not
all users would heed the warnings and implement the suggested fixes,
some would, and I think it would increase visibility of the problem
significantly enough so that the end users (or his ISPs tech support)
will know what to do the moment the site is dual-stacked and the user
starts having connectivity problems.

Imagine if several major content providers (Yahoo, Google, MS, ...)
would agree on a common date for deploying IPv6 on their services,
possibly after a common period of testing/warning users.  In that case,
the business problem with being an early IPv6 adopter would be somewhat
diminished - as a user experiencing the apparent site down problem would
not be able to jump ship to the competitors, either.  And it event would
likely be well publicised, too, so the end user (or his ISPs tech
support) would know what to do.  But I guess having competitors actually
cooperate on something like that is an utopian idea.

> If and when the mentioned OS problems are documented, it will be 
> possible to fix them with patches in OSes, where they belong.

In that regard I hope you have found my measurements useful.  To the
Yahoo people:  The sooner we know what the problems are, the sooner we
can start to fix them.  Could you provide us with some preliminary
results from your measurements before June?

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

---

From: Rémi Després <[email-masked]>
Date: Thu, 1 Apr 2010 16:42:41 +0200

Tore, 

Many thanks for the detailed information you disclose.
Further comments below.

>  enough to accept.  (Of course, other content providers might feel

These 95%+ problems, being related to 6to4 and Teredo, don't concern ISPs that offer native IPv6 addresses (independently from whether they offer them with 6rd, like Free did in 2007, or otherwise).
Note that this is consistent with Free's users not encountering such problems.

Would you have some information about the 5%- remaining losses?

> I agree, not in all cases at least.

A "way of knowing" that doesn't work all the time is not a real "way of knowing", right?

> In that regard I hope you have found my measurements useful.

I did, and thanks again.

>  To the

+1 

Regards,
RD

---

From: Tore Anderson <[email-masked]>
Date: Thu, 01 Apr 2010 17:31:03 +0200

Hi again,

* Rémi Després

> These 95%+ problems, being related to 6to4 and Teredo, don't concern
> ISPs that offer native IPv6 addresses (independently from whether
> they offer them with 6rd, like Free did in 2007, or otherwise). Note
> that this is consistent with Free's users not encountering such
> problems.

Indeed, providing end users with native IPv6 will make the problematic
transitional techniques obsolete.  However, this is not something that
content providers such as Yahoo or myself have much influence over.  We
can (and obviously want to!) dual-stack our content, but as the
end-users can still reach it over IPv4 just as well, even if we did is
not a big incentive for end-users ISPs to get their IPv6 deployment
going.  I'm hoping it's a small one, though.

I applaud Free and yourself, if all ISPs were as progressive as you've
been, we'd have very few problems.

> Would you have some information about the 5%- remaining losses?

I haven't been able pinpoint any common denominator(s).  I have some
ideas, but it's mostly speculation.  In no particular order:

1) Higher latency with IPv6, less developed peerings, etc.  If a client
   navigates away from the test page exactly when the IPv4 PNG has
   loaded, but while the IPv6 request is for instance 50ms away from
   completion, he will be (falsely) counted as a lost client.
2) Linux clients with 6to4/RFC 1918 as described in my previous message.
3) Users with CPEs/SOHO routers that do "evil things" to DNS responses
   containing AAAA records.
4) Tunnels (HE, SixXS, etc.) that the user configured and forgot about,
   and that broke at some point for some reason (like his IPv4 address
   changed).
5) Software that (like Opera did earlier) does not behave according to
   RFC 3484 and prefers transitional/any IPv6 connectivity over IPv4.

In any case it's hard to determine it accurately.  Also I think I'm
approaching the statistical margin of error (on some days the client
loss measurement turns out negative).

Oh, and by the way - #3 in the list over could conceivably be a big
problem for Yahoo or other global players, since there could be huge
geographical differences.  For instance, if an ISP in some other country
far away from here distributes have distributed a CPE to all its
subscribers that breaks AAAA responses, it would hardly be noticable for
me.  But again, that is pure speculation.

I'll be sure to let you know if I do discover more problems, though.

> A "way of knowing" that doesn't work all the time is not a real "way

I'd call it a heuristic estimation or something like that.

Best regards,

---

From: Rémi Després <[email-masked]>
Date: Thu, 1 Apr 2010 17:57:29 +0200

Le 1 avr. 2010 à 17:31, Tore Anderson a écrit :

> content providers such as Yahoo or myself have much influence over.

At least, they should:
- get native IPv6 addresses
-  make sure they advertise both a native-IPv6 and 6to4 addresses.
- transmit IPv6 packets not exceeding 1280 octets.

Note also that the proposed hack, which is in ISP DNS servers, in not more in Yahoo's hands.

> loss measurement turns out negative).

Thanks, I will think about these.

> I'll be sure to let you know if I do discover more problems, though.

Thanks.

> I'd call it a heuristic estimation or something like that.

Right, and to increase operational costs, nothing is better than heuristics, especially if they are supposed to solve ill identified problems, and are known to create new ones for sure!

Regards,
RD

---

From: Mark Andrews <[email-masked]>
Date: Fri, 02 Apr 2010 09:38:07 +1100

In message <[email-masked]>, =?iso-8859-1?Q?R=E9m
i_Despr=E9s?= writes:
> 
> Le 1 avr. 2010 =E0 17:31, Tore Anderson a =E9crit :
> 
> > Hi again,
> > =
> 
> > * R=E9mi Despr=E9s
> > =
> 
> > =

They shouldn't need a 6to4 address.  They should however configure
the exit routers to gateway the 6to4 destined traffic onto IPv4.

ISP's, in general, should be providing 6to4 gateways even if they
are not offering IPv6 native to their customers.  While CPE equipment
that supports IPv6 is still hard to get, there really is no excuse
for ISP's to not be doing this any longer.  You only need to advertise
the route internally.  This distributes the load and gets optimal
routing for their customers that are using 6to4.

> Note also that the proposed hack, which is in ISP DNS servers, in not more =
> in Yahoo's hands.
> 
> > =
> > =
> > =
> > =
> > =
> Right, and to increase operational costs, nothing is better than heuristics=
> , especially if they are supposed to solve ill identified problems, and are=
>  known to create new ones for sure!
> 
> Regards,
> RD
> 
> 
> 
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Igor Gashinsky <[email-masked]>
Date: Fri, 2 Apr 2010 01:03:44 -0400 (EDT)

Tore,

	First of all, I'd like to say thank you for this data, it is 
*extremely* useful! So, we now that we have 2 different tests that show 
the same ballpark of breakage. Some questions in-line...

On Thu, 1 Apr 2010, Tore Anderson wrote:

:: To that end, I've been running my own measurement for quite some time
:: now.  It's very regional in scope (Norwegian end users mostly), so it is
:: very likely that Yahoo, with their global audience, are observing
:: problems that I do not, but I'll try to make a summary of my conclusions
:: from March:
:: 
:: - Total client loss to the dualstacked host is at 0.074%.
:: - (At least) 95% of the client loss is due to clients choosing to use
::   inherently unreliable transitional IPv6 (6to4/Teredo) instead of IPv4.
:: - I've identifed three groups of clients that behave in this way:
::   1) Users of Opera <10.50, which did not use the RFC 3484-compliant
::      getaddrinfo() call, at least on Windows.  It would prefer any form
::      of IPv6 connectivity above IPv4.  This is especially a problem on
::      Windows Vista and 7, where 6to4 and/or Teredo are enabled by
::      default.  This accounts for about half of the 95%.
::   2) Dualstacked Mac OS X users with RFC 1918 IPv4 addresses (using
::      NAT) and transitional IPv6 addresses.  In this case, getaddrinfo()
::      will sort the IPv6 address above the IPv4 address, causing the
::      transitional connectivity to be used.  This accounts for the other
::      half of the 95%.
::   3) Dualstacked Linux users with RFC 1918 IPv4 addresses and
::      transitional IPv6 addresses, as GNU libc's getaddrinfo()
::      implementation behaves exactly like the one in Mac OS X.  However,
::      the overall client loss caused by this is miniscule compared to
::      #1 and #2 (I have problems measuring any at all).

One question on this awesome data -- if the browser gets AAAA, hangs on 
connecting to it, then goes into a time-out retry state, gets the A, and 
hits the site over ipv4, say, 75 seconds later, does your method still 
concider that connect successfull, or have you controlled for that via 
time-outs in javascript or something? If you concider that to be 
successful, do you have any way of re-doing the calculation to say "if it 
took you longer then 10 seconds to connect, you are also broken"?

:: - When disregarding all hits from users in problem groups #1 and #2,
::   the total client loss is at 0.003% - this is, in my opinion, low
::   enough to accept.  (Of course, other content providers might feel
::   differently.)

Honestly, I'd prefer the loss to be 0.001% to be comfortable with it, but 
I have a feeling that you are right, and 0.003% will have to be 
sufficient, provided that those numbers include the unacceptable latency 
people. 

:: In that regard I hope you have found my measurements useful.  To the
:: Yahoo people:  The sooner we know what the problems are, the sooner we
:: can start to fix them.  Could you provide us with some preliminary
:: results from your measurements before June?

When we have some preliminary data to share, we will absolutely do so.. 

Thank you again for your data, extremely useful!

-igor

---

From: Tore Anderson <[email-masked]>
Date: Fri, 02 Apr 2010 13:43:34 +0200

Hi Igor,

* Igor Gashinsky

> the same ballpark of breakage. Some questions in-line...

I'm glad you find it interesting!

> took you longer then 10 seconds to connect, you are also broken"?

There's no timeout with the current numbers, no.  The methology is as
follows (I've anonymised the hostnames to prevent spiders and such from
skewing my results):

- There's a (hidden) IFRAME with src=http://exp.foo.no/linkgen.php
  in the HTML of the web site the end user is visiting.
- The linkgen.php script spits out two IMG links:
  a) http://ds.foo.no/1x1,png?uuid=<random>:<unixtime>&src=<v4 src addr>
  b) http://v4.foo.no/1x1.png?uuid=<random>:<unixtime>&src=<v4 src addr>
  The ordering of the links in the generated HTML is random.
- All of the hostnames mentioned point to different IPv4 addresses, and
  the "ds" one to a IPv6 address also.  However they're all found on the
  same test server, served by the same Apache process.  (There's only
  one copy of 1x1.png on the sever.)
- The DNS TTL for all hostnames are 5 seconds.

So basically what I do is to parse the logs, and count the following
(numbers in parenthesis are from yesterday, April 1st):

* N  - number of hits to linkgen.php                       1986244
* Ns - number of hits to 1x1.png via the v4 virtual host   1971340
* Nd - number of hits to 1x1.png via the ds virtual host   1970605
= Client loss: 100 / N * (Ns - Nd)                         0.037%

There's no timeout of any sort, the only hits I discard are duplicates,
that is any 1x1.png request that comes in with an UUID I've already seen
on that virtual host.

But it would absolutesly be interesting to see how a timeout would
affect the numbers, thanks for the suggestion!  Since the time when the
linkgen script ran in is embedded in the 1x1.png link, that was trivial
to implement, and I've just run the numbers for March with a timeout of
ten seconds, like you suggested.  Results for VG:

- Overall client loss:                                  0.122%
- With Opera <10.50 on Windows and Mac OS X excluded:   0.015%
- IOW, Opera/Mac OS X accounts for 88.056% of the client loss.

So the root cause is for client loss is still overwhelmingly unnecessary
use of transitional IPv6 connectivity.  I will attempt to look more into
the set of hits that arrive more than 10 seconds late on the dualstack
host to see if I can uncover any other sources of client loss there,
other than the ones I already know about.

By the way, it would be fantastic if you/Yahoo would (in addition to
pursuing the proposed DNS solution) help make an effort to help get rid
off the known problems, by for example talking to Apple about their
getaddrinfo() implementation, and warning users with old versions of
Opera to upgrade - at least if you measure bad IPv6 connectivity from them.

Best regards,

---

From: Rémi Després <[email-masked]>
Date: Fri, 2 Apr 2010 15:07:00 +0200

> the exit routers to gateway the 6to4 destined traffic onto IPv4.

What you propose:
- does guarantee a good path from Yahoo servers to 6to4 clients, you are right
- but doesn't guarantee a good path from 6to4 clients to Yahoo servers => it is not sufficient.

Without Yahoo 6to4 routers operated by Yahoo, paths from 6to4 clients depend on a 6to4 Relay Router being reachable from them, which is not guaranteed. 

On the contrary, by operating its own 6to4 Routers (and embedding their IPv4 address in Yahoo-server 6to4 addresses), IPv6 packets always go directly from 6to4-client sites to Yahoo-server sites (encapsulated in IPv4 from client 6to4 router to Yahoo 6to4 router), i.e. without depending on any Relay Router.

> are not offering IPv6 native to their customers.

They MAY not do it, though (taking obviously "gateways" as meaning, in RFC 3056 and RFC 3964 terminology, "Relay Routers")

Reality is that 6to4 guarantees connectivity *between two 6to4 sites* (in RFC 3056, the Simple scenario of  section 5.1), but not between 6to4 sites and native-IPv6 sites (the Mixed scenario of section 5.2).

Note that RFC 3964 says about "Security Considerations for 6to4" (emphasis added):
"There are mainly four classes of potential problem sources:
1. 6to4 routers not being able to identify whether RELAYS are
legitimate
...
The first is the TOUGHEST PROBLEM, still UNDER RESEARCH."

>  While CPE equipment
> for ISP's to not be doing this any longer.

Reasons ISPs may have for not providing 6to4 Relay Routers include:
- Relay Routers encourage use of 6to4 in scenarios others than those where connectivity is guaranteed, which is bad for IPv6 in general
- An ISP 6to4 Relay Router can be used for traffic that concerns none of its customers.  
- The wish to not depend on a subject under research (the RFC 3964 quotation above). 
They are all IMHO legitimate.
 
I hope this helps to understand that there are precautions that CAN be taken, and that, this being done, the FUD about IPv6 in general may dissipate faster.

Regards,
RD

---

From: Jason Livingood <[email-masked]>
Date: Mon, 19 Apr 2010 12:28:34 -0400

This thread seems to have died out.  But I will note that it appears to be
on the agenda at the upcoming ISOC IPv6 workshop (from 16:30 - 18:00 - see
https://www.isoc.org/isoc/conferences/registration/index.php?id=7ce20e4c88b7
e328).  It is also addressed in a short whitepaper some colleagues and I put
together in advance of that meeting (available at
http://www.comcast6.net/IPv6_DNS_Whitelisting_Concerns_20100416.pdf)

Jason

---

From: Edward Lewis <[email-masked]>
Date: Mon, 19 Apr 2010 14:02:55 -0400

>http://www.comcast6.net/IPv6_DNS_Whitelisting_Concerns_20100416.pdf)

After reading the white paper I can think of a lot of things to say, 
but not much related to DNS operations.  What comes to mind is 
something that could make use of this proposal:
    http://tools.ietf.org/html/draft-vandergaast-edns-client-ip-00

That is not a savior here, I don't propose it to be.  What I wanted 
to point out is that while that proposal is talking about conveying 
addressing information in support of tailoring answers, the white 
paper is talking about the design of the tailoring.

In some cases, responses can be tailored according to a simple design 
- "all addresses in North America get these servers, in Europe those 
servers, in Asia those servers, and so on.  But the problem with IPv6 
connectivity testing is that the granularity (needed for tailoring) 
is not very consistent.

It depends on whether the matter is that an entire /LIR's with of 
addresses can't be reached from the provider or just those clients 
using 6to4 or just a particular client behind a cable modem still 
running from June 1942.

I used to think the problem was "how to get around discontinuous IPv6 
routing when I have a local "global" address."  Now it seems like 
it's a problem of scattered and varied (to use the same ill-defined 
word) "brokenness" sprinkled all over the place.  What's this got to 
do with DNS operations?  I can't see...how the DNS can really be of 
help in this matter.
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis
NeuStar                    You can leave a voice message at +1-571-434-5468

Wouldn't it be nice if all of the definitions of equivalence were the same?

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 20 Apr 2010 21:51:44 +0900

Jason Livingood wrote:

> This thread seems to have died out.

As I already summarized, the problem is that:

> The problem is lack of support of TCP/IP for hosts with multiple
> addresses.
> Its scope is not limited to IPv6 nor HTTP but TCP/IP in general.
> A host with two IPv4 addresses also suffers, when one of its
> addresses is sometimes unreachable from its peer.

and the short and long term approaches to the solution are: that

> The solution can be properly constructed not at the connectionless
> IP layer but at the lowest layer above IP to support connection,
> which means TCP needs a major revision. Or, you can say SCTP,
> though it may also need a major revision to be really usable
> against the problem.

and that

> When almost all hosts have single IPv4 addresses (and single IPv6
> addresses), the most practical tentative solution is not to deploy
> IPv6.
> Long term solutions need drastic changes on treatment of multihomed
> ISPs and address allocation policies, which also excludes, with
> RFC2374 obsoleted, IPv6.

That's all.

							Masataka Ohta

---

From: Brett Watson <[email-masked]>
Date: Wed, 28 Apr 2010 14:47:45 -0700

On Apr 19, 2010, at 11:02 AM, Edward Lewis wrote:

>  But the problem with IPv6 connectivity testing is that the granularity (needed for tailoring) is not very consistent.

While I somewhat entertained this whole idea at first, that's the conclusion I came to. I think there are too many corner cases where determining whether the client system making the original query has v6 connectivity or not is just not possible. That's not the only issue with this idea but I think a significant one.

-b

---

From: Mark Andrews <[email-masked]>
Date: Thu, 29 Apr 2010 10:13:18 +1000

In message <[email-masked]>, Brett Watson
 writes:
> >  But the problem with IPv6 connectivity testing is that the granularity (ne
> eded for tailoring) is not very consistent.
> 
> While I somewhat entertained this whole idea at first, that's the conclusion 
> I came to. I think there are too many corner cases where determining whether 
> the client system making the original query has v6 connectivity or not is jus
> t not possible. That's not the only issue with this idea but I think a signif
> icant one.

And the solution is "just deploy IPv6 where ever you can".  Having
IPv6 deployed stops these boxes auto configuring IPv6 using methods
that are blocked.

Turn it on for your nameservers, authoritative and recursive.  Turn
it on for your clients nets.  Selectively turn it on for other
services.  MTA are good candidates as they are normally not real
time.

Check that your load balancer actually gives good answers for AAAA
queries as too many of them are mis-configured, correct rcode and
correct SOA.  Report ones that return bad answers.  I've actually
had two "Thank You"s in the last month doing that to ones that have
shown up on bind-users just by Bcc the bind-users reply.

Mark