Subject: Re: [DNSOP] FYI: DNSOPS presentation

From: Mohacsi Janos <[email-masked]>
Date: Tue, 30 Mar 2010 17:56:17 +0200 (CEST)

Dear All,

Sorry for crossposting.

This proposal is the opposite with the principle how the DNS is developed 
a while ago. The DNS is a highly distributed, hierarchical, autonomous, 
reliable database with very useful extensions. This modification is 
proposing lying about the existence of the record

The modification is proposed to hide the database record that is used for 
communication. I am not favor such a modification since:

1. I think we need evidence, that majority of the AAAA queries are going 
via IPv6 (if the client has working IPv6 and the DNS zones has the 
necessary AAAA for the zones).

2. Plenty of users still use WinXP, where DNS query via IPv6 is not 
possible. Probably other systems with similar limitations also exists. 
This can very negatively impacting the IPv6 usage.

3. There is only few broadband CPE devices deployed at homes are capable 
of providing DNS information via DHCPv6. Only very recently they started 
to appear. Thus the addresses of the DNS servers are propagated via DHCP. 
Therefore these broadband users, however they are IPv6 enabled, will 
essentially use IPv4 address to reach DNS servers. Will you enable 
returning AAAA records to these caching servers?

If yes. Will you get any information about the broadband ipv6 
connectivity?

If no. You are excluding caching server users from ipv6?

4. How your scheme will work the caching DNS server - commonly deployed by 
ISPs to serve their constituency.

5.  What about the caching in the DNS server? Will they be flushed when 
IPv6 or IPv4 connectivity changed? IPv6 connectivity changes of who?

6. Your modification is treating some portion of the users as a second 
class netizen. They cannot really query AAAA records.... IPv6 is so 
secret, that we have to hide from them?

Sorry for being sometimes sarcastics

Janos Mohacsi
Head of HBONE+ project
Network Engineer, Deputy Director of Network Planning and Projects
NIIF/HUNGARNET, HUNGARY
Key 70EF9882: DEC2 C685 1ED4 C95A 145F  4300 6F64 7B00 70EF 9882

On Mon, 29 Mar 2010, Ed Jankiewicz wrote:

> Probably no one on either of the IPv6 lists attended the DNSOPS WG meeting in 
> Anaheim, since it was at the same time as 6man.
>
> Presentation by Yahoo! of a proposal to "do an ugly hack on DNS" to work 
> around an issue with "broken OSes" that send out AAAA requests when they have 
> no intention/ability to actually use an IPv6 address.  Google experience is 
> that a small percentage of their users would lose connectivity because of 
> this, if google.com serves both IPv6 and IPv4 addresses.  I can't recall if 
> this particular issue has been discussed here, but either way anyone with an 
> interest should probably pop comments over to the DSNOPS WG list.
>
> http://www.ietf.org/proceedings/10mar/slides/dnsop-7.pdf
>
> Also FYI - this has gotten press coverage, not necessarily accurately 
> characterizing the problem or proposed solution
>
> http://www.networkworld.com/podcasts/360/2010/032910-nw360-daily.html
> http://www.networkworld.com/news/2010/032610-dns-ipv6-whitelist.html
>
> -- 
> Ed Jankiewicz - SRI International
> Fort Monmouth Branch Office - IPv6 Research Supporting DISA Standards 
> Engineering Branch
> 732-389-1003 or  [email-masked] 
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: Nicholas Weaver <[email-masked]>
Date: Tue, 30 Mar 2010 09:04:39 -0700

On Mar 30, 2010, at 8:56 AM, Mohacsi Janos wrote:

> Dear All,
> 
> Sorry for crossposting.
> 
> 
> This proposal is the opposite with the principle how the DNS is developed a while ago. The DNS is a highly distributed, hierarchical, autonomous, reliable database with very useful extensions. This modification is proposing lying about the existence of the record
> 
> The modification is proposed to hide the database record that is used for communication. I am not favor such a modification since:
> 
> 1. I think we need evidence, that majority of the AAAA queries are going via IPv6 (if the client has working IPv6 and the DNS zones has the necessary AAAA for the zones).

This is clearly not the case.  

Linux clients in particular seem to always do AAAA as well as A queries: Netalyzr is an IPv4 service, (we detect IPv6 usage, its only a couple of percent and Netalyzr has a very geek-biased dataset), but there are a LOT of clients which are doing v6 queries on DNS as well as V4.

EG, a colleague had very problematic network connections from his parent's home back to ICSI, because the stupid NAT's built in DNS proxy (until he reflashed an update) was blocking AAAA queries.  

His linux host would do an A and an AAAA query and, until the AAAA query timed out, delay creating connections eg, through SSH, web browsing, etc.  An amazingly painful experience for him until he diagnosed it.

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 30 Mar 2010 12:15:27 -0400

On Tue, Mar 30, 2010 at 09:04:39AM -0700, Nicholas Weaver wrote:
> His linux host would do an A and an AAAA query and, until the AAAA
> query timed out, delay creating connections eg, through SSH, web
> browsing, etc.  An amazingly painful experience for him until he
> diagnosed it.

But the answer to this breakage that is being proposed is to turn on,
apparently at ISPs' recursive resolvers, a different kind of breakage,
and one without a useful way to detect when it's no longer needed.  In
other words, we'll introduce some sort of bizarre breakage to
dual-stack systems (ones that might work correctly, even) without any
plan for how we'll even know when to turn this breakage off.  And this
while we desperately need to wean people off IPv4 and onto IPv6.

Rather than having the DNS magically lie to people, why not use the
DNS detection mechanism as an indicator that a customer has a broken
v6 implementation.  Then you can turn off _that customer's_ IPv6
connectivity, contact them, and tell them what their problem is.  This
has three benefits:

    1.  The customer doesn't break in surprising ways.

    2.  Other customers don't break for no reason.

    3.  The customer learns s/he has an issue, and can take steps to
    correct it before IPv4 is too expensive to use any more.

I am not among those who think that the number of clients involved
with this is "insignificant".  I know that something people sometimes
hear, but the abolute number of people involved does make this a real
problem.  I just don't think that the right answer is to break
perfectly well-functioning systems for everyone else in order to work
around clients that are implemented wrong.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Nicholas Weaver <[email-masked]>
Date: Tue, 30 Mar 2010 09:27:09 -0700

> around clients that are implemented wrong.

Agreed on both accounts.  

All I was really pointing out is that most V6 queries are done over V4, and you can't tell who does or does not use V6 for the DNS server.

---

From: Edward Lewis <[email-masked]>
Date: Tue, 30 Mar 2010 13:46:07 -0400

(Maybe this is better on v6ops, but I'm not on that list and I don't 
really like to cross-post.)

>And this while we desperately need to wean people off IPv4 and onto IPv6.

Right there is where I think the problem is - "trying wean people off".

First, the network layer does not offer features that are impressive 
to the user of the application layer.

Second, engineers ought to be neutral to the requirements.  It isn't 
up to us to dictate what technology is to be used.

Maybe I'm missing something - if the round trip time via IPv4 and 
IPv6 is the same for me from my client to some server and there's no 
cost difference [I pay a monthly flat fee], what's the benefit of 
using IPv6?

>Rather than having the DNS magically lie to people

The DNS is a poor heuristic function when it comes to determining how 
to connect from A to B.  There have been discussions over how to sort 
IPv4 addresses to

1) Show the least loaded server first (that changes too much)
2) Show the closest connection first (the DNS server is not A; 
closeness != speediness)
3) Tell the user which server has the most appropriate content (eg language)

an so on.  These have some benefit, but none are perfect.  Why should 
IPv6 or IPv4 be any different?

I've fallen victim to applications that felt "if I have IPv6 locally 
and there's a AAAA to go to, I won't try IPv4" regardless that the 
IPv6 was link-local.  That's just frustrating.  On the other hand, 
when I see "you go here by IPv6" on a website, I think "gee whiz."

When a user performs an action that requires mapping a domain name 
into an address, all addresses ought to be picked up, without regard 
to a versionX bias.  The application should then attempt to connect 
simultaneously on all address via the appropriate address family. 
The goal is to quick connect, not show a bias.

If IPv4 and IPv6 are available, what's the difference?  What does it 
matter if IPv6 is used or IPv4 is used?  If only one or the other is 
working, then the user gets through on that.  So what if IPv6 is 
broken and IPv4 works.  It's likely that even if the user knew this 
they couldn't do much more than "turn off IPv6 on their machine" or 
maybe even contact IT.

Why is there a need to wean people off IPv4?
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis
NeuStar                    You can leave a voice message at +1-571-434-5468

New pithy statement under construction...

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 30 Mar 2010 13:58:06 -0400

> Why is there a need to wean people off IPv4?

Because we're about to run out of v4 addresses, according to the
people in charge of giving them out.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Edward Lewis <[email-masked]>
Date: Tue, 30 Mar 2010 14:15:49 -0400

>people in charge of giving them out.

I've heard that before.  The "run out" does not mean an end to the 
IPv4 network.  There will still be 4 billion IPv4 network addresses 
(yes, a fraction are unusable) in working order plus all the NATted 
pools out there.  The "run out" is a problem for network growth.

"Run out" of the IANA and then the RIR free pools does not change the 
addresses I am handed by my ISP.  And as long as my ISP is alive, 
price-competitive, and handing my IPv4, I'm still going to use IPv4. 
If my ISP instead starts only offering me IPv6, I'll use that, and if 
that means I need to upgrade my cable modem I assume they'll handle 
just like they handled the conversion from analog to digital TV, with 
a box swap or add-on or firmware or such.

Why should I want to use IPv6 if IPv4 is still working for me?  I'm 
not saying "you'll get my IPv4 when it pry it from my cold, dead 
hands" - it's simply a question of "why try to bias my choice towards 
IPv6?"  To keep this from an IPv6 rathole - "and why try to involve 
DNS in this?  Don't we want to keep the DNS policy neutral?"

(One of my ten commandments: "No long terms solutions for short term 
problems.")
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis
NeuStar                    You can leave a voice message at +1-571-434-5468

---

From: Ted Lemon <[email-masked]>
Date: Tue, 30 Mar 2010 11:25:03 -0700

> DNS in this?  Don't we want to keep the DNS policy neutral?"

You want to use IPv6 because:

- it has significant new features that will make things like VoIP work better for you
- if you can't use IPv6, as more and more IPv6 nodes are deployed, your ability to use the network will slowly get less and less good.
- the more NATs you are behind, the fewer connections you will be able to hold open at a time, and this will impact your use of connection-intensive AJAX applications like Google Maps and Facebook.

All this has been covered at length before--dunno why you want to rehash it again.

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 30 Mar 2010 14:25:22 -0400

On Tue, Mar 30, 2010 at 02:15:49PM -0400, Edward Lewis wrote:
>
> I've heard that before.  The "run out" does not mean an end to the IPv4 
> network.  There will still be 4 billion IPv4 network addresses (yes, a 
> fraction are unusable) in working order plus all the NATted pools out 
> there.  The "run out" is a problem for network growth.

Right.  That's all I meant.  I don't think we need to get people who
are using v4 off v4.  But there will be people for whom v6 will surely
become a more economic option (since as v4 addresses become scarcer,
people who have v4 addresses might find it worth moving to v6 because
the resource is more plentiful).

The point I was trying to make was that, since dual-stack is likely to
be around for a while, with growth on the v6 side, it seems like the
wrong time to break a perfectly good feature (DNS answers on v4 for a
possible v6 connection).  There's something else we ought to be able
to do.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Jim Reid <[email-masked]>
Date: Tue, 30 Mar 2010 19:43:22 +0100

On 30 Mar 2010, at 19:25, Andrew Sullivan wrote:

> it seems like the wrong time to break a perfectly good feature (DNS  
> answers on v4 for a possible v6 connection).

+1

It is a very, very bad idea to couple the network protocol to the  
answers given by a DNS server. The end client may well have v6  
connectivity (say) even if it has to use a v4-only resolving server or  
middleware box. Even if there was a magic "I do/don't speak v6" flag  
bit that the client could signal to the authoritative server, this is  
still a bad idea. For example an application may be stuck on a v4 only  
box but is trying to survey how much v6 is in the DNS by asking for  
AAAA records at interesting domain names.

Let's wait now until Yahoo's idea makes its way to the WG.

---

From: Edward Lewis <[email-masked]>
Date: Tue, 30 Mar 2010 15:40:27 -0400

>- it has significant new features that will make things like VoIP work

Fine, that's a reason for v6 to come along.  But why should I prefer 
to run SSH over v6 rather than v4?

>- if you can't use IPv6, as more and more IPv6 nodes are deployed, your
>   ability to use the network will slowly get less and less good.

But why bias v6 over v4 for situations where I can get there over v4?

>- the more NATs you are behind, the fewer connections you will be able to
>   hold open at a time, and this will impact your use of connection-intensive
>   AJAX applications like Google Maps and Facebook.

That's a good reason to avoid a dependency on NAT and encourage 
deployment of IPv6.  But then, in this case again the bias doesn't 
help.

Let's say I can manage just "X" connections via v4 and I have a 
choice of v6 or v4.  When I open up connections to fill in a page I 
might find that v4 connections "beat" v6 until I use up my "X" and 
then from then on, v6 wins out.  The bias doesn't help here because 
if I try to put v4 at a *further* disadvantage, I'm just losing my 
capability of using IPv4.

>All this has been covered at length before--dunno why you want to rehash

Because it recently dawned on me that biasing in favor of v6 is the 
root of the evil, not the run out of v4.

This isn't about what's better, v4 or v6.  It's about the bias.
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis
NeuStar                    You can leave a voice message at +1-571-434-5468

---

From: Edward Lewis <[email-masked]>
Date: Tue, 30 Mar 2010 15:44:43 -0400

At 14:25 -0400 3/30/10, Andrew Sullivan wrote:

>There's something else we ought to be able to do.

And that's what I am calling into question.

It seems to me- adding a bias essentially just puts IPv4 at more of a 
disadvantage without any gain.

If IPv4 can get through faster than IPv6, why not continue to use it? 
When IPv6 is the only way through, use IPv6.  When IPv6 is faster, 
again, use it.

Let the end host decide.

---

From: John Schnizlein <[email-masked]>
Date: Tue, 30 Mar 2010 15:53:05 -0400

Just a point of clarification before the list moderator shuts down  
this off-topic thread..

Ed's unstated assumption is that the condition being considered is  
communication between two hosts that are both dual-stack.  It is not  
that he fails to understand that hosts that are now IPv4-only should  
become dual-stack in order to communicate with the IPv6-only hosts  
that are implied by the design of IPv6 (and the pending exhaustion of  
spare IPv4 addresses).

 From what we know of forwarding capacity in routers currently  
deployed, dual-stack hosts may get better connectivity over IPv4 than  
IPv6, or the other way around, depending on factors not worth adding  
to the off-topic mix.  Ed's "let the host choose" is the right answer  
because in the future IPv6-only world hosts will need that capability,  
and mostly do not have it now.

John

On 2010Mar30, at 3:40 PM, Edward Lewis wrote:

> ...
>> All this has been covered at length before--dunno why you want to  
>> rehash
>> it again.
>

---

From: Edward Lewis <[email-masked]>
Date: Tue, 30 Mar 2010 16:13:13 -0400

At 15:53 -0400 3/30/10, John Schnizlein wrote:
>Just a point of clarification before the list moderator shuts down this
>off-topic thread..

To bring this back to on-topic for DNS operations...

>communication between two hosts that are both dual-stack.

Not necessarily an assumption.  The protagonist (the client) knows 
it's network-layer experience (v4 vs. v6) in getting through to the 
server, more so than the DNS.  The DNS is not the place to inject 
"policy" to influence the protagonist's choice.  Or to feed a bias 
that backfires.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 30 Mar 2010 13:44:58 -0700

> root of the evil, not the run out of v4.

Biasing stupidly is the root of all evil.  Stuart Cheshire explained how to make this work, and it's going to be a standard feature in MacOS 10.7, but there's no reason why we can't fix ssh now.   Your syscall, or library call, should be saying "here's a name, give me a connection to it," not "what's the IP address of this name" and then "now make a connection to that IP address."

Your argument boils down to saying that as apps become IPv6-capable, they should still prefer IPv4 if it's available.   But that's just squandering an opportunity for win.   The right thing to be doing is fixing applications so that they do the *right* thing, not fixing them so that they do something very much like what they were doing before, only with a different bias.

The question of whether to break AAAA records is one small piece of this larger discussion, and if you make a choice here that prevents the right thing being done at the higher layer, that's a *really* bad outcome.

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 30 Mar 2010 16:45:19 -0400

> When IPv6 is the only way through, use IPv6.  When IPv6 is faster,  

I thought I saw you in the DNSOP meeting in Anaheim where this was
outlined?  Anyway, the problem right now is not that the end host
can't decide, but that the end host is deciding _wrong_: it does DNS
over v4, tries to use v6, but is in fact broken on v6 and therefore
won't get the communication it desires.  So what we're trying to cope
with is real breakage in real deployed stuff really on the Internet.
This is not a theoretical exercise in preferring IPv6.  It's an
exercise in trying to do the least-bad thing, given that there is a
tiny minority of hosts (which represents a large number of eyeballs)
who are having trouble today.

Dual-stack and IPv6-only installations are in some cases broken today.
It's unrealistic to say, "Let them feel the pain & they'll upgrade,"
because the people this affects are unlikely to be able to understand
what is happening to them.  As a result, people are going to do
something bad for the DNS (especially over the long term) unless we
find some other thing to suggest to them.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 30 Mar 2010 13:47:17 -0700

> "policy" to influence the protagonist's choice.  Or to feed a bias 

Hm, that's the opposite of what I understood you to be saying.   If you're arguing that transport-based filtering is bad, we're 100% in agreement.

---

From: Edward Lewis <[email-masked]>
Date: Tue, 30 Mar 2010 21:12:49 -0400

At 16:45 -0400 3/30/10, Andrew Sullivan wrote:

>outlined?

Doesn't mean I was paying attention.  Didn't you notice that there 
were three light bulbs out in the ceiling.  (No, just joking.)

>Anyway, the problem right now is not that the end host

Yeah, that's what I thought the problem to be, and yes, the end host 
is doing the wrong thing today.  That doesn't mean DNS should be used 
to correct that.  The end host ought to be fixed - by removing it's 
bias towards v6.

So - "does DNS over v4, tries to use v6" should be "does DNS over v4, 
tries to use v6 and v4 simultaneously" - that's what I'm trying to 
say.

>find some other thing to suggest to them.

It's not a DNS problem, right?  It is bad behavior in hosts that try 
v6 first or only that is the issue.  The solution is to stop the end 
host bias.

---

From: Igor Gashinsky <[email-masked]>
Date: Wed, 31 Mar 2010 03:28:24 -0400 (EDT)

On Tue, 30 Mar 2010, Edward Lewis wrote:

:: >Dual-stack and IPv6-only installations are in some cases broken today.
:: >It's unrealistic to say, "Let them feel the pain & they'll upgrade,"
:: >because the people this affects are unlikely to be able to understand
:: >what is happening to them.  As a result, people are going to do
:: >something bad for the DNS (especially over the long term) unless we
:: >find some other thing to suggest to them.
:: 
:: It's not a DNS problem, right?  It is bad behavior in hosts that try v6 first
:: or only that is the issue.  The solution is to stop the end host bias.

You are absolutely right -- it's not a DNS problem, it *is* a host 
behavior problem. The issue is that it takes *years* to fix a host 
behavior problem, and we need to engineer and deploy a fix much sooner 
then that (hopefully about a year before the v4 exhaustion date). Given 
that, is there something other then DNS that can address it better/faster?

Thanks,
-igor

---

From: Andras Salamon <[email-masked]>
Date: Wed, 31 Mar 2010 10:10:27 +0100

>that, is there something other then DNS that can address it better/faster?

I don't understand why DNS is seemingly being tasked here with mopping
up the messes of other parts of the network.

DNS works because it has a clearly defined purpose.  Adding more and
more random "features" to DNS, because it is a part of the network
which is seemingly not broken, is a great way to ensure DNS fails to
achieve its purpose.  DNS is not the network janitor.

(By the way, I am not claiming DNS is not broken.)

-- Andras Salamon                   [email-masked]

---

From: Jim Reid <[email-masked]>
Date: Wed, 31 Mar 2010 10:21:58 +0100

+1

---

From: Nicholas Weaver <[email-masked]>
Date: Wed, 31 Mar 2010 05:16:39 -0700

> that, is there something other then DNS that can address it better/faster?

Except that, since it IS a host problem, why not focus on improved detection and debugging....

EG, yahoo is one of the big pushes behind it.  THey could easily add auto-debugging to their homepgae would would detect this condition and NOTIFY the users that this will be a problem, same for google.

This to me seems like a "cure" worse than the disease.

---

From: Edward Lewis <[email-masked]>
Date: Wed, 31 Mar 2010 09:42:51 -0400

>that, is there something other then DNS that can address it better/faster?

On topic of DNSOP: Reversing a fix slipped into the DNS for some 
other segment's problem will take years to remove.  We still have 
round-robin in the DNS, for example, added to help load balancing for 
an application (mail?) way way back in time.  Today round-robin is a 
pain for DNSSEC (for example).

This is off-topic for DNSOP: I don't believe it takes "years" to fix 
host behavior problems.  Yes, some hosts will run the same software 
they have today for years to come.  But most won't last that long. 
If you don't fight the problem in the right place, you won't 
eradicate the issue.
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis
NeuStar                    You can leave a voice message at +1-571-434-5468

---

From: Nicholas Weaver <[email-masked]>
Date: Wed, 31 Mar 2010 07:59:39 -0700

On Mar 31, 2010, at 6:42 AM, Edward Lewis wrote:

> On topic of DNSOP: Reversing a fix slipped into the DNS for some other segment's problem will take years to remove.  We still have round-robin in the DNS, for example, added to help load balancing for an application (mail?) way way back in time.  Today round-robin is a pain for DNSSEC (for example).
> 
> This is off-topic for DNSOP: I don't believe it takes "years" to fix host behavior problems.  Yes, some hosts will run the same software they have today for years to come.  But most won't last that long. If you don't fight the problem in the right place, you won't eradicate the issue.

Also, you can make an EASY in-browser Javascript check.

Load 3 images in a hidden DIV.  These images should ideally be set to be non-cached and have a cache-buster in the URL (akin to how Google Analytic's hidden GIF is loaded: it contains a cache-buster in the URL).

One is hosted on an IPv4 only site, one on a IPv4/IPv6 dual stack, and one on a IPv6 only site, and bind onload/onerror to Javascript for reporting.

If the first two load, the host is a successful V4 host.

If only the first loads, the host has the described problem with a link local V6, and needs to be patched: have the Javascript notify the user.

If only the second and third loads, the host is V6 only (HA!).

Put such a Javascript test on the Yahoo or Google homepage, as it is small (you could compress it down to probably a couple hundred bytes), you can make it so its effectively non-blocking on rendering (put it in a infinitesemal iframe), so it doesn't impact page load times.

Voila, you not only FIND the hosts that are the problem, but notify them to FIX the problem, especially since it is Yahoo that is specifically worried about clients with this problem.

---

From: Rémi Després <[email-masked]>
Date: Wed, 31 Mar 2010 17:32:06 +0200

Hi Ed,
Thanks for this info, very useful to those who were at 6man during the dnsop meeting.
Reactions below to some quoted sentences.

1.
"Yahoo's worry is that some operating systems issue quad-A records by default, even if the user has broken IPv6 connectivity and needs single A records to access IPv4-based content."
"Work with OS/app vendors to fix IPv6 issues – Awful long lead times/upgrade cycles"
"This is a really ugly hack, but it may be necessary to get widespread IPv6 adoption,"
=> If there are indeed OS bugs that break connectivity, they should justify quick patches like those that concern security. 

2.
"Gashinsky adds that Yahoo is conducting its own analysis of broken IPv6 connectivity, which it will share with the Internet engineering community in June."
=> As a minimum, what the problem really is should be documented before proposing to adopt any solution to solve it, in particular if it is "ugly".

3.
"Only way of knowing the user has working IPv6 connectivity, is if the AAAA query came over IPv6!"
=> This DOESN'T WORK :
- Today, dual-stack hosts on Free's network query Free's DNS in IPv4 (at the only DNS address they know, received in DHCPv4)
- These hosts, because they have valid IPv6 addresses (i.e. have IPv6 enabled), ask for both As and AAAAs.
- For maps.google.fr, for example, BOTH types of RRs are in the DNS.
- They are included in DNS responses
- Hosts then use IPv6 (preferred in case of choice).

4.
"Return 0 answers for AAAA if, and only if: - Query comes over Ipv4; - “A” record exists for same name; - DNSSEC is not used."
=> This hack would NOT ONLY be "ugly" (as acknowledged by their proponents), BUT ALSO would BREAK some of the IPv6 connectivities that are available today !!!

==> This hack MUST therefore be flatly rejected.

If and when the mentioned OS problems are documented, it will be possible to fix them with patches in OSes, where they belong.

RD

Le 30 mars 2010 à 00:21, Ed Jankiewicz a écrit :

> Probably no one on either of the IPv6 lists attended the DNSOPS WG meeting in Anaheim, since it was at the same time as 6man.
> 
> Presentation by Yahoo! of a proposal to "do an ugly hack on DNS" to work around an issue with "broken OSes" that send out AAAA requests when they have no intention/ability to actually use an IPv6 address.  Google experience is that a small percentage of their users would lose connectivity because of this, if google.com serves both IPv6 and IPv4 addresses.  I can't recall if this particular issue has been discussed here, but either way anyone with an interest should probably pop comments over to the DSNOPS WG list.
> 
> Also FYI - this has gotten press coverage, not necessarily accurately characterizing the problem or proposed solution
> 
> Fort Monmouth Branch Office - IPv6 Research Supporting DISA Standards Engineering Branch

---

From: Ted Lemon <[email-masked]>
Date: Wed, 31 Mar 2010 08:43:51 -0700

> ==> This hack MUST therefore be flatly rejected.

Unfortunately we don't have any control over what Yahoo or Google do to their name servers.   I agree with you completely on what we SHOULD do, but if Igor decides to filter AAAAs on IPv4 queries, we can't stop him.   We can refuse to endorse his solution, but really what's going on here is that Igor (and Google before him) have come to us in good faith to tell us about hacks they've done that they feel are necessary.   We shouldn't discourage them from doing that, even if we do discourage them from doing the hacks.

So to my mind, the question is whether or not we want to (and can!) have some say in what these hacks look like, not whether or not we should forbid them.

---

From: Rémi Després <[email-masked]>
Date: Wed, 31 Mar 2010 18:02:57 +0200

Le 31 mars 2010 à 17:43, Ted Lemon a écrit :

> Unfortunately we don't have any control over what Yahoo or Google do to their name servers.   I agree with you completely on what we SHOULD do, but if Igor decides to filter AAAAs on IPv4 queries, we can't stop him.

Sure, and that's fair.
But it has to remains his problem, not an IETF specification problem.

>   We can refuse to endorse his solution, but really what's going on here is that Igor (and Google before him) have come to us in good faith to tell us about hacks they've done that they feel are necessary.   We shouldn't discourage them from doing that,

Agreed.
My strong reaction is only against this particular hack because:
- it is based on the idea that OSes cannot be patched where they are bugged
- it destroys legitimate connectivity for OSes that aren't bugged.

> even if we do discourage them from doing the hacks.
> 
> So to my mind, the question is whether or not we want to (and can!) have some say in what these hacks look like,

We do want it.
(That's what I did in the technical explanation.)
 
> not whether or not we should forbid them.

"Flatly rejecting" the hack, as I proposed, doesn't mean "forbidding" it.
(Vendor makes its own choices anyway.)
It just means, in my understanding, a clear refusal to endorse it.

Regards,
RD

---

From: John Jason Brzozowski <[email-masked]>
Date: Wed, 31 Mar 2010 13:38:20 -0400

Remi,

I hope you do not mind, I have take the liberty to pull some text from your
original post and will reply to them here:

On 3/31/10 10:20 AM, "Rémi Després" <[email-masked]> wrote:

> 1.
> "Yahoo's worry is that some operating systems issue quad-A records by default,
> even if the user has broken IPv6 connectivity and needs single A records to
> access IPv4-based content."
> "Work with OS/app vendors to fix IPv6 issues  Awful long lead times/upgrade
> cycles"
> => If there are indeed OS bugs that break connectivity, they should justify
> quick patches like those that concern security.
[jjmb] anyone who has had to deal with releasing software can certainly
attest to the fact that these sort of fixes/enhancements may not occur in a
timely manner.  There are things we need to be doing now to advance IPv6,
the ability to wait for a release is not always an option.  I do agree that
the issues should be properly documented in fact, I fully expect to
contribute to the work and share data from my point of view.
> 
> 2.
> "Gashinsky adds that Yahoo is conducting its own analysis of broken IPv6
> connectivity, which it will share with the Internet engineering community in
> June."
> => As a minimum, what the problem really is should be documented before
> proposing to adopt any solution to solve it, in particular if it is "ugly".
[jjmb] some of us have been gathering data and have experiences that are
insightful.  I believe the intention here minimally is to document the same
so the community can review and comment.  I do not believe there is a large
population of people that have attempted these sort of activities in scale,
I can assure that scale does matter.
> - Today, dual-stack hosts on Free's network query Free's DNS in IPv4 (at the
> only DNS address they know, received in DHCPv4)
[jjmb] it is not perfect but it is an indicator.  The hosts on the Free
network are clearly not fully dual stack capable, if they were the transport
could be used as an indicator of IPv6 reachability.
> 
> 4.
> "Return 0 answers for AAAA if, and only if: - Query comes over Ipv4; - ³A²
> record exists for same name; - DNSSEC is not used."
> => This hack would NOT ONLY be "ugly" (as acknowledged by their proponents),
> BUT ALSO would BREAK some of the IPv6 connectivities that are available today
> !!!
[jjmb] I do not see how this would break IPv6 connectivity.  I do agree that
there will likely be an impact to DNSSEC.
> ==> This hack MUST therefore be flatly rejected.
[jjmb] I see a number of people agree that this should be rejected.  I can
tell you from first hand experience over the past several months there are
some enigmas here and that issues with 6to4 and other transition
technologies present challenges to readying infrastructure for real
broadband IPv6 traffic.  And these challenges are likely not temporary.  I
suggest deferring rejection until the issues, challenges, and documentation
have been documented.
> 
> 
> If and when the mentioned OS problems are documented, it will be possible to
> fix them with patches in OSes, where they belong.

>> Unfortunately we don't have any control over what Yahoo or Google do to their
>> name servers.   I agree with you completely on what we SHOULD do, but if Igor
>> decides to filter AAAAs on IPv4 queries, we can't stop him.
> 
> Sure, and that's fair.
>>   We can refuse to endorse his solution, but really what's going on here is
>> that Igor (and Google before him) have come to us in good faith to tell us
>> about hacks they've done that they feel are necessary.   We shouldn't
>> discourage them from doing that,
> 
> Agreed.
[jjmb] see my point above, do you really assume the turn around time here is
short? 
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

=========================================
John Jason Brzozowski
Comcast Cable
e) mailto:[email-masked]
o) 609-377-6594
m) 484-962-0060
w) http://www.comcast6.net
=========================================

---

From: Igor Gashinsky <[email-masked]>
Date: Wed, 31 Mar 2010 14:51:02 -0400 (EDT)

On Wed, 31 Mar 2010, Nicholas Weaver wrote:

:: 
:: On Mar 31, 2010, at 12:28 AM, Igor Gashinsky wrote:
:: > 
:: > You are absolutely right -- it's not a DNS problem, it *is* a host 
:: > behavior problem. The issue is that it takes *years* to fix a host 
:: > behavior problem, and we need to engineer and deploy a fix much sooner 
:: > then that (hopefully about a year before the v4 exhaustion date). Given 
:: > that, is there something other then DNS that can address it better/faster?
:: 
:: Except that, since it IS a host problem, why not focus on improved detection and debugging....
:: 
:: EG, yahoo is one of the big pushes behind it.  THey could easily add auto-debugging to their homepgae would would detect this condition and NOTIFY the users that this will be a problem, same for google.

Yes, we can identify the users (this is how you get the stats in the first 
place), but afterwards, the problem then still becomes 2-fold. There is 
a *very large* subset of users who will ignore any warnings (see attempts 
to deprecate IE6), and there is another subset of users who will do just 
about anything that the the websites asks them to do (those who get 
phished all the time), and then there are a ton of people in-between. 

So, assuming the user doesn't ignore the warning, and is prompted to 
action, what do we tell the user to do? We don't know what's breaking v6 
on their system/network, and while we can put together an FAQ that the 
more technically inclined may follow, the majority of users won't know what 
it's talking about, so, do we tell them to call their ISPs, or contact 
GeekSquad or somebody and pay them to fix it? 

>From an ISP perspective, that may not be what they would want to do 
(support calls cost a lot of money), and from a user perspective, they are 
going to go "what is this ipv6 thing that's breaking, why do i care, and 
why should I waste money trying to fix it", or worse, "why the hell is 
$content_site probing my pc, I'm going to contact the FEDS!" (you'd be 
surprised how often that happens)..

This is why we are proposing this hack -- it will allow an ISP to turn on 
ipv6 on their network without worrying about breaking existing IPv4 users 
(with the caveat that it will prevent some v6-capable users for using the 
ipv6 internet until they can v6-transport DNS queries, and the negative 
impact on DNSSEC). And, it is going to be that operator's choice if they 
want to enable this feature or not.. It has also been pointed out to me by 
somebody who wishes to remain anonymous that I have not adequately 
described the benefits for deployment of this feature to ISP's, so quoting 
his message:

"Let's say that wireless Service Provider A has done their home work and
implemented Ipv6. Let's say that Wireless Service Provider B is happy
with IPv4 and have no plans to implement IPv6. There are all sorts of
devices, operating systems and versions in the network (Symbian,
Android, Linux, Windows, MacOS, PalmOS, embedded devices, aircards,
etc). Most of them are not field upgradable (or very hard to upgrade).

A user will not be able to tell why it takes 5 seconds longer to get to
Facebook on Service Provider A's network compared to Legacy Network B.
The user will blame the Network and probably switch to Legacy Network B.

A solution like the one you are proposing will allow a service provider
to roll out Ipv6 without fearing increased customer churn."

Fundamentally, what we are proposing is a way to ease the transitional 
pain, and making it an optional feature that the operator may enable, if 
they so chose (and see that the benefits outweigh the drawbacks). Is there 
a better way to address the problem?

Thanks,
-igor

---

From: "Dan Wing" <[email-masked]>
Date: Wed, 31 Mar 2010 12:19:28 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of John Jason Brzozowski
> Sent: Wednesday, March 31, 2010 10:38 AM
> To: Rémi Després; dnsop
> Cc: Ted Lemon; Jason Fesler; Igor Gashinsky
> Subject: Re: [DNSOP] Ugly DNS ack
> 
> Remi,
> 
...
> > BUT ALSO would BREAK some of the IPv6 connectivities that 
> > are available today !!!
>
> [jjmb] I do not see how this would break IPv6 connectivity.  

Any host that sends its AAAA queries over IPv4 would lose
IPv6 connectivity.  That includes common configurations of
hosts doing 6to4, Teredo, Windows XP, ISATAP, and any that 
-- for whatever reasons -- are using an IPv4 DNS server instead 
of an IPv6 DNS server.  On many OSs, the DNS server list is 
just a simple list and all DNS queries are sent to the 
same DNS server.  This is an issue with split-zone DNS and 
multiple interfaces (in the MIF working group), but split-zone
DNS is another topic for another time.  Igor's proposal 
requires, effectively, accelerating host modifications to 
perform A queries over an IPv4 interface and AAAA queries 
over an IPv6 interface, which MIF is working on.  How IPv6
DNS servers are configured is another problem with RFC5006 
being experimental (which is being fixed) and lack of 
RFC5006 implementations in routers (I know Cisco, and perhaps
others, lack RFC5006).

-d

> I do agree that there will likely be an impact to DNSSEC.
>
> >> So to my mind, the question is whether or not we want to 
> (and can!) have some
> >> say in what these hacks look like,
> > 
> > We do want it.

---

From: Nicholas Weaver <[email-masked]>
Date: Wed, 31 Mar 2010 12:21:36 -0700

A far better "solution" would be to instead segregate with different DNS server IPs.  

ISPs already have multiple DNS resolvers (eg, "no wildcarding" resolvers, DNSSEC test resolvers).  And the ISP knows if its giving out a v6 address or not for a client and routing IPv6 for that client.

And even then, I really wonder about the benefit.

I also object somewhat to the claim that you can't necessarily diagnose the cause.  With a combination of Java and JavaScript, plus user-agent examining, you probably can to a great degree, especially if you can convince the user to say "OK" to the signed applet.  But even without that, I'd suspect that with only a few root causes you could build a nice auto-diagnoser.

---

From: John Jason Brzozowski <[email-masked]>
Date: Wed, 31 Mar 2010 16:24:37 -0400

> RFC5006 implementations in routers (I know Cisco, and perhaps
[jjmb] some might consider this a feature given the issues and challenges
with some of the transition technologies you mention.  Support for RFC5006
from a residential point of view requires the home gateway/router to support
the same.  DHCPv6 (RFC3315) addresses this as well.

---

From: John Payne <[email-masked]>
Date: Wed, 31 Mar 2010 17:12:40 -0400

On Mar 31, 2010, at 3:19 PM, Dan Wing wrote:

> IPv6 connectivity.

Isn't this a misdirection?   

I suspect it's more like: any (address family agnostic) clients of a dual stacked nameserver will (non?) deterministically lose IPv6 connectivity to DNS-determined destinations.

ie, even if I only send DNS over IPv6 to my recursive nameserver, if it is dual stacked (often beyond my control), and for this specific query it prefers IPv4, then I will not get an answer for my AAAA under this proposal.

---

From: "Dan Wing" <[email-masked]>
Date: Wed, 31 Mar 2010 14:45:09 -0700

> I suspect it's more like: any (address family agnostic) 
> clients of a dual stacked nameserver will (non?) 
> deterministically lose IPv6 connectivity to DNS-determined 
> destinations.
> 
> ie, even if I only send DNS over IPv6 to my recursive 
> nameserver, if it is dual stacked (often beyond my control), 
> and for this specific query it prefers IPv4, then I will not 
> get an answer for my AAAA under this proposal.

It's likely cached by the ISP's nameserver, so it would work
fine under Igor's proposal.  And even if not cached, I would 
expect the content provider's authoritative DNS server to have 
whitelisted both IPv4 and IPv6 addresses of the ISP's 
nameservers.

-d

---

From: Jason Livingood <[email-masked]>
Date: Wed, 31 Mar 2010 18:11:01 -0400

> EG, yahoo is one of the big pushes behind it.  THey could easily add
> auto-debugging to their homepgae would would detect this condition and NOTIFY
> the users that this will be a problem, same for google.
> 

That is also a concern I share.

It seems that have the cart before the horse, so to speak.  IMHO, we need to
do the following (and there's no reason they cannot occur rapidly):

1 - Develop a clear problem statement that outlines (1) how "broken" users
are defined and (2) what effects this "brokenness" has on these end users
(or other parts of the Internet).

2 - Describe all the various methods and tactics by which end user
"brokenness" can be detected.  This may include website-based detection,
DNS-query-based detection, or a variety of other methods.

3 - Then, after we have agreement on Problem Definition and Problem
Detection, we can measure the problem to understand what the scope or scale
of the problem is. 

4 - After we understand Problem Definition, Problem Detection, and Problem
Scope, then you can arrive at possible solutions.  Seems like in this case
we sort of *started* here, which concerns me and I think we need to be
careful that discussion thus far does not constrain development of a full
list of Solution Options.  I further believe we will need to encourage the
pursuit of multiple solutions simultaneously.  Lastly, just because end user
software upgrades may be difficult doesn't mean we shouldn't do them and
shouldn't focus just as much energy on those than other options.

Jason

---

From: John Jason Brzozowski <[email-masked]>
Date: Thu, 01 Apr 2010 00:31:22 -0400

What ultimately gets white-listed is the query source IP address, IPv4 and
or IPv6.  This in part depends on whether the authoritative name servers of
the content provider are dual stack or not.

John

---

From: John Jason Brzozowski <[email-masked]>
Date: Thu, 01 Apr 2010 00:29:48 -0400

Not necessarily, if a dual stack hosts communicates with a recursive name
server over both IPv4 and IPv6 and other conditions are met then I believe
it would be fine based on what was presented.

John

On 3/31/10 5:12 PM, "John Payne" <[email-masked]> wrote:

> I suspect it's more like: any (address family agnostic) clients of a dual
> stacked nameserver will (non?) deterministically lose IPv6 connectivity to
> DNS-determined destinations.
> 
> ie, even if I only send DNS over IPv6 to my recursive nameserver, if it is
> dual stacked (often beyond my control), and for this specific query it prefers
> IPv4, then I will not get an answer for my AAAA under this proposal.
> 
>

---

From: "Patrick W. Gilmore" <[email-masked]>
Date: Thu, 1 Apr 2010 01:41:49 -0400

> it would be fine based on what was presented.

What other conditions need to be met?

I did not think there was any way for a host to signal a recursive NS to use v6 or v4 transport.

-- 
TTFN,
patrick

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 31 Mar 2010 22:51:00 -0700

> I did not think there was any way for a host to signal a recursive NS to use v6 or v4 transport.

It seems to me that you'll need to: (at least)
1) ask for a AAAA from the client -> recursive resolver
2) dual-stack the recursive resolver
3) provide AAAA's with 'better' latency/response than the A records
associated with the NS's for your domain (the domian being looked up)
4) decide to answer AAAA queries over ipv6 transport from these NS hosts.
5) also reply with A (and other normal records) when asked for them
over either transport

Just because 'if the AAAA query comes over ipv6, auto-whitelist +
answer' there's nothing stopping the server from responding to A
queries over ipv6 with ipv4 addresses. The NS here has no real idea
about the original requestor, only what the recursive resolver decides
to use for a transport.

you can suppose that if a recursive resolver has 'better' ipv6
transport it should be used 'more', and that potentially the
AS/customer-set represented by it will have 'better' (or at least
'good enough') ipv6 transport, but that's not guaranteed.

All that said, I think I like Igor's idea, I'm not sure I'd implement
it without some research first... the same issues that keep
large-content-providers from adding blanket AAAA records would likely
also affect DNS queries over ipv6 transport.

-Chris

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 31 Mar 2010 23:09:45 -0700

> people in charge of giving them out.

all that means is that people need to start being able to USE ipv6,
not that they need to stop using ipv4. It may be useful for individual
organizations to stop using ipv4, but it's certainly NOT a requirement
that 'the people that give them out' have been putting forth. It may
be useful for organizations which can not get allocations of ipv4
space to 'buy' space (get a link to a provider with spare ipv4) and
NAT their entire enterprise behind this...

Really that entire discussion is an aside, and a distraction from the
actual conversation here, should a service provider find a simpler way
to automatically whitelist users/enterprises for ipv6 services. The
service provider here cares deeply about their customers' service
experience, they do not want to break users, slow them down or appear
to provide 'worse' service all because someone decided long ago that
this new fangled 128bit address thingy was a good plan.

Taking a measured approach to roll out of services to existing
customers is a good thing. Moving toward more ipv6 adoption is a good
thing, is there a reason that we seem to want to quash ipv6 adoption?
(based on the conversation so far it surely seems as though we do...)

-Chris