Subject: Re: [DNSOP] Ugly DNS ack

From: [email-masked]
Date: Thu, 1 Apr 2010 08:13:38 +0000

On Wed, Mar 31, 2010 at 02:45:09PM -0700, Dan Wing wrote:
> > On Mar 31, 2010, at 3:19 PM, Dan Wing wrote:
> > 
> > > Any host that sends its AAAA queries over IPv4 would lose
> > > IPv6 connectivity.
> > 
> > Isn't this a misdirection?   
> > 
> > I suspect it's more like: any (address family agnostic) 
> > clients of a dual stacked nameserver will (non?) 
> > deterministically lose IPv6 connectivity to DNS-determined 
> > destinations.
> > 
> > ie, even if I only send DNS over IPv6 to my recursive 
> > nameserver, if it is dual stacked (often beyond my control), 
> > and for this specific query it prefers IPv4, then I will not 
> > get an answer for my AAAA under this proposal.
> 
> It's likely cached by the ISP's nameserver, so it would work
> fine under Igor's proposal.  And even if not cached, I would 
> expect the content provider's authoritative DNS server to have 
> whitelisted both IPv4 and IPv6 addresses of the ISP's 
> nameservers.
> 
> -d

	i would hope that "big Brother" would quit trying to 
	be helpful.  If I ask for AAAA, regardless of what
	transport you receive the query on, please give me
	what I ask for.  

	With forwarders, IMRs, and proxies josteling for position
	between the stub and the authoritative server, its presumptious
	and arrogant to presume that the transport the query was received 
	on is the same as the stub has.

	DNS data has zero to do with the transport(s) queries are received
	and responded to.  Lets not conflate the two.

--bill

---

From: Jason Fesler <[email-masked]>
Date: Wed, 31 Mar 2010 21:25:11 -0700

I wanted to clarify a couple things on this thread.

Our concern is more than just "os issues".  Many apps today already ask for A/AAAA.  The bigger issue to me is related to when the host tries connecting to the IPv6 address, using a route that exists but is either broken or suffers serious performance problems.  Users see that as "Site Down".  The percentage is high [see #1]; our business people would never let us deploy IPv6 unless we can mitigate it by things like selectively enabling IPv6 towards specific operators and end users that appear to have IPv6 "for real".

Re: getting clients to use and prefer IPv6 resolvers:   this issue is not lost on us.  Today, we don't have a clear solution on this issue that works for everyone.    However, the access providers we talked to seemed to think this would become a resolvable issue (no pun intended).   This may be via access provider specific means, or perhaps standards updates (and product updates) have happen by the time IPv6 gains significance on the internet.  

As to when the filter-aaaa hack works, there are very limited conditions:

   1: Extra build option for the resolver.  Not compiled by default.
   2: Enabled in the configuration.  By default, disabled.    
   3: Query is for AAAA.
   4: Query arrived from client via INET, not INET6.
   5: Results have both A and AAAA.  Perform an "A" lookup if needed to satisfy this request.
   6: Results are not signed.
   
IPv6 only web sites, will still return the AAAA.

Any site that is DNSSEC signed, will return results *unaltered*.  We don't want to see DNSSEC broken.  

Yes, there is a "filter-aaaa break-dnssec;" option.   We see it being useful in very limited cases (enterprise or lab networks where V6 routes might exist, but not public connected yet).  It is just a tool; shoot your foot at your own risk.   When DNSSEC is involved, we absolutely do not want to break any trust (or functionality).  I look forward to the day when the majority of users are protected by DNSSEC.

Hope this helps,

-jfesler

[#1] http://www.ripe.net/ripe/meetings/ripe-58/content/presentations/Colitti-IPv6atGoogle-RIPE58.pdf  publicly shows this as 0.1%.  Thanks, Lorenzo.

---

From: Jim Reid <[email-masked]>
Date: Thu, 1 Apr 2010 10:49:54 +0100

On 1 Apr 2010, at 05:25, Jason Fesler wrote:

> Our concern is more than just "os issues".  Many apps today already  
> ask for A/AAAA.  The bigger issue to me is related to when the host  
> tries connecting to the IPv6 address, using a route that exists but  
> is either broken or suffers serious performance problems.  Users see  
> that as "Site Down".  The percentage is high [see #1]; our business  
> people would never let us deploy IPv6 unless we can mitigate it by  
> things like selectively enabling IPv6 towards specific operators and  
> end users that appear to have IPv6 "for real".

This is a valid concern. It does not and should not need to be  
addressed (excuse the pun) by making authoritatiev servers do stupid/ 
wrong/bad things. Others have already pointed out -- and it looks like  
they will have to continue -- that it is just wrong to make a name  
server return different data based on the network protocol used to  
make the query. For starters, the thing making the query to an  
authoritative server usually isn't the edge device. There's almost  
always a resolving server and/or crippled middleware boxes in the  
query path. And even if the edge device is IPv4 only, that doesn't  
mean it has no interest in IPv6 addresses. [It's not just SMTP  
delivery software that looks up MX records for instance.] Then there's  
the impact on caches. If your scheme goes ahead, IPv6-capable hosts  
will sometimes be told there's no IPv6 for some name because the  
resolver cached that from an earlier query over IPv4 which resulted in  
the authoritative server jumping to the wrong conclusion and then  
telling lies.

Why don't yahoo approach the problem the same way google has done for  
IPv6 to www.google.com? They only hand out AAAA records for this name  
to ISPs who can demonstrate they have solid IPv6 connectivity. This is  
ugly and distasteful. But it doesn't involve egregious damage to the  
DNS or resolver/cache behaviour.

---

From: Mohacsi Janos <[email-masked]>
Date: Thu, 1 Apr 2010 12:41:30 +0200 (CEST)

> [#1] http://www.ripe.net/ripe/meetings/ripe-58/content/presentations/Colitti-IPv6atGoogle-RIPE58.pdf  publicly shows this as 0.1%.  Thanks, Lorenzo.

I still cannot see, how can you assses the quality of IPv6 connectivity of 
someone, based on a DNS query. DNS queries mostly are coming from the ISP 
public DNS servers. They are nothing to do with IPv6 connectivity of the 
end-users.

Will it be effective?

Best Regards,
 		Janos Mohacsi

---

From: Jason Fesler <[email-masked]>
Date: Thu, 1 Apr 2010 04:38:24 -0700

On Apr 1, 2010, at 2:49 AM, Jim Reid wrote:
> wrong/bad things.

Actually, that dns hack is for resolvers.  That's the only place the end client talks to a DNS server; that's the only place it makes sense to modify the result.  It also only makes sense in a limited type of deployments; particularly those where clients are directly expected to ask the ISP resolver (or, where the router at the house forwards the request on the same protocol, without caching).  Very limited use case.  

> to ISPs who can demonstrate they have solid IPv6 connectivity.

For our authoritative name servers, we will indeed be using a whitelisting approach.  There is more than just the ISP offering solid AAAA service; even once an ISP offers it, there is still a significant time until the users have moved to it. The filter-aaaa option permits us to still whitelist a set of DNS resolvers, when our numbers indicate the user base has a higher than acceptable number of broken systems that would effectively consider us "site down".   

An alternative is have the access provider run two DNS pools - one for the v4-only provisioned user base, and one for the v6-provisioned user base (they'd whitelist just the latter).  This alternative is certainly less controversial; but it is a larger footprint (for some, this is significant, where space/power is limited).  Additionally, it doesn't do anything to mitigate the users with broken v6 routes.  

The bottom line is, that if a given provider's users send in too many complaints about connectivity problems, the business folks will push for dewhitelisting.   That's bad for both the content and access provider sides for a variety of reasons.

---

From: John Jason Brzozowski <[email-masked]>
Date: Thu, 01 Apr 2010 07:56:54 -0400

I think you missed something slightly.  Based on the proposal the
connectivity to the recursive name server from the client is being leveraged
as *partial* indicator, not the connectivity from the recursive name server
to the authoritative name server(s).

John

On 4/1/10 6:41 AM, "Mohacsi Janos" <[email-masked]> wrote:

>> Our concern is more than just "os issues".  Many apps today already ask for
>> A/AAAA.  The bigger issue to me is related to when the host tries connecting
>> to the IPv6 address, using a route that exists but is either broken or
>> suffers serious performance problems.  Users see that as "Site Down".  The
>> percentage is high [see #1]; our business people would never let us deploy
>> IPv6 unless we can mitigate it by things like selectively enabling IPv6
>> towards specific operators and end users that appear to have IPv6 "for real".
>> 
>> Re: getting clients to use and prefer IPv6 resolvers:   this issue is not
>> lost on us.  Today, we don't have a clear solution on this issue that works
>> for everyone.    However, the access providers we talked to seemed to think
>> this would become a resolvable issue (no pun intended).   This may be via
>> access provider specific means, or perhaps standards updates (and product
>> updates) have happen by the time IPv6 gains significance on the internet.
>> 
>> Yes, there is a "filter-aaaa break-dnssec;" option.   We see it being useful
>> in very limited cases (enterprise or lab networks where V6 routes might
>> exist, but not public connected yet).  It is just a tool; shoot your foot at
>> your own risk.   When DNSSEC is involved, we absolutely do not want to break
>> any trust (or functionality).  I look forward to the day when the majority of
>> users are protected by DNSSEC.
>> 
>> Hope this helps,
>> 
>> -jfesler
>> 
>> 
>> [#1] 
>> http://www.ripe.net/ripe/meetings/ripe-58/content/presentations/Colitti-IPv6a
>> tGoogle-RIPE58.pdf  publicly shows this as 0.1%.  Thanks, Lorenzo.
> 
> 

=========================================
John Jason Brzozowski
Comcast Cable
e) mailto:[email-masked]
o) 609-377-6594
m) 484-962-0060
w) http://www.comcast6.net
=========================================

---

From: Suzanne Woolf <[email-masked]>
Date: Thu, 1 Apr 2010 12:06:28 +0000

On Thu, Apr 01, 2010 at 10:49:54AM +0100, Jim Reid wrote:
> make the query.

Modulo Jason's correction-- we're talking about recursive servers
inside ISP networks....

I admit I'm getting thoroughly confused about any kind of general
architectural principle here. There seem to be several proposals out
there that suggest easing v4-v6 co-ex/transition problems by having
recursive resolvers "magically lie" to end systems, either by dropping
records that exist in the auth zone or synthesizing ones that don't.

I've looked closely at filter-aaaa and DNS64 recently, and glanced at
others, including various forms of split-brain madness. I follow this
area reasonably closely and can occasionally claim to know a little
about DNS. But I don't know what to say at any level higher than the
specifics of particular combinations of v4/v6 connectivity about
proper (or not) ways to bend or break DNS.

It would be nice for us, as DNS Operations experts, to be able to say
something about uses and abuses of the DNS besides "this particular
thing is broken because you could do this particular other thing
instead".

Are there any thoughts here on when, as a general principle, the "ends
justify the means" in support of v4/v6 co-ex/transition and what forms
of brutality against DNS purity seem to cause the least collateral
damage?

Jim and a few others seem to me to be taking the position "changing
DNS answers in the resolver based on end system transport is always
bad, in this particular case the auth server operator/content provider
should whitelist instead". OTOH, a co-author of the DNS64 "here's how
to synthesize AAAAs for v6-only hosts" spec (which I realize implies
no endorsement, hold your fire) is also arguing against filter-aaaa
here.

I'm wondering whether there's a slightly more nuanced principle hiding
with the pony.

Suzanne

---

From: Rémi Després <[email-masked]>
Date: Thu, 1 Apr 2010 14:25:05 +0200

John,

Thanks for the detailed reaction.
Further comments below.

Le 31 mars 2010 à 19:38, John Jason Brzozowski a écrit :

> I hope you do not mind, I have take the liberty to pull some text from your
> original post

[rd] No problem at all.

> and will reply to them here:
> 
> On 3/31/10 10:20 AM, "Rémi Després" <[email-masked]> wrote:
>> ...
>> => If there are indeed OS bugs that break connectivity, they should justify
>> quick patches like those that concern security.
> [jjmb] anyone who has had to deal with releasing software can certainly
> attest to the fact that these sort of fixes/enhancements may not occur in a
> timely manner.

[rd] Aren't security patches quickly dispatched? Wouldn't a modification of all recursive DNS servers be an even more difficult task? 

>  There are things we need to be doing now to advance IPv6,
> the ability to wait for a release is not always an option.  I do agree that
> the issues should be properly documented in fact,

[rd] Yes. That's the key.

> I fully expect to
> contribute to the work and share data from my point of view.

[rd] Agreeing on what is the proper behavior OF HOSTS is IMHO most urgent, for the remainder of the discussion to start on solid grounds.

Observed broken IPv6 connectivities shouldn't exist between communicating hosts (clients and/or servers) that apply these NATURAL RULES:
a) Send packets to an IPv6 public address only from a public IPv6 source address. Otherwise, the return path will be broken.
b) Send IPv6 packets from a 6to4 address only to destinations that are also a 6to4 addresses. Otherwise, return paths may be inexistent.
c) By default, send IPv6 packets that don't exceed the guaranteed PMTU in IPv6, i.e. 1280 octets. Otherwise, some packets may be systematically discarded on their way because of their size.  (Note that TCP/IP stacks that can determine than larger PMTUs exist on some identified e2e paths, by PMTUD or otherwise, may send larger packets on these paths, but this is far more sophisticated, and can remain optional.)  

>> ...
>> => As a minimum, what the problem really is should be documented before
>> proposing to adopt any solution to solve it, in particular if it is "ugly".
> [jjmb] some of us have been gathering data and have experiences that are
> insightful.  
> I believe the intention here minimally is to document the same
> so the community can review and comment.  I do not believe there is a large
> population of people that have attempted these sort of activities in scale,
> I can assure that scale does matter.

[rd] Agreed: scale does matter, and documenting observed facts is very precious.
I look forward to what you have gathered.

>> 3.
>> "Only way of knowing the user has working IPv6 connectivity, is if the AAAA
>> query came over IPv6!"
>> => This DOESN'T WORK :
>> - Today, dual-stack hosts on Free's network query Free's DNS in IPv4 (at the
>> only DNS address they know, received in DHCPv4)
>> - These hosts, because they have valid IPv6 addresses (i.e. have IPv6
>> enabled), ask for both As and AAAAs.
>> - For maps.google.fr, for example, BOTH types of RRs are in the DNS.
>> - They are included in DNS responses
>> - Hosts then use IPv6 (preferred in case of choice).
> [jjmb] it is not perfect but it is an indicator.  The hosts on the Free
> network are clearly not fully dual stack capable,

> if they were the transport
> could be used as an indicator of IPv6 reachability.

[rd]
- My host has OS X, which makes it IPv6 "capable".
Since it gets a native IPv6 address (with the IPv6 prefix it receives from Free's CPE), it becomes IPv6 "enabled".
As it also works in IPv4 as usual, it is  AFAIKT fully dual stack.
- Note that, so far, the dual-stack model keeps the DNS application layer independent from the IP layer, so that DNS servers may provide both As and AAAAs even if reached in IPv4. 
Changing this as would be a step backward.

>> "Return 0 answers for AAAA if, and only if: - Query comes over Ipv4; - “A”
>> record exists for same name; - DNSSEC is not used."
>> => This hack would NOT ONLY be "ugly" (as acknowledged by their proponents),
>> BUT ALSO would BREAK some of the IPv6 connectivities that are available today
>> !!!
> [jjmb] I do not see how this would break IPv6 connectivity.

[rd] As I said, "dual-stack hosts on Free's network query Free's DNS in IPv4 (at the only DNS address they know, received in DHCPv4)".
The DNS server returns available addresses, IPv4 and/or IPv6.
This has worked satisfactorily for more than two years, and is used extensively, in particular to reach Google and IETF servers.
If Free's DNS server would stop returning IPv6 addresses, as proposed in the hack being discussed, its dual-stack hosts would no longer reach Google and IETF in IPv6.  

>  I can
> tell you from first hand experience over the past several months there are
> some enigmas here and that issues with 6to4 and other transition
> technologies present challenges to readying infrastructure for real
> broadband IPv6 traffic.

[rd] Being a long time user of IPv6, and having at least enough knowledge of IPv6 to have designed 6rd, I will be glad, if I can, to help resolving these enigmas once they are identified.
My guess is that the three natural rules above do resolve a number of them, if not all, but let us see. 

>  And these challenges are likely not temporary.  I
> suggest deferring rejection until the issues, challenges, and documentation
> have been documented.

[rd] The proposed immediate rejection of the proposed hack is only because it would break some fully legitimate existing connectivities.
But I do agree that, when more detailed facts are documented, we can look at the problem with fresh eyes, and see *which* hacks could be useful, and *where*. 

>> My strong reaction is only against this particular hack because:
>> - it is based on the idea that OSes cannot be patched where they are bugged
>> - it destroys legitimate connectivity for OSes that aren't bugged.
> [jjmb] see my point above, do you really assume the turn around time here is
> short? 

In any case, the earlier host-originated problems are identified, the earlier they are likely to be fixed (as they must be).

Note that if an ISP (e.g. Comcast) implements the proposed hack only in its own recursive DNS, this is a choice IETF is not in a position to object to: it doesn't change anything for others.
This will prevent some of the well behaved hosts of this ISP to actually use the IPv6 service, but no one else will be harmed. 

Regards,
RD

---

From: Thierry Moreau <[email-masked]>
Date: Thu, 01 Apr 2010 10:42:39 -0400

Dear all:

Thanks Suzanne for this post. Some informational evidence below.

> thing is broken because you could do this particular other thing

While supporting my household LAN based on Linux boxes and visiting 
Laptops of indeterminate OSes (user base including a few teenagers and 
college students) in 2007, I encountered these DNS inefficiencies 
associated with useless AAAA queries from applications (notably a web 
browser).

I was surprised that the readily available fix was in the *application* 
configuration, and not in a glibc setting (/etc/nsswitch.conf) that 
would apply to every applications running on the box.

So the "particular thing" being broken could be shifted from the OS 
functions to the application developer. Inside the box, a filter for 
AAAA queries could be seen by some as a barrier to IPv6 migration (being 
application by application).

Just my very casual observation of the source of useless AAAA queries.

Regards,

-- 
- Thierry Moreau

CONNOTECH Experts-conseils inc.
9130 Place de Montgolfier
Montreal, QC, Canada H2M 2A1

Tel. +1-514-385-5691

---

From: Jason Livingood <[email-masked]>
Date: Thu, 01 Apr 2010 10:41:02 -0400

> The bottom line is, that if a given provider's users send in too many
> complaints about connectivity problems, the business folks will push for
> dewhitelisting.   That's bad for both the content and access provider sides
> for a variety of reasons.

Or, you could just refer them back to their ISP for support, which is how
things work in the v4 world today.  This is certainly my preference.

Jason

---

From: Ted Lemon <[email-masked]>
Date: Thu, 1 Apr 2010 08:18:43 -0700

> DNS or resolver/cache behaviour.

Do you really think this is better than what Igor's proposing?   To me it seems worse. You're still giving out wrong answers - you're just giving them out to more devices this way.

I think a principled position can be taken that giving out wrong answers should not be done, but once you've decided that you're willing to give out wrong answers, we're really just haggling over the price.

---

From: Andras Salamon <[email-masked]>
Date: Thu, 1 Apr 2010 16:51:28 +0100

On Thu, Apr 01, 2010 at 08:18:43AM -0700, Ted Lemon wrote:
>I think a principled position can be taken that giving out wrong
>answers should not be done, but once you've decided that you're willing
>to give out wrong answers, we're really just haggling over the price.

Would "RA=0, NODATA" as a response to an AAAA query really be a
wrong answer?

-- Andras Salamon                   [email-masked]

---

From: Jim Reid <[email-masked]>
Date: Thu, 1 Apr 2010 19:02:39 +0100

On 1 Apr 2010, at 16:18, Ted Lemon wrote:

> Do you really think this is better than what Igor's proposing?

Yes. Although it is unpalatable, it's the lesser of two evils IMO.  
What google are doing is essentially split DNS IIUC. AAAA records for  
www.google.* are only returned to ISPs who have some sort of IPv6  
peering with them. The decision to supply IPv6 data isn't based on  
whether the query came in over IPv6 or not.

I do have reservations about this approach because it's not just  
another sticking plaster over the gaping chest wound that is  
widespread IPv6 deployment. It's another way to avoid tackling that  
bigger issue or deferring it.

> answers should not be done, but once you've decided that you're  
> willing to give out wrong answers, we're really just haggling over  
> the price.

I agree with your point about the haggling Ted. I'm not so sure we  
agree on the definition of a wrong answer.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 2 Apr 2010 02:04:46 +0800

> agree on the definition of a wrong answer.

I think Ed's point about caching behavior proves you to be correct.

---

From: [email-masked]
Date: Thu, 1 Apr 2010 18:35:18 +0000

one might actually extrapolate here (and maybe look back a couple 
decades) ... there used to be many different transports around - 
and about the timethe DNS "gel'ed", most had become vestigal.  We
are now in the evoultionary "fork in the road" when we have an
emergent, new transport that demands attention.  I suspect that 
as things evolve, giving the DNS the ability to be transport agile
is a good thing.  So I don't want to be too short-sighted here and
bless one or two hacks that might ease todays transitional requirements.

We should have some clear long-range thought on transport agility

--bill

On Thu, Apr 01, 2010 at 10:49:54AM +0100, Jim Reid wrote:
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop