Subject: Re: [DNSOP] FYI: DNSOPS presentation

From: Igor Gashinsky <[email-masked]>
Date: Tue, 30 Mar 2010 19:10:37 -0400 (EDT)

On Tue, Mar 30, 2010 at 12:15:39AM -0400, Andrew Sullivan wrote:

:: Rather than having the DNS magically lie to people, why not use the
:: DNS detection mechanism as an indicator that a customer has a broken
:: v6 implementation.  Then you can turn off _that customer's_ IPv6
:: connectivity, contact them, and tell them what their problem is.  This
:: has three benefits:

The problem here is the ISP doesn't actually know that their users have a 
broken v6 implementation -- the only way they can test for it would be if 
users went to their portals, which, in most cases, users won't do. So, the 
only people who do know if the user has a working v6 implementation are 
the content/hosting providers, who have a very limited action they can 
take -- either give out AAAA to ISP recursive servers behind which there 
are broken users and break those users, or don't hand out AAAA to those 
resolvers, thereby not enabling ipv6 for *any* user behind that 
resolver, whether they are working or not...

I can tell you that as a content provider, we are not willing to break 
those users (would you be willing to negatively impact 400K+ users who 
were able to get to you over ipv4 just fine, for the *maybe* 1k users who 
could only get to you over ipv6? How about break 400k users so that 
1.2M users can get to you over ipv6, even though all of those 1.2M 
can get to you over ipv4 just fine?)

So, the question now is, what can be done? By no means do I think that 
lying based on transport is a good idea, however, I simply don't have a 
better one, and, this is a real problem, which is delaying ipv6 deployment 
for a number of people. So, if anybody else has a better option, I (and 
other content providers) would love to hear it! Unfortunately, this is the 
best we've come up with thusfar...

Thanks,
-igor (the crazy yahoo guy who presented this) :)

---

From: "Dan Wing" <[email-masked]>
Date: Tue, 30 Mar 2010 16:29:16 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of Igor Gashinsky
> Sent: Tuesday, March 30, 2010 4:11 PM
> To: Andrew Sullivan
> Cc: [email-masked]
> Subject: Re: [DNSOP] FYI: DNSOPS presentation
> 

Solve it in the browser, which is well-placed to know if there
really is connectivity and can even determine if IPv6 (or IPv4)
is temporarily broken or abnormally slow:

http://tools.ietf.org/html/draft-wing-http-new-tech-01
http://tools.ietf.org/html/draft-yourtchenko-tran-announce-dns-00

-d

> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

---

From: Pekka Savola <[email-masked]>
Date: Wed, 31 Mar 2010 10:10:35 +0300 (EEST)

On Tue, 30 Mar 2010, Igor Gashinsky wrote:
> So, the question now is, what can be done? By no means do I think that lying 
> based on transport is a good idea, however, I simply don't have a better one, 
> and, this is a real problem, which is delaying ipv6 deployment for a number 
> of people. So, if anybody else has a better option, I (and other content 
> providers) would love to hear it! Unfortunately, this is the best we've come 
> up with thusfar...

I'm not sure how this trick is going to work when you step back a bit: 
You'll still need to maintain a whilelist of networks where the ISP 
has agreed to implement this hack in their DNS recursive server.  You 
can't start serve AAAA records to others.

The only reason for doing the hack would be if you had stricter 
requirements than Google.  They whilelist ISPs' resolvers but trust 
that the ISPs resolve issues that get reported to them.  You would be 
whilelisting ISPs but require a mechanism that cuts down on broken 
IPv6 experience.  You're right that even a google-whitelisted ISP user 
could experience issues (but not report to the ISP).

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Igor Gashinsky <[email-masked]>
Date: Wed, 31 Mar 2010 03:25:35 -0400 (EDT)

:: Solve it in the browser, which is well-placed to know if there
:: really is connectivity and can even determine if IPv6 (or IPv4)
:: is temporarily broken or abnormally slow:
:: 
:: http://tools.ietf.org/html/draft-wing-http-new-tech-01
:: http://tools.ietf.org/html/draft-yourtchenko-tran-announce-dns-00
:: 
:: -d

I will completely agree with you that this is where the problem *should* 
be solved. However, we are about 5 years (if not more) too late in solving 
it that way if we wanted to deploy ipv6 right now -- that is what we are 
trying to address. Hell, IE6 still makes up close to 18% of all users out 
there despite everybody trying to deprecate that browser, and the 
percentage of ipv6 capable users is roughly the same as the percentage of 
windows 98 users out there (0.3%)... So, given that clearly users don't 
update their software often enough, it is too late to fix the applications 
that are already deployed on users PCs (and their broken home 
gateways/firewalls/etc), so, what do we do *right now*, to get those 
"broken users" through the next 3-5 years till they upgrade? 

Now, we should absolutely get the software fixed to stop breakage going 
forward, but, we still have to address the currently broken users.

I *hate* doing this in DNS, but, I don't see another way to effect change 
fast (let's face it, ISP's are much more likely to roll out a new version 
of bind/powerdns/secure64/whatnot then users updating their PCs/home 
gateways/whatnot in a short timeframe). Is there a better way? If not, and 
something will have to "take one for the team" to make IPv6 work without 
breaking IPv4, is there anything other then DNS that would be a better 
candidate, with a deployment horizon of under a year?

Thanks,
-igor

---

From: Igor Gashinsky <[email-masked]>
Date: Wed, 31 Mar 2010 03:44:32 -0400 (EDT)

On Wed, 31 Mar 2010, Pekka Savola wrote:

:: On Tue, 30 Mar 2010, Igor Gashinsky wrote:
:: > So, the question now is, what can be done? By no means do I think that
:: > lying based on transport is a good idea, however, I simply don't have a
:: > better one, and, this is a real problem, which is delaying ipv6 deployment
:: > for a number of people. So, if anybody else has a better option, I (and
:: > other content providers) would love to hear it! Unfortunately, this is the
:: > best we've come up with thusfar...
:: 
:: I'm not sure how this trick is going to work when you step back a bit: You'll
:: still need to maintain a whilelist of networks where the ISP has agreed to
:: implement this hack in their DNS recursive server.  You can't start serve
:: AAAA records to others.

You are absolutely correct, this is not to replace the whitelist, this is 
on top of it. There are different discussions going on right now on how to 
make the whitelisting process easier/more universal/more automated, but 
this realisticly is on top of that. In the grand scheme of things, once 
people measure the # of broken users for each resolver that hits them, 
they can then automagicly whitelist the servers which are within the SLA 
for "broken users" without any ISP out there applying for the whitelist -- 
doing this hack is simply a tool to guarantee that your resolver will pass 
the "SLA test" (ie we won't break too many users if we whitelist your 
resolver).

:: The only reason for doing the hack would be if you had stricter requirements
:: than Google.  They whilelist ISPs' resolvers but trust that the ISPs resolve
:: issues that get reported to them.  You would be whilelisting ISPs but require
:: a mechanism that cuts down on broken IPv6 experience.  You're right that even
:: a google-whitelisted ISP user could experience issues (but not report to the
:: ISP).

Yes, I think most people *will* have a stricter requirement then Google 
does for their "ipv6 beta program" (say, no more then 1/10,000 broken users 
behind your resolvers), and I will be very surprised if google doesn't 
tighten up their requirements as well (or at least conciders doing so) in 
the near future, but, clearly, working for a competitor, I won't speak 
for them :)

To me, breaking 1/2000 users (which is what happends today) is not 
acceptable when rolled out on a larger scale (10's or 100's of millions of 
users), but that's just me, and, I do not believe that the Google ipv6 
pilot is anywhere near that scale yet...

Thanks,
-igor

---

From: "Dan Wing" <[email-masked]>
Date: Wed, 31 Mar 2010 08:45:28 -0700

> windows 98 users out there (0.3%)... 

Users running IE6 today are IPv4-only users.  If/when they go
to IPv6, they will be running Windows 7 and whatever browser
is shipped by Microsoft.

And the two drafts cited above are not solely useful for IPv6 
transition.  Many are pining for another transition -- from TCP
to SCTP -- which shares a lot of characteristics and user-experience
problems with the IPv6 transition.  (Worse, in some instances; 
better in other instances.)

> So, given that clearly users don't 

Those users will continue to run IPv4 for the next 3-5 years and
will not do anything with IPv6 even if their ISP offered them
IPv6.

> breaking IPv4, is there anything other then DNS that would be 
> a better candidate, with a deployment horizon of under a year?

It seems solvably operationally, by asking ISPs to point their
IPv4-only subscribers at an ISP-operated DNS server which 
purposefully breaks AAAA responses (returns empty answer), and 
to point their dual-stack subscribers at an ISP-operated DNS 
server which functions normally.

Advanced IPv4-only users wanting to do AAAA queries (e.g., 
Teredo users, 6to4 users, etc.) should be sufficiently advanced 
to point themselves at the ISP's normal nameserver or a 
public DNS server on the Internet (e.g., Hurricane 
Electric's, Google's, etc.).  That won't affect users running
uTorrent (which uses Teredo to provide IPv6 connectivity)
because it doesn't do AAAA queries to find peers.

-d

---

From: Igor Gashinsky <[email-masked]>
Date: Wed, 31 Mar 2010 16:37:49 -0400 (EDT)

On Wed, 31 Mar 2010, Dan Wing wrote:

:: Users running IE6 today are IPv4-only users.  If/when they go
:: to IPv6, they will be running Windows 7 and whatever browser
:: is shipped by Microsoft.

Why do you say that? As far as I know, IE6 is an ipv6-capable browser, 
as long as it's going to FQDN's.. So, what about IE6/XP users who 
installed bittorent clients (or spyware/trojans) that enabled ipv6 for 
them without the user knowing about it?

:: It seems solvably operationally, by asking ISPs to point their
:: IPv4-only subscribers at an ISP-operated DNS server which 
:: purposefully breaks AAAA responses (returns empty answer), and 
:: to point their dual-stack subscribers at an ISP-operated DNS 
:: server which functions normally.
:: 
:: Advanced IPv4-only users wanting to do AAAA queries (e.g., 
:: Teredo users, 6to4 users, etc.) should be sufficiently advanced 
:: to point themselves at the ISP's normal nameserver or a 
:: public DNS server on the Internet (e.g., Hurricane 
:: Electric's, Google's, etc.).  That won't affect users running
:: uTorrent (which uses Teredo to provide IPv6 connectivity)
:: because it doesn't do AAAA queries to find peers.

This is *exactly* what we are proposing -- the feature to return empty 
answers would be needed for ipv4-only subscribers in order to keep them 
ipv4-only. Also, if a fully ipv6-capable user visits that person's home, 
the recursor would then be able to make the call on if they should pass 
through AAAA to that particular user or not... I am by no means advocating 
to make this behavior a default, just a feature.

Thanks
-igor

---

From: "Dan Wing" <[email-masked]>
Date: Wed, 31 Mar 2010 13:55:02 -0700

> installed bittorent clients (or spyware/trojans) that enabled 
> ipv6 for them without the user knowing about it?

Yes, thanks for correcting me.

I agree they will have a poor experience (due to Teredo).

But Remi's point is that those same systems (running Windows XP
and IE6) using 6rd will be denied the ability to access content 
via IPv6.  Which removes an incentive for ISPs to add 6rd (and
offload the NAT44 they may soon have to install).

> to make this behavior a default, just a feature.

I'm saying this should be do-able entirely within the ISP's
DNS, without coordination or involvement with the content 
provider's DNS.  

For example, imagine the ISP's nameserver has a new "allow-aaaa-response"
option that would be configured like:

  #
  # list IPv4 addresses that are known to have real IPv6 connectivity.
  # (e.g., this is all of the ISP's subscribers that are also 
  #  connected using 6rd).
  #
  acl aaaa_whitelist { 172.16.72.0/24; 192.168.1.0/24; };
  ...
  
  options {
  ...
  allow-aaaa-response { aaaa_whitelist };
  ...
  }

Which should work until there is DNSSEC validation built into the subscriber's
DNS client.  

I believe this is effectively what Nicholas described in
http://www.ietf.org/mail-archive/web/dnsop/current/msg08339.html

-d

---

From: John Jason Brzozowski <[email-masked]>
Date: Wed, 31 Mar 2010 16:57:19 -0400

> :: is shipped by Microsoft.
[jjmb] this is not what the Free people have indicated.  I think this is a
flawed assumption.
> them without the user knowing about it?
[jjmb] Again from first hand experience, I can tell you there were
unexpected non-trivial increase in P2P over IPv6 traffic.
> :: server which functions normally.
[jjmb] Solvable perhaps, however, there are operational impacts that are
non-trivial.  Not to mention provisioning and in some cases financial
implications.
> :: because it doesn't do AAAA queries to find peers.
[jjmb] what percentage of broadband users fall into the advanced category
and will that be adequate to facilitate IPv6 adoption.  I suspect no and
this is not really an option in most cases for non-advanced users.
> to make this behavior a default, just a feature.

=========================================
John Jason Brzozowski
Comcast Cable
e) mailto:[email-masked]
o) 609-377-6594
m) 484-962-0060
w) http://www.comcast6.net
=========================================

---

From: "Dan Wing" <[email-masked]>
Date: Wed, 31 Mar 2010 14:12:43 -0700

> From: John Jason Brzozowski 
> [mailto:[email-masked]] 
> Sent: Wednesday, March 31, 2010 1:57 PM
> To: Igor Gashinsky; Dan Wing
> Cc: Andrew Sullivan; [email-masked]

Sorry, it appears I was not clear.

I will describe it another way.  There are two categories of ISP
subscribers:

    1. If subscriber is provisioned for IPv6, they are pointed at
       the ISP's DNS server which responds to AAAA normally --
       this is the ISP's "normal" nameserver.  All is well.  
       DNSSEC works, even if the validation is done by the client.
       No muss, no fuss.

    2. If subscriber is NOT provisioned for IPv6, they are
       pointed at the ISP's DNS server which responds to AAAA
       with an empty answer.  This helps with the transition 
       without losing eyeballs.  DNSSEC breaks if the client 
       queries AAAA and the client does DNSSEC validation.

       An advanced subscriber might be in this category (not
       provisioned for IPv6).  But that advanced user might want
       to query AAAA and get an answer.  That advanced user can 
       use the ISP's "normal" DNS server, or Google's, or 
       HE's, or opendns.org's, or whatever.  An advanced 
       subscriber might want to do that to *purposefully* 
       run Teredo, or to analyze a packet trace that 
       includes IPv6 traffic (and do DNS reverse queries 
       on the packet trace), get full results from the 'host'
       command, etc.

Clearer?

-d

---

From: Igor Gashinsky <[email-masked]>
Date: Wed, 31 Mar 2010 17:19:23 -0400 (EDT)

:: > On Wed, 31 Mar 2010, Dan Wing wrote:
:: > 
:: > :: Users running IE6 today are IPv4-only users.  If/when they go
:: > :: to IPv6, they will be running Windows 7 and whatever browser
:: > :: is shipped by Microsoft.
:: > 
:: > Why do you say that? As far as I know, IE6 is an ipv6-capable 
:: > browser, 
:: > as long as it's going to FQDN's.. So, what about IE6/XP users who 
:: > installed bittorent clients (or spyware/trojans) that enabled 
:: > ipv6 for them without the user knowing about it?
:: 
:: Yes, thanks for correcting me.
:: 
:: I agree they will have a poor experience (due to Teredo).
:: 
:: But Remi's point is that those same systems (running Windows XP
:: and IE6) using 6rd will be denied the ability to access content 
:: via IPv6.  Which removes an incentive for ISPs to add 6rd (and
:: offload the NAT44 they may soon have to install).

Yes, absolutely, and Remi's recursive DNS servers don't need to turn this 
feature on. Similarly, I, as a content provider don't need to whitelist 
his DNS recursive servers to receive AAAA if doing so would break too many 
users. However, if there are no broken IPv4 users behind Remi's recursors, 
then there is absolutely no need for him to use this feature, and I would 
be happy to give him AAAA replies when I'm ready to do so, and everybody 
is happy..

:: > :: It seems solvably operationally, by asking ISPs to point their
:: > :: IPv4-only subscribers at an ISP-operated DNS server which 
:: > :: purposefully breaks AAAA responses (returns empty answer), and 
:: > :: to point their dual-stack subscribers at an ISP-operated DNS 
:: > :: server which functions normally.
:: > :: 
:: > This is *exactly* what we are proposing -- the feature to 
:: > return empty 
:: > answers would be needed for ipv4-only subscribers in order to 
:: > keep them 
:: > ipv4-only. Also, if a fully ipv6-capable user visits that 
:: > person's home, 
:: > the recursor would then be able to make the call on if they 
:: > should pass 
:: > through AAAA to that particular user or not... I am by no 
:: > means advocating 
:: > to make this behavior a default, just a feature.
:: 
:: I'm saying this should be do-able entirely within the ISP's
:: DNS, without coordination or involvement with the content 
:: provider's DNS.  
:: 
:: For example, imagine the ISP's nameserver has a new "allow-aaaa-response"
:: option that would be configured like:
:: 
::   #
::   # list IPv4 addresses that are known to have real IPv6 connectivity.
::   # (e.g., this is all of the ISP's subscribers that are also 
::   #  connected using 6rd).
::   #
::   acl aaaa_whitelist { 172.16.72.0/24; 192.168.1.0/24; };
::   ...
::   
::   options {
::   ...
::   allow-aaaa-response { aaaa_whitelist };
::   ...
::   }

This is *exactly* what I'm proposing -- all my presentation does is 
document this as a feature and describes how it will work (the switch is 
actually "disable-aaaa-on-v4-transport")... The content provider's DNS may 
or may not then independently whitelist the ISP recursive servers that the 
users are behind, depending on breakage stats. If the breakage stats are 
sufficiently low for that content provider not to bother with the 
whitelist, so much the better -- the whole point of advocating for this 
feature is to allow us to get to that point.

So, I think the 2 of us are in complete agreement here.. the only question 
is if you do this for all users, a subset of them, or none at all, and 
that's going to be up to every individual ISP to do what is right for 
them.. I'm merely trying to document the behavior and functionality of 
this feature, and let people know that it will be available to them if 
they choose to use it to help with the breakage during the transition.

Thanks,
-igor

---

From: "Dan Wing" <[email-masked]>
Date: Wed, 31 Mar 2010 14:25:04 -0700

> From: Igor Gashinsky [mailto:[email-masked]] 
> Sent: Wednesday, March 31, 2010 2:19 PM
> To: Dan Wing
> Cc: [email-masked]; 'Andrew Sullivan'
> Subject: RE: [DNSOP] FYI: DNSOPS presentation
> 
> This is *exactly* what I'm proposing 

They're not quite the same.  Your proposal requires the IPv6-
capable hosts send their AAAA queries on IPv6.

-d

> -- all my presentation does is

---

From: Jason Livingood <[email-masked]>
Date: Wed, 31 Mar 2010 18:00:06 -0400

> non-trivial.  Not to mention provisioning and in some cases financial

+1

It would probably cost be far more money to roll out this separate DNS
server view, have folks monitor it and troubleshoot it, test and certify it
in the lab, etc. than just calling and fixing the "broken" users.

Jason

---

From: "Dan Wing" <[email-masked]>
Date: Wed, 31 Mar 2010 15:52:39 -0700

> > non-trivial.  Not to mention provisioning and in some cases 
> > financial implications.
> 
> +1
> 
> server view, have folks monitor it and troubleshoot it, test 
> and certify it in the lab, etc. than just calling and fixing 
> the "broken" users.

There is a way for the ISP to detect IPv6-broken users?  (Who can 
then be called and convinced to fix whatever is broken.)

-d

---

From: Jason Livingood <[email-masked]>
Date: Wed, 31 Mar 2010 19:30:35 -0400

> then be called and convinced to fix whatever is broken.)

Sounds as if there may be*, but I think that all depends upon what "broken"
means.  I'd love to have some discussion on that topic, so we can see if
people are all on the same page.

Igor - How do you define broken?  And what technical issues do you believe
underlie this condition?

* I am assuming there is and that an ISP could as readily host the web page
with the detection logic as Yahoo or Google could.  As well, if we could
make this widely available then presumably regulators could put it on their
sites, SDOs could, news sites could, etc., and there could be a general
awareness and detection campaign organized.  I suspect there may well be
other methods to detect the condition, which is why I think we should ensure
there is clear agreement and understanding of the underlying condition so we
can then find our way to detection.

Jason

---

From: John Jason Brzozowski <[email-masked]>
Date: Thu, 01 Apr 2010 00:18:43 -0400

> offload the NAT44 they may soon have to install).
[jjmb] this is true, this would be an issue for Free users that do not have
native IPv6.  Free users that have native IPv6 connectivity would be fine,
however.
>   allow-aaaa-response { aaaa_whitelist };
[jjmb] So this would require the authoritative DNS servers of the content
providers to globally be advertising AAAA RR rights?  What if IPv4 only
users are configured to use the same recursive name servers that IPv6
capable users are, we are back to points that have been made earlier related
to operational challenges, etc.  What you describe above also looks a lot
like what what proposed during DNSOPs with the exception that it is assumed
that AAAAs are being advertised globally, right?
> Which should work until there is DNSSEC validation built into the subscriber's
[jjmb] this would be problematic with DNSSEC, this has been acknowledged.

---

From: John Jason Brzozowski <[email-masked]>
Date: Thu, 01 Apr 2010 00:23:08 -0400

>        on the packet trace), get full results from the 'host'
[jjmb] I see how you categorize things, the clarification does not, however,
change some of my points.  Having advanced users (people like us) manually
configure their DNS servers to point to HE (for example) will pertain to a
small percentage of the overall Internet using population that must start
using IPv6 without special configurations.  The former assumes there is a
separate infrastructure in place with in the service providers which, as
mentioned earlier, has non-trivial challenges.

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 31 Mar 2010 23:26:53 -0700

> offload the NAT44 they may soon have to install).

I'm not sure this is true:
o the end-station sends a dns query to the ISP recursive resolver
o the recursive resolver uses whichever protocol is 'best' for the
lookup (assuming something like BIND's NS RTT caching happens in the
majority of cases)
o if the query goes over ipv6, is for a AAAA, a AAAA response could be returned
o if the query goes over ipv4, is for a AAAA, no response is returned
and presumably a follow-up A query happens.

Igor/Yahoo are proposing that recursive resolvers which have v6
transport use it and be rewarded for doing so... if they have a
longer/worse path over ipv6 there's a good chance the user experience
will also suffer, so the users should use v4.

-chris

-chris