Subject: Re: FYI: DNSOPS presentation

From: Brian E Carpenter <[email-masked]>
Date: Sat, 03 Apr 2010 04:16:28 +1300

As every successful IPv6 user on Windows XP knows, the hack
would break all Windows XP IPv6 usage (since it can't resolve
DNS over IPv6 at all).

I prefer the current situation, where occasionally I have to
disable IPv6 manually due to being on an IPv6-broken network
where DNS serves up AAAA records and my regular tunnel is too
slow. This seems OK, since people using v6 on XP are normally doing
so as conscious early adopters.

I do think that Google's solution to this, i.e. being selective
about who gets the AAAA records in the first place, is much less
of hack with less harmful side effects.

Surely a better hack would be for recursive resolvers in IPv6-broken
networks not to serve up AAAA records at all? Tunnel users could
always find another resolver.

Regards
   Brian

---

From: Rémi Després <[email-masked]>
Date: Fri, 2 Apr 2010 18:09:53 +0200

Le 2 avr. 2010 à 17:16, Brian E Carpenter a écrit :

> where DNS serves up AAAA records

So do I.

> of hack with less harmful side effects.

+1

> networks not to serve up AAAA records at all?

Agreed.

RD

---

From: [email-masked]
Date: Fri, 2 Apr 2010 16:55:52 +0000

> > networks not to serve up AAAA records at all?

	aren't you making the assumption that the recursive resolver is
	on/in the same broadcast domain as the stub which it is serving?
	how can you ensure that the IP reachability of the stub is "semantically
	simialar"** to the IP reachability of the recursive resolver?

	the DNS has no hooks into topological reachability. 

** identical, equivalent, etc... See IDN equivalence

--bill

---

From: "Perkins, Carroll G" <[email-masked]>
Date: Fri, 2 Apr 2010 13:00:18 -0400

Several thousand hours of IPv6 usage on XP and Windows 7 prompts me to wholly agree with Brian.
The way DNS works now fits 95% plus of the transition environments.
The proposed DNS hack will be obsolete before it can be passed thru the IETF anyway since IPv6 usage is growing exponentially during 2010.
Comcast alone will account for 50% of the transition market during this calendar year.  And they have hard dates for completing IPv6 implementation during 2010.
So don't break DNS for the 95% using it successfully today for the 5% that are not working today without manual intervention.
Carroll Perkins

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Brian E Carpenter [[email-masked]]
Sent: Friday, April 02, 2010 11:16 AM
To: Ed Jankiewicz
Cc: IPv6 Operations; 6man
Subject: Re: FYI:  DNSOPS presentation

Regards
   Brian
--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]
Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------

---

From: Mark Townsley <[email-masked]>
Date: Fri, 02 Apr 2010 19:21:36 +0200

> would break all Windows XP IPv6 usage (since it can't resolve

Effectively all MACs as well, as the only way to get DNS resolved over
v6 is by manually configuring the IPv6 DNS Server.

- - Mark

---

From: james woodyatt <[email-masked]>
Date: Fri, 2 Apr 2010 11:11:03 -0700

On Apr 2, 2010, at 08:16, Brian E Carpenter wrote:
> of a hack, with less harmful side effects.

While everyone else was in the 6MAN session, I was actually jumping back and forth between 6MAN and DNSOP because I didn't want to miss this presentation.  As I pointed out at the microphone in DNSOP, the proposed hack is absolutely broken because it relies on a pair of TOTALLY INOPERATIVE ASSUMPTIONS:

  A) good IPv6 connectivity between the DNS resolving server and the DNS content server implies good IPv6 connectivity between the client application host and the application server host.

  B) good IPv6 connectivity between the client application host and the application server host requires good IPv6 connectivity between the DNS resolving server and the DNS content server.

The obvious illustration that comes to mind is a host behind an Apple AirPort Extreme base station at a public IPv4 address and configured in 6to4 tunnel mode, but in a domain where the 6to4 relay service is unreliable, badly-impaired or black-holed, and also configured to forward its DNS queries to resolving servers that are dual-stack IPv4/IPv6 capable.  This configuration is sufficiently common that any proposed solution to the problem must have an answer for it, and the proposed solution doesn't deal with it at all.

Shorter james: the proposed hack cannot solve the problem it intends to solve, and it breaks other networks that work fine today.  It should not be endorsed.

--
james woodyatt <[email-masked]>
member of technical staff, communications engineering

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 03 Apr 2010 07:45:44 +1300

Bill,

> 	the DNS has no hooks into topological reachability. 

Correct, but a hack in this area by definition can't cover all cases.

Er, much better to mend the IPv6 brokenness, of course.

   Brian

---

From: james woodyatt <[email-masked]>
Date: Fri, 2 Apr 2010 12:26:18 -0700

On Apr 2, 2010, at 12:05, Joel Jaeggli wrote:
> 
> actually they don't rest on those assumptions.

Actually, I spoke at length with Igor Gashinsky on Tuesday night at IETF 77 before the presentation in DNSOP, and I'm pretty sure I accurately characterized the assumptions behind his proposal.

Where he and I disagree, I think, is on the question of whether the assumptions are *operative*.  He believes they are; I believe they are not.  It's *his* train set, though, so I suppose he can break it however he wants.

---

From: [email-masked]
Date: Fri, 2 Apr 2010 20:35:08 +0000

> Correct, but a hack in this area by definition can't cover all cases.

	does it even cover a statistically significant fraction of cases?
> Er, much better to mend the IPv6 brokenness, of course.

	then why not spend the energy/resouce on ensuring v6 transport
	works instead of wasting this on what is clearly a DNS haq for 
	a temporary condition?  Or is this a case of "when all we have is
	the DNS, everything looks like a query/response"?

---

From: Igor Gashinsky <[email-masked]>
Date: Fri, 2 Apr 2010 16:50:19 -0400 (EDT)

On Fri, 2 Apr 2010, james woodyatt wrote:

:: On Apr 2, 2010, at 08:16, Brian E Carpenter wrote:
:: > 
:: > I do think that Google's solution to this, i.e. being selective
:: > about who gets the AAAA records in the first place, is much less
:: > of a hack, with less harmful side effects.
:: 
:: While everyone else was in the 6MAN session, I was actually jumping back and forth between 6MAN and DNSOP because I didn't want to miss this presentation.  As I pointed out at the microphone in DNSOP, the proposed hack is absolutely broken because it relies on a pair of TOTALLY INOPERATIVE ASSUMPTIONS:
:: 
::   A) good IPv6 connectivity between the DNS resolving server and the DNS content server implies good IPv6 connectivity between the client application host and the application server host.
:: 
::   B) good IPv6 connectivity between the client application host and the application server host requires good IPv6 connectivity between the DNS resolving server and the DNS content server.

Actually, it does not rely on these assumptions. It does however rely on 
the following:

1) Good connectivity between the DNS resolving server and the DNS content 
server is easily quantifiable by both the service and content providers 
via a variety of techniques

*PLUS* 

2) good connectivity between client application host and DNS resolving 
server can be identified by the ISP recursive server being able to get 
DNS queries over IPv6 transport

Those 2 combined are an excellent indicator of good connectivity between 
the client application host and the application server host. Ie, if the 
user can get to his dns server of ipv6, and the content provider 
determines that ipv6 connectivity between their network and the 
ISP network works correctly (which we can do -- this is what whitelisting 
is all about), then we are reasonably sure that the end-user will be able 
to get to the content service they want to get to over ipv6, w/o breaking 
anything.

:: The obvious illustration that comes to mind is a host behind an Apple AirPort Extreme base station at a public IPv4 address and configured in 6to4 tunnel mode, but in a domain where the 6to4 relay service is unreliable, badly-impaired or black-holed, and also configured to forward its DNS queries to resolving servers that are dual-stack IPv4/IPv6 capable.  This configuration is sufficiently common that any proposed solution to the problem must have an answer for it, and the proposed solution doesn't deal with it at all.

Actually, I think it can:

If the end user's 6to4 tunnel is "flakey", then the ipv6-transported DNS 
queries over said tunnel would fail, causing the user to try to resolve 
them over ipv4 transport, (I'm assuming the user would get both the v4 and 
the v6 ips of the dualstack recursors in this example), which should 
work just fine. This would then cause the recursive servers running this 
"hack" to only reply back with A records (since the query comes in over 
ipv4), therefore the client would work just fine. No?

-igor

---

From: Igor Gashinsky <[email-masked]>
Date: Fri, 2 Apr 2010 16:51:44 -0400 (EDT)

:: -----BEGIN PGP SIGNED MESSAGE-----
:: Hash: SHA1
:: 
:: On 4/2/10 5:16 PM, Brian E Carpenter wrote:
:: > As every successful IPv6 user on Windows XP knows, the hack
:: > would break all Windows XP IPv6 usage (since it can't resolve
:: > DNS over IPv6 at all).

Yes, this was one of the identified caveats.

On Fri, 2 Apr 2010, Mark Townsley wrote:
:: Effectively all MACs as well, as the only way to get DNS resolved over
:: v6 is by manually configuring the IPv6 DNS Server.

Until the next release, when this, I believe will be fixed.

-igor

---

From: Igor Gashinsky <[email-masked]>
Date: Fri, 2 Apr 2010 16:57:51 -0400 (EDT)

On Sat, 3 Apr 2010, Brian E Carpenter wrote:

:: I prefer the current situation, where occasionally I have to
:: disable IPv6 manually due to being on an IPv6-broken network
:: where DNS serves up AAAA records and my regular tunnel is too
:: slow. This seems OK, since people using v6 on XP are normally doing
:: so as conscious early adopters.

A few months ago I woudl have agreed with you -- anybody who configures 
IPv6 on windows XP woudl be concidered a "power user", and therefore can 
suppot and troubleshoot themselves. 

Unfortunately, a few months ago I found out that one of the torrent 
clients is enabling IPv6 on XP machine without users even knowing about 
it. Now, the game has changed, because it's not only powerusers who have 
ipv6 configured, it's everyone, and I can no longer rely on them to fix 
themselves, so, now I need to find a way to not break those users when I 
hand out AAAA without having them be involved.

:: I do think that Google's solution to this, i.e. being selective
:: about who gets the AAAA records in the first place, is much less
:: of hack with less harmful side effects.
:: 
:: Surely a better hack would be for recursive resolvers in IPv6-broken
:: networks not to serve up AAAA records at all? Tunnel users could
:: always find another resolver.

It is the same hack. You would use that very same feature that was 
developed to accomplish this (filter-aaaa) -- what's more, when your 
network stops being v6-broken, you get the extra benefit that the users 
who can query your dns resolvers over ipv6 can now get AAAA's as first 
step of rollout...

-igor

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 03 Apr 2010 10:30:10 +1300

On 2010-04-03 09:57, Igor Gashinsky wrote:
...
> It is the same hack. 

Not if applied administratively, and reversed administratively
when the v6 brokenness is fixed. From a network ops point of view,
this is much cleaner than automated detection, and will also
put the responsibility in the right place: the network manager
who has not deployed v6.

And early adopters who want to bypass the hack can simply
configure a DNS server that does deliver AAAA.

    Brian

    Brian

You would use that very same feature that was

---

From: Joel Jaeggli <[email-masked]>
Date: Fri, 02 Apr 2010 12:05:00 -0700

On 04/02/2010 11:11 AM, james woodyatt wrote:
>> about who gets the AAAA records in the first place, is much less of
>> a hack, with less harmful side effects.
> 
> While everyone else was in the 6MAN session, I was actually jumping
> back and forth between 6MAN and DNSOP because I didn't want to miss
> this presentation.  As I pointed out at the microphone in DNSOP, the
> proposed hack is absolutely broken because it relies on a pair of
> TOTALLY INOPERATIVE ASSUMPTIONS:
> 
> A) good IPv6 connectivity between the DNS resolving server and the
> DNS content server implies good IPv6 connectivity between the client
> application host and the application server host.
>
> B) good IPv6 connectivity between the client application host and the
> application server host requires good IPv6 connectivity between the
> DNS resolving server and the DNS content server.

the assumption being made is that queriers who are customers of a given
set of name-servers have available to them a particular kind of
connectivity...

this is exactly the same kind of assumption made when selectively
returning different A records in an ipv4 global loading balancing setup.

The consequences of getting it wrong in ipv4 are you go to a more
distant instance of a service.

The consequence of getting it wrong in ipv6 are you hand a AAAA record
to a host that doesn't use it (this causes no harm) or you hand a AAAA
record to host that shouldn't use it (because soemthing about their
connectivity is broken). presumably the decision to make a particular
set of queriers as blessed for the purpose of reciveing AAAA records is
based on a judgement about the likelyhood of the hosts behind that
resolver being non-broken.

> The obvious illustration that comes to mind is a host behind an Apple
> AirPort Extreme base station at a public IPv4 address and configured
> in 6to4 tunnel mode, but in a domain where the 6to4 relay service is
> unreliable, badly-impaired or black-holed, and also configured to
> forward its DNS queries to resolving servers that are dual-stack
> IPv4/IPv6 capable.  This configuration is sufficiently common that
> any proposed solution to the problem must have an answer for it, and
> the proposed solution doesn't deal with it at all.
> 
> Shorter james: the proposed hack cannot solve the problem it intends
> to solve, and it breaks other networks that work fine today.  It
> should not be endorsed.
> 
> 
> -- james woodyatt <[email-masked]> member of technical staff,
> communications engineering
> 
> 
>

---

From: Jason Livingood <[email-masked]>
Date: Fri, 02 Apr 2010 15:17:09 -0400

I agree with what James just posted.

Since this has hopped over to v6ops from dnsops, I thought I'd post here
what I wrote over there, as I think it's important as we're working on
solutions w/o a clear agreement on the problem:

It seems that have the cart before the horse, so to speak.  IMHO, we need to
do the following (and there's no reason they cannot occur rapidly):

1 - Develop a clear problem statement that outlines (1) how "broken" users
are defined and (2) what effects this "brokenness" has on these end users
(or other parts of the Internet).

2 - Describe all the various methods and tactics by which end user
"brokenness" can be detected.  This may include website-based detection,
DNS-query-based detection, or a variety of other methods.

3 - Then, after we have agreement on Problem Definition and Problem
Detection, we can measure the problem to understand what the scope or scale
of the problem is. 

4 - After we understand Problem Definition, Problem Detection, and Problem
Scope, then you can arrive at possible solutions.  Seems like in this case
we sort of *started* here, which concerns me and I think we need to be
careful that discussion thus far does not constrain development of a full
list of Solution Options.  I further believe we will need to encourage the
pursuit of multiple solutions simultaneously.  Lastly, just because end user
software upgrades may be difficult doesn't mean we shouldn't do them and
shouldn't focus just as much energy on those than other options.

Jason

---

From: Olivier MJ Crepin-Leblond <[email-masked]>
Date: Sat, 03 Apr 2010 11:01:54 +0200

On 02/04/2010 21:17, Jason Livingood wrote :
> Scope, then you can arrive at possible solutions.

Completely in agreement, although I'd like to add "...and whether such
solutions are required."

I feel uneasy messing around with the DNS by introducing "ugly hacks" to
resolve a problem with "0.078% of people", thus affecting 99.922% of
people who had no problem and who might then end up having a problem due
to the hack.
Agreed, 470K people is no small number, but we have to look at the
bigger picture with 600M people. If we start designing Internet techno
to look at the lowest common denominator, we're in for a very bad ride.

Warmest regards,

Olivier

-- 
Olivier MJ Crépin-Leblond, PhD
http://www.gih.com/ocl.html

---

From: Rémi Després <[email-masked]>
Date: Sat, 3 Apr 2010 15:45:35 +0200

> Shorter james: the proposed hack cannot solve the problem it intends to solve, and it breaks other networks that work fine today.  It should not be endorsed.

+1
An approach that would break what works today in IPv6 MUST be rejected.

It remains that a clear description of which problem is encountered, and by which organization, might help to find a acceptable solution.

RD

---

From: Tore Anderson <[email-masked]>
Date: Sat, 03 Apr 2010 17:11:44 +0200

Hi,

* Olivier MJ Crepin-Leblond

> Completely in agreement, although I'd like to add "...and whether 
> such solutions are required."

Currently, the number of end users who can connect to a single-stack
IPv4 web site is greater than the number of end users who can connect
to a dual-stack web site.  This situation must change, or you won't be
seeing much IPv6 content appearing on the internet anytime soon.

> I feel uneasy messing around with the DNS by introducing "ugly hacks"
> to resolve a problem with "0.078% of people", thus affecting 99.922%
> of people who had no problem and who might then end up having a
> problem due to the hack.

The obvious "solution" to this is currently to _not_ deploy IPv6 on
existing content.  For now, that will keep ~100% of users happy, and
doesn't require any DNS hacks/whitelists to be maintained.

But I still would like to deploy IPv6 somehow.  Suggestions on how
exactly I can go about doing that, without messing around with DNS or
breaking connectivity for users who up until now have had no problems,
would be greatly appreciated.

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

---

From: Mark Andrews <[email-masked]>
Date: Sun, 04 Apr 2010 02:44:06 +1100

> would be greatly appreciated.

You turn IPv6 on a service at a time.  Turn it on for delay tolerent
services.  e.g. SMTP.  Turn it of for services which try different
servers quickly.  e.g. DNS.  Turn it on for the client side.  There
may be few place to connect to now but the ratio of good to bad
will get better as more people turn up IPv6 as a production service
on the client side.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Tore Anderson <[email-masked]>
Date: Sat, 03 Apr 2010 20:06:27 +0200

Hi Mark,

* Mark Andrews

> servers quickly.  e.g. DNS.  Turn it on for the client side.

Sure thing, I've already done some of those, but it won't really change
much.  There's no real reason for a end-user to contact my SMTP or DNS
servers directly, as he will normally relay e-mail/DNS queries via his
own ISPs servers anyway.  And the fact that all the workstations at my
office have native IPv6 connectivity doesn't have any effect on all the
broken end users in other networks.

The problematic service is without question the web sites.

> There may be few place to connect to now but the ratio of good to
> bad will get better as more people turn up IPv6 as a production
> service on the client side.

Absolutely.  I'm sure that at some point in the future there'll be more
end users able to connect to a dualstacked web site than there is end
users able to connect to a IPv4-only site.  At that point you'll see
lots of IPv6 content showing up.  No problem.

However, when will that point in time be?  The way it's going right now
it seems to me that it will be well after IPv4 pool is exhausted, when
the IPv4 internet is starting really to crumble under CGNs and other
life-extending band-aids.  I'm more impatient than that though, and
don't want to just sit around and wait until then.  It appears other
content providers like Yahoo and Google don't either.  But short of
resorting to DNS tricks, or attempting to persuade all the end users'
ISPs and/or software vendors to fix their problems, I really don't see
what else can be done.

By the way I actively pursue the option of asking problematic networks
and software vendors to fix their problems.  It's a very slow process,
though.  Even when software patches are released, it takes lots of time
before a sufficient amount of end-users have applied them.

Best regards,

---

From: Mark Andrews <[email-masked]>
Date: Sun, 04 Apr 2010 04:30:05 +1000

> broken end users in other networks.

Except it does, people saying we have IPv6 connectivity and it
didn't break things will cause more people to turn on IPv6 as a
production service which will disable the automatic transition
mechanisms.
 
> However, when will that point in time be?

Well we've had dual stack servers for years now so for us that point
has long ago passed.  We log bug reports when a service is not
available over IPv6 as well as IPv4.

> The way it's going right now

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 04 Apr 2010 11:37:09 +1200

On 2010-04-04 06:06, Tore Anderson wrote:
...
> before a sufficient amount of end-users have applied them.

We used to do a lot of that about 20 years ago when IPv4 connectivity
and applications support was patchy. The Internet only works today because
the message eventually got through (or the providers and vendors who
didn't get the message vanished).

There's no short cut.

  Brian

---

From: Mark Andrews <[email-masked]>
Date: Mon, 05 Apr 2010 13:44:13 +1000

> would be greatly appreciated.

The first thing is to publish a complete description of the testing
methodology and envionment.  So far I havn't see it.  Most of the
transition technology involves encapsulation so was the mtu set to
prevent pmtud being triggered due to encapsulation in IPv4?

	route add net 2002::/16 mtu 1480

or 

	route add net 2002::/16 mtu 1280

What are the figures when you do this?  Does either of these change the
percentages?

Similarly for

	route change net ::/0 mtu 1480
and

	route change net ::/0 mtu 1280

Mark

---

From: Tore Anderson <[email-masked]>
Date: Mon, 05 Apr 2010 11:45:50 +0200

Hi Mark,

* Mark Andrews

> methodology and envionment.  So far I havn't see it.

I've posted descriptions a couple of times before:

http://lists.cluenet.de/pipermail/ipv6-ops/2009-November/002658.html
http://www.ietf.org/mail-archive/web/dnsop/current/msg08406.html

Have a look, and don't hesitate to let me know if you're still wondering
about something.

> Most of the transition technology involves encapsulation so was the
> mtu set to prevent pmtud being triggered due to encapsulation in
> IPv4?

I haven't set the MTU explicitly, but the objects that are being served
are (intentionally) very small, so I never transmit any packets that are
anywhere near 1280 octets.  However I see that I occasionally receive
incoming packets that exceed 1280 (HTTP requests with large amounts of
header information).  I will try to reduce the advertised TCP MSS to
prevent PMTUD in the inbound direction from ever becoming an issue.
These jumbo requests occur very seldom though, so I'll be very surprised
if the lowered MSS will make a noticeable impact on the measured client
loss.

Thank you for your feedback!

BR,

---

From: Gert Doering <[email-masked]>
Date: Mon, 5 Apr 2010 12:29:48 +0200

Hi,

> Shorter james: the proposed hack cannot solve the problem it intends to solve, and it breaks other networks that work fine today.  It should not be endorsed.

Seconded.

(I can't remember how many times I have written in the last 10 years that
the transport protocol used for DNS queries coming from recursor servers
has nothing whatsoever to do with the transport protocol capabilities of
the client sending the DNS query - even without any caching involved)

Gert Doering
        -- NetMaster
-- 
Total number of prefixes smaller than registry allocations:  150584

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (89) 32356-444            USt-IdNr.: DE813185279

---

From: Olivier MJ Crepin-Leblond <[email-masked]>
Date: Mon, 05 Apr 2010 18:20:35 +0200

> will get better as more people turn up IPv6 as a production service

This works exactly in line with what I have been suggesting my corporate
clients should do when they are looking at implementing IPv6.
A copy of my presentation slides can be found on:
http://www.slideshare.net/ocl999/suggestion-for-an-ipv6-roll-out

Turning the delay tolerant services on, migrating such services to dual
stack, enables personnel to gain real IPv6 traffic & admin experience
with minimal potential disruption - at low cost.
I realise that I am digressing from the matter in the subject, so I'll
stop here.

Kindest regards,

Olivier