Subject: Operational experience with 3 degrees

From: "Christian Huitema" <[email-masked]>
Date: Wed, 19 Mar 2003 17:55:55 -0800

Bob,
 
Due to timing, I could not present the scheduled paper "Operational experience with Microsoft IPv6 based P2P application ("threedegrees") - Huitema, 10 mins". So I guess I will instead send a short abstract of my presentation in this message. Here it is.
 
On 2/26/03 we started the beta trial of "three degrees" (http://www.threedegrees.com <http://www.threedegrees.com/> ), a P2P IPv6 based application. ThreeDegrees allows users to organized in Small groups of peers, exchange IM, photos and winks, i.e. on screen animations, as well as jointly listen to music. There have been several thousand download of the beta.

The peer to peer exchanges are performed only over IPv6; in places where native connectivity is not available, the hosts use either 6to4 if they have a global IPv4 address, or Teredo if they don't. This deployment is in effect the first beta trial of the Teredo technology; Microsoft is providing a Teredo server for use by the Three Degrees hosts.

The short summary is that "it works", and that many of the early users are very happy. IPv6 is just "instantly deployed", validating the use of 6to4 and Teredo. The quality of the service, be it RTT or bandwidth, is adequate. But there are issues: some web servers became unreachable, some native IPv6 labs complained that they could not use the application, users behind symmetric NAT were left out, and there is a lack of Teredo relays.

After downloading and installing the beta, some users found themselves without access to a few of their favorite web sites; there were wild speculations that Microsoft had somehow disabled access to the sites. Well, Microsoft indeed did not do anything like that; the disabling of some web sites was a side effect of enabling IPv6. The version of Internet Explorer on Windows XP is IPv6 enabled. If IPv6 enabled on machine, IE will call "getaddrinfo" which in conformance with the IETF recommendation will look for AAAA record for the web site, and then look for an A record if there is no AAAA available. The problem is that, upon receiving a query for a AAAA record, some DNS servers returned a "no such domain" error, upon which getaddrinfo gave up. The issue affects about 20 of the top 500 web sites. We are glad that we found it during a limited size beta rather than during a wide scale deployment. We are working with the vendors of the affected DNS products and are confident that the bug will soon be fixed.

The second issue affected some native IPv6 networks, typically test laboratories in which a tester would load the three degrees application, and then complain that "it does not work in my IPv6 lab". The symptom was that the application will refuse to start, because it could not initialize the P2P software. This initialization requires that the host contacts the P2P "seed server", in order to obtain the address of a few other peers and connect to the peer-to-peer "cloud". The "seed server" is located at a 6to4 address, and is unreachable if UDP traffic is blocked by an IPv6 firewall, or in absence of a route to 2002:/16. The problem can be fixed by loosening firewall rules, and by making sure that the native IPv6 network has good connectivity with 6to4 nodes.

The third issue was not really a surprise. In our deployment, the only way for users located behind NAT to get IPv6 connectivity is to use Teredo, and Teredo only works through about 90% of the NAT available on the market; the others are "symmetric" NAT. Since users buying NATs have no clue whether their NAT is symmetric or not, this results in complaints such as "the application does not work behind a NAT of brand X". After the feedback from the beta, we better know the size of the problem, and we are making sure that several solutions will be available, to be used as appropriate. The Teredo software will be updated to automatically reserve a UDP port using the UPNP protocol when the NAT support this protocol, or to use a user assigned port when the user is able to open a port in the NAT using a management interface. If that fails, the user will in some cases be able to get a new firmware for the existing NAT, without having to buy a new NAT. Indeed, another solution would be to have generous third parties provide tunnel servers...

The fourth issue concerns the lack of Teredo relays. In our early deployment, Microsoft only runs a Teredo server, and each peer tries to run a "host specific" Teredo relay - a piece of software that intercept traffic bound to Teredo addresses and send it over UDP. However, a peer located behind symmetric NAT or firewall does not have enough IPv4 connectivity to properly run a Teredo relay, which results in connection failures in the P2P application. The solution is obviously to deploy a Teredo relay in the site exit routers of the peer sites, or possibly in the ISP networks. Teredo relays are even simpler to implement than Teredo clients or servers, and can incorporate a number of "stateful" traffic control features that make them resistant to attacks. To facilitate this deployment, we should provide a very short Teredo relay specification.

-- Christian Huitema

---

From: Pekka Savola <[email-masked]>
Date: Thu, 20 Mar 2003 18:35:19 +0200 (EET)

Commeting on the last issues.

On Wed, 19 Mar 2003, Christian Huitema wrote:
> The third issue was not really a surprise. In our deployment, the only
> way for users located behind NAT to get IPv6 connectivity is to use
> Teredo, and Teredo only works through about 90% of the NAT available on
> the market; the others are "symmetric" NAT. Since users buying NATs have
> no clue whether their NAT is symmetric or not, this results in
> complaints such as "the application does not work behind a NAT of brand
> X". After the feedback from the beta, we better know the size of the
> problem, and we are making sure that several solutions will be
> available, to be used as appropriate. 

This has been one of my objections to Teredo: even if it works on more or 
less 90% of NAT boxes, and fails on the rest, the 10% is still a large 
amount of appliances, and..

> The Teredo software will be
> updated to automatically reserve a UDP port using the UPNP protocol when
> the NAT support this protocol, or to use a user assigned port when the
> user is able to open a port in the NAT using a management interface. If
> that fails, the user will in some cases be able to get a new firmware
> for the existing NAT, without having to buy a new NAT. 

.. such firmware would *actually* make the NAT more insecure.  You can
implement NAT's in "loose" or "strict" mode.  What you're advocating is
turning relatively secure NAT's to insecure ones.

(Btrw, I don't find it realistic to assume NAT's which have this issue 
will support UPNP, but YMMV.)

> Indeed, another
> solution would be to have generous third parties provide tunnel
> servers...

This seems similar to 6to4 relays; they are not there in abundance, and 
there seems to be little sign the trend will change any time soon.

What Teredo is doing is splitting the IPv6 address space in two: those who 
can use it, can communicate bidirectionally relatively nicely.   But the 
connectivity to the rest of the Internet is a pain.  That doesn't look me 
to as a desirable approach.

>  The solution is obviously to deploy a
> Teredo relay in the site exit routers of the peer sites, or possibly in
> the ISP networks. Teredo relays are even simpler to implement than
> Teredo clients or servers, and can incorporate a number of "stateful"
> traffic control features that make them resistant to attacks. To
> facilitate this deployment, we should provide a very short Teredo relay
> specification.

I have doubts about this actually happening.  Believe me, it's not about 
the code size.  6to4 relay implementation is less than 10 lines of code at 
th minimum.  Code size is a factor for those who implement it (and 
consequently those who want to deploy it), but *not* those who don't 
intend to.

To avoid fragmentation of IPv6 Internet and keeping things simple and
working for all NAT users, I'm strongly favoring "bidirectional tunneling
through NAT" approach.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 20 Mar 2003 23:19:10 +0100 (CET)

Thanks for sharing this information.

> If IPv6 enabled on machine, IE will call "getaddrinfo" which in conformance
> with the IETF recommendation will look for AAAA record for the web site, and
> then look for an A record if there is no AAAA available. The problem is
> that, upon receiving a query for a AAAA record, some DNS servers returned a
> "no such domain" error, upon which getaddrinfo gave up. The issue affects
> about 20 of the top 500 web sites. We are glad that we found it during a
> limited size beta rather than during a wide scale deployment. We are working
> with the vendors of the affected DNS products and are confident that the bug
> will soon be fixed.

Is this caused by load balancing boxes?
This is an issue for IPv6 deployment in general and not specific
to Teredo. If it would be useful
to have a document to point the vendors to, perhaps we can arm-twist somebody
to write a short document (could be done in dnsop or by a
willing individual I think?)

  Erik

---

From: "Jeroen Massar" <[email-masked]>
Date: Thu, 20 Mar 2003 23:55:28 +0100

> Thanks for sharing this information.

<SNIP>

> The problem is
> > that, upon receiving a query for a AAAA record, some DNS
> > servers returned a "no such domain" error, upon which
> > getaddrinfo gave up. The issue affects about 20 of the
> > top 500 web sites. We are glad that we found it during a
> > limited size beta rather than during a wide scale deployment.
> > We are working with the vendors of the affected DNS products
> > and are confident that the bug will soon be fixed.
> 

There was something related, or most probably the same problem
last year with the resolvers running on the bbc.co.uk domain and
some others, check here:

http://www.cctec.com/maillists/nanog/historical/0204/msg00559.html

"incorrect NXDOMAIN response from DNS server"

For the over 2 years now that I have been using an IPv6 enabled IE
I haven't come across that problem much times though, at least not
that I have noticed. But then again I am probably not a typical
top500 website crawler :)

Greets,
 Jeroen

---

From: "Jeroen Massar" <[email-masked]>
Date: Fri, 21 Mar 2003 00:10:24 +0100

Pekka Savola wrote:

> This has been one of my objections to Teredo: even if it 
> works on more or less 90% of NAT boxes, and fails on the rest,
> the 10% is still a large amount of appliances, and..

Can someone enlighten me with the reasons why there is no defacto
IPv6-over-IPv4-TCP standard. Or is it too common to use a tinc/vtund
tunnel to establish those connections. I have been thinking about
implementing such a scheme for tunneling to the SixXS systems.
Thus allowing connections to be made from behind most firewalls
especially if we combinate it with the fact that we could use
port 143 to connect to, or even better use a real ssl tunnel
or httpproxy to get across it. In most ways I think that a TCP
connection has more chance of crossing in&out a NAT and even
a firewall than any UDP packet.

Ofcourse the big 'disadvantage' here is that it requires
configured tunnels.

PS: I've even take some of my spare time to fix myself up with
a /dev/tun for Win2k/XP which acts like a network card, using
a small netcat alike tool it simply acts like a native IPv6
link, without IPv4 over it :) Though it is experimental and
really needs to be implemented correctly using tinc/vtund or
a similar tool. (And yes I was annoyed by the fact that I
couldn't do IPv6 from behind some firewalled places.
And as I have some handy boxes that are only accessible
over v6 I do find it quite handy to be able to access them
from everywhere ;)

Greets,
 Jeroen

---

From: JINMEI Tatuya / 神明達哉 <[email-masked]>
Date: Sun, 23 Mar 2003 11:37:19 +0900

>>>>> On Thu, 20 Mar 2003 23:19:10 +0100 (CET), 
>>>>> Erik Nordmark <[email-masked]> said:

> willing individual I think?)

I agree.  As far as I know, there are several "broken" DNS servers (or
load balancers) wrt AAAA queries:

- some of them return NXDOMAIN against AAAA (as reported in this thread).
- some ignore AAAA queries (causing a delay to fallback).
  you can see this when you try to resolve AAAA for "ftp.mozilla.org"
  at 64.12.182.70 (gdns-02-ftp-dom.ns.aol.com)
- some return broken answers for AAAA queries.  Some resolvers
  handle such cases as "server fail," and will not use the servers).
  you can see this by asking an AAAA RR for "www.gslb.mainichi.co.jp"
  at 210.173.172.2.
- some constantly indicates a (loop) delegation for any queries except
  A.  Resolvers interpret this as a lame delegation.  You can see this
  by asking AAAA for "www.united.com" at 64.95.89.4.

I guess there may be a comprehensive document already, but I myself
don't know one.

					JINMEI, Tatuya
					Communication Platform Lab.
					Corporate R&D Center, Toshiba Corp.
					[email-masked]

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 24 Mar 2003 11:21:24 +0100 (CET)

>   at 64.12.182.70 (gdns-02-ftp-dom.ns.aol.com)

And those boxes do not ignore A queries for the same name?
The reason I'm asking is because I think there are DNS servers which
silently ignore queries for names that they don't have any information
(instead of sending a referral).
If they treat A and AAAA differently that would be news to me.

> I guess there may be a comprehensive document already, but I myself

It seems useful to take your list (and other issues folks report)
and put in an I-D as a starting point.

  Erik

---

From: JINMEI Tatuya / 神明達哉 <[email-masked]>
Date: Thu, 27 Mar 2003 01:13:07 +0900

>>>>> On Mon, 24 Mar 2003 11:21:24 +0100 (CET), 

> and put in an I-D as a starting point.

(I'll first answer the second point) okay, if no one else has a
comprehensive list, I'll try to make one.  (I'm not sure if an I-D is
the best place to describe this kind of topic, but an I-D is probably
not a bad one.)

BTW: we may need to raise this issue in a place where DNS experts are.
Perhaps dnsop?

> And those boxes do not ignore A queries for the same name?

No.  The boxes only ignore AAAA (actually non-A) queries, but process
A queries correctly.

> If they treat A and AAAA differently that would be news to me.

The following is a more detailed description.  

If you try to resolve an RR for ftp.mozilla.org, you'll follow a CNAME
chain and will eventually see the name is in fact an alias for
ftp-mozilla.gftp-mozilla.netscape.com.

Also, you'll see 64.12.182.70 (gdns-02-ftp-dom.ns.aol.com) is an NS
for the gftp-mozilla.netscape.com zone.  Now we're ready to do some
experiments.

The server answers a query for an A RR:

% dig a +norec +short @64.12.182.70 ftp-mozilla.gftp-mozilla.netscape.com.
64.12.168.243

but (apparently) ignore a query for a AAAA RR:

% dig aaaa +norec +short @64.12.182.70 ftp-mozilla.gftp-mozilla.netscape.com.
;; connection timed out; no servers could be reached

or for an MX RR:

% dig mx +norec +short @64.12.182.70 ftp-mozilla.gftp-mozilla.netscape.com.
;; connection timed out; no servers could be reached

or for an A6 RR:

% dig a6 +norec +short @64.12.182.70 ftp-mozilla.gftp-mozilla.netscape.com.
;; connection timed out; no servers could be reached

					JINMEI, Tatuya

---

From: "Jeroen Massar" <[email-masked]>
Date: Wed, 26 Mar 2003 19:59:33 +0100

JINMEI Tatuya wrote:

<SNIP>

> BTW: we may need to raise this issue in a place where DNS experts are.

That might be quite a good place indeed as it is mostly DNS orientated.
Subject change would be nice too :)

<SNIP>

From RFC2181:
8<-----------------------------
10.1. CNAME resource records

   The DNS CNAME ("canonical name") record exists to provide the
   canonical name associated with an alias name.  There may be only one
   such canonical name for any one alias. 
----------------------------->8

I interpret this as "CNAME chains are not allowed" thus the above
system should not work at all... apparently in some cases it works
though.
I wonder why this does work for A records ;)

Greets,
 Jeroen

---

From: JINMEI Tatuya / 神明達哉 <[email-masked]>
Date: Thu, 27 Mar 2003 13:06:06 +0900

>>>>> On Wed, 26 Mar 2003 19:59:33 +0100, 
>>>>> "Jeroen Massar" <[email-masked]> said:

>> >> at 64.12.182.70 (gdns-02-ftp-dom.ns.aol.com)

(snip)

>    such canonical name for any one alias. 
-----------------------------> 8

> system should not work at all... apparently in some cases it works

First of all, "CNAME chains" are not the essential part of this
problem.  The point is that we cannot resolve a non-A RR for
gftp-mozilla.netscape.com zone at 64.12.182.70, to which CNAMEs are
not related at all.

Secondly, Section 10.1 of RFC 2181 talks about so-called "multiple
CNAMEs", which is a different notion from what I called "a CNAME
chain".  Multiple CNAMEs are multiple different CNAMEs for a single
domain name in a single zone, like:

www.jinmei.org. IN CNAME www1.jinmei.org.
www.jinmei.org. IN CNAME www2.jinmei.org.

While I used the word a "CNAME chain" like this:

foo.jinmei.org. IN CNAME bar.jinmei.org.
bar.jinmei.org. IN CNAME baz.jinmei.org.

					JINMEI, Tatuya

---

From: "Jeroen Massar" <[email-masked]>
Date: Thu, 27 Mar 2003 12:53:44 +0100

<SNIP>

> domain name in a single zone, like:

That's indeed more logical than my interpretation ;)

Greets,
 Jeroen

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 27 Mar 2003 16:27:10 +0100 (CET)

> From RFC2181:

My interpretation of the above text is that you can only have one CNAME
record for a given name; not that there can not be chains.

   Erik

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 27 Mar 2003 16:25:19 +0100 (CET)

> BTW: we may need to raise this issue in a place where DNS experts are.

Yes, once we have the list it would make sense to present it to dnsop.
Your email is a good starting point but the issues might need to be fleshed out
a bit more.

If there is a need to produce a document pointing out existing bad
practises we need to discuss where that can be best made - it definitely
needs to be done with dns clue involved.

   Erik