Subject: Re: draft-ietf-nat-protocol-complications-02.txt

From: Leonid Yegoshin <[email-masked]>
Date: Mon, 24 Apr 2000 20:50:23 -0700

>From: Keith Moore <[email-masked]>
>
>> Or seperate the end system identifer from the routing goop.  This
>> solves lots of problems (while introducing others).
>
>even if you do this the end system identifier needs to be globally
>scoped, and you need to be able to use the end system identifier
>from anywhere in the net, as a means to reach that end system.
>
  DNS is a bright and successfull example of such deal.

  It could be better to use domain names as system IDs but today
there is too much software which assume IP as system ID
(network ID of system in real).

>otherwise you're in about the same shape as with a NATted network.

		       - Leonid Yegoshin.

---

From: Keith Moore <[email-masked]>
Date: Tue, 25 Apr 2000 09:57:45 -0400

>   DNS is a bright and successfull example of such deal.

actually, DNS is slow, unreliable, and often out of sync with reality.

DNS reverse lookup tables (PTR) are not as well maintained as forward 
lookup tables (A) so they're even less reliable.

hosts often don't know their own DNS names, so they wouldn't know
their connection endpoint names either.

DNS names are often ambiguous - because a single DNS name corresponds
to multiple hosts (all implementing the same service) or because a
single host supports multiple DNS domains (a different name for each
service) or both.

the binding between a DNS name and an address is not the same thing
as the binding between an address and a connection endpoint.  the
two have different purposes and different lifetimes.

when people say "DNS can do the job" they may be saying different
things, e.g.
(a) they are thinking in terms of using existing DNS servers, 
(b) they are thinking in terms of using the DNS protocol, having 
translation occur at the boundaries between routing goop realms 
(similar to NAT's DNS ALG), or
(c) they are thinking in terms of a DNS-like system, but not DNS

with separate servers (whether they are existing DNS servers or not)
there is the problem of keeping the servers updated as to the 
current condition of the network.

with DNS at translation boundaraies there is the problem of 
"call setup overhead" (having queries propagate through 
multiple layers of translation until they reach their destination
network) also in this case DNS becomes a routing protocol of 
sorts, since the thing you advertise from one realm to another
becomes a DNS suffix rather than than an address prefix.
it's not at all clear that this scales.  

in either case having to do a DNS-like query before you can transmit
is slow compared to just sending a packet to an IP address.

Keith

p.s. if there's ever going to be a split between endpoint names and
routing goop, I'm convinced that endpoint names have to be usable 
by themselves (perhaps with some speed penalty), that the 
mapping between endpoint names and routing goop needs to be maintained
by the routing infrastructure rather than in some separate database, 
and that the lookup needs to be able to be done implicitly (as a side 
effect of sending packets without routing goop to their destination) 
rather than explicitly.  I think such a separation might be a good idea, 
because our current means of propagating reachability information
and computing routes does have limitations.  but I don't see any need 
to change the packet format seen by hosts (from IPv6), or any need to
change end hosts at all, in order to do this.

---

From: Bill Manning <[email-masked]>
Date: Tue, 25 Apr 2000 08:18:20 -0700 (PDT)

% > >even if you do this the end system identifier needs to be globally
% > >scoped, and you need to be able to use the end system identifier
% > >from anywhere in the net, as a means to reach that end system.
% > 
% >   DNS is a bright and successfull example of such deal.
% 
% actually, DNS is slow, unreliable, and often out of sync with reality.
% 
% DNS reverse lookup tables (PTR) are not as well maintained as forward 
% lookup tables (A) so they're even less reliable.

	This is an assertion that I've heard over the years
	and I've come to beleive (based on regular audits of
	the in-addr space) that this is an Internet equivalent
	of an urban legend.  I'd really like to see your backing 
	data on this.

	The 2q2000 data for the in-addr tree shows 77402 unique
	servers answering for 693,337 zones.  
	19515 servers blocked/refused data. Of the 57887 that 
	answered, these are the numbers for improper configuration:

	BAD_SERVER:	4278
	FORMERR:	8
	NXDOMAIN:	28

	So, of the 57,887 visable servers, 4314 are improperly configured 
	in the visable in-addr.arpa. tree.  Thats 7.45% of the
	servers being "not well maintained".  I know of no similar data
	collected on the forward tree as a whole. I'm currently checking
	the data in a few TLDs to see if spot data may indicate a trend.

	Now as to the accuracy of the data in the zones, that depends
	on the owner of the data beleiving the data correct and that
	will be very hard to check.

--bill

---

From: [email-masked]
Date: Tue, 25 Apr 2000 12:41:50 -0400

> 	servers being "not well maintained".  I know of no similar data

Does "not well maintained" include the following:

1) DNS server for the zone is originally configured correctly, and the
first 20-30 hosts are entered with a proper A record and a PTR that matches.

2) Clueful guy leaves, new DNS "goo-roo" takes over, and adds the next 300
machines with just an A record, and no PTR matching.  The checks you make
would show this as "well maintained", even though 90% of the hosts are broken
with respect to PTR entries.

Given that 7% of the sites can't get past step (1), I'm willing to bet that
a lot MORE of the sites are accumulating cruft under step (2).

-- 
				Valdis Kletnieks
				Operating Systems Analyst
				Virginia Tech

---

From: Keith Moore <[email-masked]>
Date: Tue, 25 Apr 2000 13:56:18 -0400

So, of the 57,887 visable servers, 4314 are improperly configured 
        in the visable in-addr.arpa. tree.  Thats 7.45% of the
        servers being "not well maintained". 

a 92.55% reliability rate is not exactly impressive, at least not in
a favorable sense.

it might be tolerable if a failure of the PTR lookup doesn't cause
the application to fail. but after all, people are saying that DNS
is good enough to serve as a means to map endpoint identifiers
to realm-local addresses or routing goop.  if applications were 
written to depend on DNS reverse lookups in order to get endpoint
identifiers of their peers, they would only work as reliably as DNS,
which isn't very good.

thanks for backing up my assertion with hard data.

Keith

---

From: Bill Manning <[email-masked]>
Date: Tue, 25 Apr 2000 11:01:24 -0700 (PDT)

% 
%         So, of the 57,887 visable servers, 4314 are improperly configured 
%         in the visable in-addr.arpa. tree.  Thats 7.45% of the
%         servers being "not well maintained". 
% 
% a 92.55% reliability rate is not exactly impressive, at least not in
% a favorable sense.
% 
% it might be tolerable if a failure of the PTR lookup doesn't cause
% the application to fail. but after all, people are saying that DNS
% is good enough to serve as a means to map endpoint identifiers
% to realm-local addresses or routing goop.  if applications were 
% written to depend on DNS reverse lookups in order to get endpoint
% identifiers of their peers, they would only work as reliably as DNS,
% which isn't very good.
% 
% thanks for backing up my assertion with hard data.
% 
% Keith

	I've not backed your assertion. I've provided some data
	on the relative stability of the in-addr space. You've provided
	zero data on the efficacy of the forward delegations.
	
	Can you, with a straight face, claim the servers for the
	forward zones have a better reliability rate than the 92.55%
	that I have seen in the inverse tree?  If so, where is the
	data to back your assertion?  Or would the community like
	me to do for the forward tree what I have been doing for the
	inverse tree?

--bill

---

From: Keith Moore <[email-masked]>
Date: Tue, 25 Apr 2000 14:24:26 -0400

>         that I have seen in the inverse tree?  

actually, no.  I don't have hard data to back the assertion that the
reverse tree is less reliable than the forward tree... nor do I really
need to do that to make the point that DNS PTR lookup isn't reliable 
enough to allow DNS to be used to lookup connection endpoint identifiers.

Keith

---

From: "David R. Conrad" <[email-masked]>
Date: Tue, 25 Apr 2000 12:19:49 -0700

Keith,

> the application to fail. 

If people's livelihood depends on something, they're more likely to insure it
actually works.  Very little depends on PTR records doing anything (with a
relatively few exceptions of sites that configure it otherwise).  The fact
that Bill's getting a 92.55% reliability figure for something that the vast
majority of people use to get something other than IP addresses in logfiles is
actually surprisingly good.

> if applications were

If it "isn't very good", try using the Internet without it for a bit.

In your view, what is it in the DNS protocol(s) that results in a lack of
reliability?

Rgds,
-drc

---

From: Keith Moore <[email-masked]>
Date: Tue, 25 Apr 2000 15:30:29 -0400

> If people's livelihood depends on something, they're more likely to insure 
> it actually works.  

that's a good point.  but it's one thing to make sure that DNS mappings
for "major" services are correct, and quite another to make sure that 
the DNS mappings are correct in both directions for every single host.

even the DNS names for major services may not be well maintained.  
at one time I did a survey of the reasons for mail bounces
for one of my larger mailing lists.  about half of the mail bounces
seemed to be due to configuration errors.  about half of those
seemed to be due to DNS configuration errors - e.g. MX records pointing
to the wrong host, zone replicas not being kept in sync, zone
replicas which were different but with the same serial number.

> In your view, what is it in the DNS protocol(s) that results in a lack of

the reliability problems are mostly not the protocol...though the protocol 
does have limitations if you want to use it (as some have proposed) to 
support host or process mobility.  and in the face of even moderate 
packet losses DNS queries can take a very long time. 

mainly it's the fact that DNS is maintained as a separate entity.
if you really want it to be in sync with reality, you need some
mechanisms to ensure that updates happen automagically, and/or that
configuration errors are automatically and quickly detected and
the information about the error gets to the person who can fix it.

Keith

---

From: [email-masked]
Date: Tue, 25 Apr 2000 17:09:47 -0400

> In your view, what is it in the DNS protocol(s) that results in a lack of

Actually, in my experience, the protocol isn't the biggest problem.  To
paraphrase an old joke - "95% of all automobile accidents are caused by
a mechanical failure - a loose nut behind the wheel".

It's like open mail relays - the majority of sites *wont* fix the
problem until the cost of fixing it is less than the pain of not
fixing it.  We had a number of hosts at our site listed in ORBS and
similar databases for MONTHS that cleaned up their act quickly once
they couldn't post to Listserv..

---

From: Marc Slemko <[email-masked]>
Date: Tue, 25 Apr 2000 15:32:47 -0600 (MDT)

> actually surprisingly good.

Except that figure is just for the delegation of the reverse to somewhere
that answers and knows something.

If you look at it on a host by host basis, you will find a lot more hosts
with incorrect or no reverse DNS.

I think that the separation between reverse and forward DNS is a big
problem in keeping things in synch.  Sure, this is a UI issue in some
ways.  But I can assure you that if, by default, people just had to edit a
single file and it would automatically do both forward and reverse
mappings, reverse DNS would be a lot more reliable.  Sure, there are cases
when you need to manually setup what you really want anyway.  It is even
worse when forward and reverse DNS are controlled by different groups.

But, as you say, if it has to work period then it will work.  If only 5%
of the things require it, however, those 5% will always be getting
broken.  

In some ways, I think DNS is too reliable, strictly on the protocol level.  
I would be interested to see numbers from a sample of randomly selected
domains regarding what percentage of their listed nameservers were working
properly, and by that I mean returning a SOA for the zone or something
equally mundane.

I can start off with a good example netscape.com: 75%.  1 of their listed
nameservers hasn't been responding (well, to DNS requests; handles HTTP
requests just fine...) for months.  I am completely unable to get such a
major domain to fix their broken DNS setup, despite the fact that it
causes me (not to mention the other couple of people who ever resolve
netscape.com like, say, anyone who ever starts up Navigator, regardless of
what their home page is set to) significantly noticible delays at times...

Also of interest would be looking at the difference in nameservers between
what a zone itself lists for the zone and what the parent lists for it.

That is always the problem when designing a protocol, of course.  If you
make it robust, you end up with a net loss when things are "working
normally" compared to if it were less robust and foced people to fix their
brokenness.  But that is getting off topic, whatever the topic really
is...

---

From: George Michaelson <[email-masked]>
Date: Wed, 26 Apr 2000 09:31:39 +1000

I think there are interesting things happening in DNS. I wrote a not very
good paper for AUUG a few years back noting an error rate in DNS above 10%
for the mirror site I do stats on.

Reviewing the figures for yesterday I get 9.75% unresolvable which is pretty
close to Bill Mannings figure.

But then I checked over the last 116 days since the start of the year. I find
that for a deployed site, logging IP and DNS name into CLF format (so I can
use analog) for ftp ,rsync and www I get:

	avg=15.288431, lo=4.213000, hi=33.265000

I think Bill is saying what really exists in DNS. I'm saying what a box
deployed in the field can expect to see. Its pretty damn variable and its
a lot worse than the DNS records would themselves suggest. Remember that
DNS is timebound with timeouts in client code, uses UDP and is subject to
the same kind of loss issues as the general datapath.

(this is on a client base of around 3000 hosts, weighted for Australia/NZ)

cheers
	-George
--
George Michaelson         |  DSTC Pty Ltd
Email: [email-masked]    |  University of Qld 4072
Phone: +61 7 3365 4310    |  Australia
  Fax: +61 7 3365 4311    |  http://www.dstc.edu.au

---

From: Bill Manning <[email-masked]>
Date: Tue, 25 Apr 2000 22:40:42 -0700 (PDT)

On Tue, 25 Apr 2000 08:18:20 PDT, Bill Manning said:
>	(... for forward data ...)

	Rather than continue to whine about lack of data, this is 
	the .US forward zone data:

	9928 unique servers answering for 33299 zones. 2165 servers blocked
	or refused data. Of the 7763 that answered, these are the numbers
	for improper configuration:
	
	BAD_SERVER:	14
	FORMERR:	2
	NXDOMAIN:	29

	So, of the 7763 visable servers, 45 are improperly configured in the
	visable US. tree. Thats 4.53% of those servers being "not well
	maintained.

	Keith, These two data points seem to bear your assertion out.

	in-addr.arpa	7.45% - poorly managed
	us.		4.53% - poorly managed

--bill

---

From: "Eliot Lear" <[email-masked]>
Date: Wed, 26 Apr 2000 00:15:57 -0700

From: Bill Manning <[email-masked]>
> Keith, These two data points seem to bear your assertion out.

It is always possible to do something poorly.  You can take the best
engineered product and misconfigure it, or otherwise not maintain it.
Sorta like not changing the oil.  On the other hand, how many times do you
see a name service failure for someone in the Keynote 40?  How often do you
get a failure for a site that uses a commercial service like Akamai?  If
you still get lots of failures when people are using the mechanism as
recommended, then the mechanism itself is poorly designed.  It also strikes
me that DNS is continuing to mature, and that things are getting better, as
more products come to the market, but this is more gut feel.

There are some open questions out there: does DNS provide sufficient
granularity?  Are its semantics rich enough for a mobile world?  People are
currently playing tricks with DNS that PVM had not envisioned, and Paul
Vixie loves to say something like, "DNS is not the droid you're looking
for" (forgive me, Paul, if I didn't get that quite right).

Just some thoughts...

---

From: "David R. Conrad" <[email-masked]>
Date: Wed, 26 Apr 2000 14:23:20 -0700

Keith,

> for one of my larger mailing lists. 

You appear to be saying that because historically people screwed up
configuring their DNS that it is impossible to rely on the DNS for critical
infrastructure.  This seems wrong to me.

If a properly configured DNS was a fundamental requirement of a working
network connection as is assumed by something like 8+8, I think it fairly
certain that any misconfigurations would be fixed as quickly as (say) a BGP
misconfiguration.

Rgds,
-drc

---

From: Keith Moore <[email-masked]>
Date: Wed, 26 Apr 2000 21:21:31 -0400

> infrastructure.  This seems wrong to me.

I wouldn't say 'impossible'.  My point is that it is more difficult
to get this to work well than it might seem at first glance.

one reason I cited the DNS-related problems in email is that many 
people would consider email a critical service, and also one that
is employed on a daily basis by a large portion of one's network users.
so if people won't do what's necessary to make their email work, will 
they take the necessary steps to make other less critical services work?

> certain that any misconfigurations would be fixed as quickly as (say) a BGP

it depends ... on the size of the user population affected by a DNS
record (probably much smaller than the typical BGP misconfiguration,
and therefore less important) and also on where the errors are detected 
(email errors are usually detected by the sender of a message, since
that's who gets the bounced message.  but the party who has responsibility 
for fixing the error is usually not on the sender's end of things)

and note that even if you fix the reliability problems associated
with using DNS to do mapping from global endpoint IDs to local
routing information, you still have the performance problems to
deal with.

Keith