Subject: [DNSOP] Glue is not optional, but sometimes it *is* sufficient...

From: Warren Kumari <[email-masked]>
Date: Thu, 21 May 2020 16:07:41 -0400

Hi all,

I decided to start a new thread for this, because it isn't really
about draft-andrews-dnsop-glue-is-not-optional - it is more of an
interesting aside / rathole...

What if you *only* have glue, and no authoritative answer / server?
Can I register example.com, put in www.example.com A 192.0.2.1 as
glue, and not bother with this whole annoying authoritative server
thing?

I asked this back in 2014, and was (correctly) told that this should
not work - I was pointed at RFC2181, which says:
"Unauthenticated RRs received and cached from the least trustworthy of
   those groupings, that is data from the additional data section, and
   data from the authority section of a non-authoritative answer, should
   not be cached in such a way that they would ever be returned as
   answers to a received query.  They may be returned as additional
   information where appropriate.  Ignoring this would allow the
   trustworthiness of relatively untrustworthy data to be increased
   without cause or excuse."

I did some testing on this back in late 2014, and the "success" rate
was ~75% - this has now dropped to ~5% (using Atlas to measure).

What on earth am I talking about? For the domain wow4dns.com, I have
*only* got glue (answers edited for brevity):

  $ dig +nostat +nocmd ns wow4dns.com @a.gtld-servers.com
  ;; QUESTION SECTION:
  ;wow4dns.com. IN NS
  ;; AUTHORITY SECTION:
  wow4dns.com. 172800 IN NS www.wow4dns.com.
  wow4dns.com. 172800 IN NS www1.wow4dns.com.
  ;; ADDITIONAL SECTION:
  www.wow4dns.com. 172800 IN A 193.151.173.35
  www1.wow4dns.com. 172800 IN A 193.151.173.35

There is no name-server listening on 193.151.173.35:
  $ dig www.wow4dns.com @193.151.173.35
  ;; connection timed out; no servers could be reached

There is, just for giggles, a webserver...

Using 1000 RIPE Atlas nodes, I try to resolve the name www.wow4dns.com
-- according to RFC2181 this Should Not Work(tm) -- and yet, ~3-5% of
resolvers (in this run, 38 out of 984) will resolve it, and to the
correct IP. This is RIPE Measurement #25400908 [0] for those who want
to play along at home...

The majority of these resolvers are in RFC1918 space, but there are
also some public addresses, including open recursives - e.g:
  $ dig www.wow4dns.com @37.32.120.136
  www.wow4dns.com. 86037 IN A 193.151.173.35

  $ host 37.32.120.136
  136.120.32.37.in-addr.arpa domain name pointer ns1.systec.ir.

  $ dig www.wow4dns.com @185.210.180.6
   www.wow4dns.com. 84737 IN A 193.151.173.35

  $ host 185.210.180.6
  6.180.210.185.in-addr.arpa domain name pointer ns2.txtv-tz.com.

Looking in the webserver log, there are also some hits - e.g:
- - [21/May/2020:19:09:10 +0000] "GET /favicon.ico HTTP/1.1" 404 209
"http://www.wow4dns.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138
Safari/537.36"

What does all of this *mean*?
.
.
.
Sorry, I haven't a clue, other than maybe:
The DNS is weird.
We passed the complexity event horizon a long time back...

W
[0]: https://atlas.ripe.net/measurements/25400908/#!probes

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: "John Levine" <[email-masked]>
Date: 21 May 2020 17:41:23 -0400

>glue, and not bother with this whole annoying authoritative server

Based on my recent analysis of TLD zones, yes if the zone is managed by
Afilias, or if you have friends at Nominet.  Otherwise not so much.

For wow4dns.com it looks pretty normal other than that your NS is lame.

Here's what's in this morning's .COM zone file, but I assume you've updated the NS since then:

WOW4DNS NS NS1.AUTH-SERVERS.NET.
WOW4DNS NS NS2.AUTH-SERVERS.NET.

Your registrar record and the live .COM NS say:

$ dig @g.gtld-servers.net. wow4dns.com a
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45456
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;wow4dns.com.			IN	A

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

R's,
John

---

From: Warren Kumari <[email-masked]>
Date: Thu, 21 May 2020 17:51:37 -0400

> WOW4DNS NS NS2.AUTH-SERVERS.NET.

Yeah, that was earlier in the day (before the testing, etc).

> www1.wow4dns.com.       172800  IN      A       193.151.173.35

Yes -- but information in the additional section should not be
promoted to an answer.
These IPs are only in the ADDITIONAL section - they should not be used
as answers.

W

---

From: Töma Gavrichenkov <[email-masked]>
Date: Fri, 22 May 2020 00:59:49 +0300

Peace,

On Thu, May 21, 2020 at 11:08 PM Warren Kumari <[email-masked]> wrote:
> [..skip..]

Is there a statistically — or somehow otherwise — reason to think this
was not a coincidence?
(I'm just askin' before going to repeat the same experiment)

--
Töma

---

From: Warren Kumari <[email-masked]>
Date: Thu, 21 May 2020 18:19:00 -0400

> (I'm just askin' before going to repeat the same experiment)

Yes, no, maybe?!

www.wow4dns.com was never pointed at this machine until I added the
glue and NS records (it used to point elsewhere). It's quite probable
that this was triggered by something other than the Atlas measurement,
but all that that means is that someone actually resolved the name
"for real".
I haven't seen any more hits since then, so, um, who knows...

I initially wanted to do an Atlas HTTP measurement (instead of a DNS
measurement), but then realized that the HTTP measurement type can
only be directed at a RIPE Anchor (not an arbitrary name). If there
are any Atlas folk around (or if people want to test from other tools
/ vantage points), feel free...

W

---

From: "John R Levine" <[email-masked]>
Date: 21 May 2020 18:45:25 -0400

> Yes -- but information in the additional section should not be

A week or two ago I scannned TLD zone files to see how many signed A and 
AAAA records there were.  Quite a lot, most looks to be orphan glue in 
Afilias zones that they didn't delete after the registered zone went away. 
No promotion needed.

If anyone wants, I can rerun the scan and show you what I found.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 22 May 2020 08:17:56 -0400

[ObDisclaim: I work for the Internet Society, but I'm not speaking for
them.]

>These IPs are only in the ADDITIONAL section - they should not be used

Are you quite sure they're not getting used as answers though?  Are
you sure query minimization is on for all cases?  If not, you'll ask
the parent-side server for the A record and may get an answer, though
non-authoritative.

The _reason_ you'll get an answer is because of the need for the glue
-- it could be that you're asking the question because you didn't have
the glue because of TC or something, and so you're coming back and
asking explicitly.  I know that at least one system that I worked on
would definitely respond this way, because under some circumstances it
was certainly necessary to be able to give such an answer.  The AA bit
wasn't set due to the delegation, but you could get the answer by
asking for it.

A

-- 
Andrew Sullivan
[email-masked]

---

From: Tony Finch <[email-masked]>
Date: Fri, 22 May 2020 13:47:32 +0100

John R Levine <[email-masked]> wrote:
>
> A week or two ago I scannned TLD zone files to see how many signed A and AAAA
> records there were.  Quite a lot, most looks to be orphan glue in Afilias
> zones that they didn't delete after the registered zone went away.

I vaguely remember a policy change in .com and .net years ago when they
stopped including orphan glue in the zones. Was this to do with prep work
for DNSSEC? I'm slightly surprised .org didn't follow suit.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Rattray Head to Berwick upon Tweed: Southwest 5 to 7, increasing gale 8 at
times later. Moderate or rough. Rain then showers. Good, occasionally poor at
first.

---

From: "John R Levine" <[email-masked]>
Date: 22 May 2020 09:24:51 -0400

> for DNSSEC? I'm slightly surprised .org didn't follow suit.

I believe that the policy is to remove orphan glue, and the glue in the 
Afilias zones is due to software bugs.  It's not just .org, it's also 
.info and .mobi and other zones they manage directly.

Regards,

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 22 May 2020 09:35:37 -0400

[ObDisclaimer: I work for ISOC, but don't speak for them.]

On Fri, May 22, 2020 at 09:24:51AM -0400, John R Levine wrote:
>I believe that the policy is to remove orphan glue, and the glue in 
>the Afilias zones is due to software bugs.  It's not just .org, it's 
>also ..info and .mobi and other zones they manage directly.
>

There were historical reasons why glue was not removed on various occasions, on purpose.  So I doubt it's just bugs.  I know there was a policy decision sometime around 05 or 06 that caused orphan glue to be removed, because I wrote the database  code for it.  That is, by the way a good reason to suppose that bugs could be part of the problem!

A

-- 
Andrew Sullivan
[email-masked]

---

From: Joe Abley <[email-masked]>
Date: Fri, 22 May 2020 10:33:01 -0400

Hi John,

> Afilias, or if you have friends at Nominet.  Otherwise not so much.

I think that some of the things you have been looking at concern orphan glue, John -- glue records that have been promoted to authoritative, signed RRSets in the TLD zone following the removal of a zone cut.

I think what Warren is talking about is the behaviour of the DNS as a whole (stub through authoritative) in the case where the zone cut is still there, so the A/AAAA records he is exercising are glue in the proper sense.

Joe

---

From: Joe Abley <[email-masked]>
Date: Fri, 22 May 2020 10:44:14 -0400

Hi Tony,

> for DNSSEC? I'm slightly surprised .org didn't follow suit.

I think the change you remember was a change in the way that responses were returned from the Atlas nameserver. Once upon a time, glue was returned in the answer section alongside a referral. The change I am thinking of changed that behaviour to follow other implementations, returning glue in the additional section instead.

ORG has not been served using the Atlas nameserver since the redelegation to PIR.

Joe

---

From: Shumon Huque <[email-masked]>
Date: Fri, 22 May 2020 10:49:04 -0400

On Fri, May 22, 2020 at 10:44 AM Joe Abley <[email-masked]> wrote:

> I think the change you remember was a change in the way that responses
> were returned from the Atlas nameserver. Once upon a time, glue was
> returned in the answer section alongside a referral. The change I am
> thinking of changed that behaviour to follow other implementations,
> returning glue in the additional section instead.
>

Here's the announcement of that change from Verisign (January 2010):

https://lists.dns-oarc.net/pipermail/dns-operations/2010-January/004841.html

Shumon.

---

From: Joe Abley <[email-masked]>
Date: Fri, 22 May 2020 10:52:28 -0400

On 21 May 2020, at 16:07, Warren Kumari <[email-masked]> wrote:

> ..
> ..
> ..

In your experiment it seems clear that all the glue records you are looking for are being returned from the involved authority-only servers in the additional section, and since for the COM zone that's a well-constrained monoculture of software it seems reasonable to imagine that's not where to look.

Similarly, by testing using Atlas probes the stub resolver presumably also represents a monoculture (or if there are different versions of probes, there are surely not that many different versions).

What remains is the tangle of resolvers, forwarders and proxies that exist between RIPE atlas probes and the authority servers, where there might actually be dragons. Not for the first time, I wish we had something like traceroute in the DNS so that we could isolate those paths rather than simply looking at exit addresses and trying to make inferences from them. I guess for some (apparently decreasing) proportion of those Atlas probes there's at least one dragon between the probe and the COM server that caches additional section glue and is happy to return it as an answer. I don't have any clever ideas about how you'd isolate any particular fictional reptile, however.

It'd be interesting to continue this kind of experiment over time and see where the success rate for those queries is trending.

Joe

---

From: "John R Levine" <[email-masked]>
Date: 22 May 2020 10:54:58 -0400

> I think what Warren is talking about is the behaviour of the DNS as a whole (stub through authoritative) in the case where the zone cut is still there, so the A/AAAA records he is exercising are gl…

You're right, they're different.

Regards,

---

From: Shumon Huque <[email-masked]>
Date: Fri, 22 May 2020 11:00:02 -0400

On Fri, May 22, 2020 at 10:52 AM Joe Abley <[email-masked]> wrote:

> In your experiment it seems clear that all the glue records you are
> looking for are being returned from the involved authority-only servers in
> the additional section, and since for the COM zone that's a
> well-constrained monoculture of software it seems reasonable to imagine
> that's not where to look.
>

Indeed. Since the COM referral response to glue queries is correct, this
means that some resolvers the Atlas probes are using are promoting glue to
answers.

Has anyone surveyed which of the following category of things are promoting
glue to answer: (1) other TLD authoritative services, (2) authoritative
server software implementations, (3) "public" (contentious term I know)
resolvers, and (4) resolver software implementations?

That would be an interesting but possibly very time consuming project - but
we could focus on the major ones.

Shumon.

---

From: Paul Wouters <[email-masked]>
Date: Fri, 22 May 2020 12:30:58 -0400 (EDT)

> It'd be interesting to continue this kind of experiment over time and see where the success rate for those queries is trending.

Although the 2010 announcement email listed only 2829 out of what? 70M
domains? And that was before DNSSEC and servers like unbound doing
referral harderning that would also lead those to not return answers
to clients that are just based on glue. So I would expect that number
to be even lower now.

So it seems compared the other DNS problems, this is pretty much
non-existing.

Paul

---

From: Tony Finch <[email-masked]>
Date: Fri, 22 May 2020 17:31:48 +0100

> https://lists.dns-oarc.net/pipermail/dns-operations/2010-January/004841.html

That's the one! - point 2 was what I was thinking of. The way they handle
glue under domains that are on hold is very tricky. And I think not
possible with standard zone files and zone transfers...

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Southeast Iceland: Northerly or northeasterly 6 to gale 8, perhaps severe gale
9 later. Moderate or rough, becoming very rough or high later. Fog patches at
first in north, otherwise rain. Moderate, occasionally very poor at first in
north.

---

From: "John Levine" <[email-masked]>
Date: 22 May 2020 14:30:18 -0400

>So it seems compared the other DNS problems, this is pretty much

It seems to me it's more a policy issue than a technical one.

It's technically trivial for any zone manager to compare the glue in
the zone with the delegated entries and see what's different. The
interesting part is what you do about it. The answer in all the TLDs I
know seems to be nothing.

There is the separate but related issue of orphan glue, of which there
is quite a lot, 28K signed A/AAAA records in .org and 54K in .info.
This is not a technical DNS issue, since technically those are just
ordinary records in the zone, but they would seem to be policy issues,
particularly for fans of POWERBIND or whatever it's called now.

R's,
John

---

From: Lanlan Pan <[email-masked]>
Date: Tue, 26 May 2020 00:34:01 +0800

Shumon Huque <[email-masked]> 于2020年5月22日周五 下午11:00写道：

> Has anyone surveyed which of the following category of things are
> promoting glue to answer: (1) other TLD authoritative services, (2)
> authoritative server software implementations, (3) "public" (contentious
> term I know) resolvers, and (4) resolver software implementations?
>
some resolvers send the NS_DOMAIN A query to glue ns ip, get the response
from authority,  and overwrite the glue.
the TTL of the authority response  is shorter than the glue.

>
> That would be an interesting but possibly very time consuming project -
> but we could focus on the major ones.
>
> Shumon.
>
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
>

---

From: Tim Wicinski <[email-masked]>
Date: Mon, 25 May 2020 19:08:25 -0400

On Mon, May 25, 2020 at 12:34 PM Lanlan Pan <[email-masked]> wrote:

>>> On 21 May 2020, at 16:07, Warren Kumari <warren@kumari..net
>>> <[email-masked]>> wrote:
>>>
> from authority,  and overwrite the glue..

(With no hats)

While in general the TTL of the authority response is shorter than the
glue, I can personally attest to domains
out in the wild, that at one point had authority TTL much larger than the
glue TTL.

tim