Subject: [DNSOP] Terminology: "primary master"

From: Havard Eidnes <[email-masked]>
Date: Thu, 23 Nov 2017 12:19:43 +0100 (CET)

Hi,

draft 07 says:

   Primary master:  "The primary master is named in the zone's SOA MNAME
      field and optionally by an NS RR".  (Quoted from [RFC1996],
      Section 2.1).  [RFC2136] defines "primary master" as "Master
      server at the root of the AXFR/IXFR dependency graph.  The primary
      master is named in the zone's SOA MNAME field and optionally by an
      NS RR.  There is by definition only one primary master server per
      zone."  The idea of a primary master is only used by [RFC2136],
      and is considered archaic in other parts of the DNS.

First: the last sentence's claim that the idea of a "primary
master" is only used by RFC 2136 is surely incorrect.  It is also
used in RFC 1996, where the first quote is taken from.

Secondly: can someone please explain to me why the idea of a
"primary master" where the zone data originates from and where
updates are performed is considered archaic?

Best regards,

- Håvard

---

From: Matthew Pounsett <[email-masked]>
Date: Thu, 23 Nov 2017 09:00:24 -0500

> used in RFC 1996, where the first quote is taken from.

I think the intent of the text is that an name is only used by the protocol
in RFC 2136.  It is referenced as a concept in RFC 1996, but not used by
the protocol.  But I agree that the text above is confusing, and if that's
the intent then it should be rewritten.

> updates are performed is considered archaic?

That doesn't make sense to me either.  Perhaps somebody thinks that Update
messages are no longer in use?

---

From: Petr Špaček <[email-masked]>
Date: Thu, 23 Nov 2017 15:22:34 +0100

On 23.11.2017 15:00, Matthew Pounsett wrote:
> On 23 November 2017 at 06:19, Havard Eidnes <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     Hi,
> 
>     draft 07 says:
> 
> I think the intent of the text is that an name is only used by the
> protocol in RFC 2136.  It is referenced as a concept in RFC 1996, but
> not used by the protocol.  But I agree that the text above is confusing,
> and if that's the intent then it should be rewritten.
>  
> 
> 
> That doesn't make sense to me either.  Perhaps somebody thinks that
> Update messages are no longer in use?

Not really, many Microsoft Active Directory instances have integrated
DNS and it uses DNS update to keep client machine's record up-to-date.

What is "archaic" is probably concept of primary master, because big
systems often have multi-master architecture (AD, or FreeIPA to name an
open-source one).

-- 
Petr Špaček  @  CZ.NIC

---

From: Joe Abley <[email-masked]>
Date: Thu, 23 Nov 2017 11:13:09 -0500

> updates are performed is considered archaic?

I think the only in-protocol use of the MNAME field is to specify the name to which UPDATE messages are sent. The MNAME was originally used to specify the name of a single master server to which slaves would send zone transfer requests, and from which zone data would be served in response, but as far as I am aware this was only ever informational: that is, I'm not aware of any implementation of a slave server that automatically detected its master by looking for the MNAME.

Today I think it's fair to say that any non-trivial DNS deployment that uses zone transfers makes use of a non-trivial graph for distribution of zone transfers, with slaves configured to send zone transfer requests to more than one master server, and in many cases with layers of servers that act as slaves and masters simultaneously for reasons of redundancy and scaling.

While a single slave sending zone transfer requests to a single master is still a valid example of such a graph, I think it's more a degenerate case than the usual case today. In that sense the idea of using a single master (which I think is implied by "primary master" and a name published in a single MNAME field) is defensibly archaic.

My perspective may not be universal, of course, but that's what I see.

Joe

---

From: Havard Eidnes <[email-masked]>
Date: Thu, 23 Nov 2017 17:36:40 +0100 (CET)

> I think the only in-protocol use of the MNAME field is to
> specify the name to which UPDATE messages are sent.

Really?  I read the NOTIFY RFC (1996) so that if the MNAME name
server is included in the NS set, it will by default be excluded
from receiving NOTIFY messages from the other name servers when
they reload the zone, ref. the definition of "Notify set" in RFC
1996.  Unsurprisingly, BIND has code to do this:

        result = dns_name_dup(&soa.origin, zone->mctx, &master);
...
        /*
         * Process NS RRset to generate notifies.
         */
...
                /*
                 * Don't notify the master server unless explicitly
                 * configured to do so.
                 */
                if (!DNS_ZONE_OPTION(zone, DNS_ZONEOPT_NOTIFYTOSOA) &&
                    dns_name_compare(&master, &ns.name) == 0) {
                        result = dns_rdataset_next(&nsrdset);
                        continue;
                }

Regards,

- Håvard

---

From: Joe Abley <[email-masked]>
Date: Thu, 23 Nov 2017 11:48:03 -0500

Hi Håvard,

On Nov 23, 2017, at 11:36, Havard Eidnes <[email-masked]> wrote:

> 1996.

Oh, good catch! I did not even know that text was there. I stand corrected!

That doesn't change the wider opinion I was trying to describe, though: in the modern/transfer graph scenario I was painting, NOTIFY configuration is far more nuanced than as described in RFC1996 and in practice (in BIND parlance) is an automatically-managed set of also-notify configuration.

Joe

---

From: Tony Finch <[email-masked]>
Date: Thu, 23 Nov 2017 17:44:49 +0000

Joe Abley <[email-masked]> wrote:
>
> In that sense the idea of using a single master (which I think is
> implied by "primary master" and a name published in a single MNAME
> field) is defensibly archaic.

It's quite difficult to have multiple masters and DNSSEC and coherent
copies of the zone from all masters - i.e. more effort than just spinning
up parallel instances of BIND or Knot in automatic signing mode. The
downstream xfer clients will get horribly muddled if you don't spend the
effort to ensure the masters are generating the same zone data for the
same serial numbers.

Easier to have one primary master which can be replaced quickly.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
Fair Isle, Faeroes: Northwest 7 to severe gale 9, occasionally storm 10. Very
rough or high. Squally wintry showers. Good, occasionally poor.

---

From: Paul Vixie <[email-masked]>
Date: Thu, 23 Nov 2017 10:16:25 -0800

Joe Abley wrote:
> On Nov 23, 2017, at 06:19, Havard Eidnes<[email-masked]>  wrote:
>...
> While a single slave sending zone transfer requests to a single
> master is still a valid example of such a graph, I think it's more a
> degenerate case than the usual case today. In that sense the idea of
> using a single master (which I think is implied by "primary master"
> and a name published in a single MNAME field) is defensibly archaic.

you need to get out more. by zone count, the IETF methodolody dominates 
the field. microsoft's AD approach, and other multi-master approaches, 
are used by the big guys, but they do not handle most zones.

> My perspective may not be universal, of course, but that's what I see.

this mailing list demonstrates the same myopia on many topics, as if a 
lack of evidence is itself evidence of nonexistence. that's just not so.

-- 
P Vixie

---

From: Joe Abley <[email-masked]>
Date: Thu, 23 Nov 2017 13:34:53 -0500

On Nov 23, 2017, at 12:44, Tony Finch <[email-masked]> wrote:

It's quite difficult to have multiple masters and DNSSEC and coherent
copies of the zone from all masters - i.e. more effort than just spinning
up parallel instances of BIND or Knot in automatic signing mode.

Note that I wasn't talking about multiple signers; I was talking about
(from the perspective of one particular slave) having multiple masters
available to serve precisely the same zone.

Feeding a large array of slaves (eg hundreds, including individual members
if clusters) with large numbers of zones from a single master doesn't scale
very well.

Joe

---

From: Paul Vixie <[email-masked]>
Date: Thu, 23 Nov 2017 10:48:56 -0800

Joe Abley wrote:
> ...
>
> Feeding a large array of slaves (eg hundreds, including individual
> members if clusters) with large numbers of zones from a single master
> doesn't scale very well.

when i had to do this i fed 100 from one or two, where the two were HA 
using non-DNS tech such as NFS or sql mirroring. then i've fed 1000 from 
each of those hundred, choosing one as primary and one as backup.

if you have a better way, using extant, open technology, to reach 
100,000 zxfr/ixfr clients with secure coherent zone content with minimal 
delay, i'd like to hear more.

when i had to feed hundreds of zones to large populations, i used MZ 
(see http://family.redbarn.org/~vixie/mz.pdf and perhaps also 
dotat.at/prog/nsnotifyd/metazone.5.pdf).

if you have a better way, using extant, open technology, to propagate a 
moderate to large and dynamic catalogue of zones toward a cooperating 
set of secondary servers, i'd like to hear more.

-- 
P Vixie

---

From: Joe Abley <[email-masked]>
Date: Fri, 24 Nov 2017 14:57:11 -0500

Hi Paul,

On 23 Nov 2017, at 13:48, Paul Vixie <[email-masked]> wrote:

> when i had to do this i fed 100 from one or two, where the two were HA using non-DNS tech such as NFS or sql mirroring. then i've fed 1000 from each of those hundred, choosing one as primary and one as backup.
> 
> if you have a better way, using extant, open technology, to reach 100,000 zxfr/ixfr clients with secure coherent zone content with minimal delay, i'd like to hear more.

I presume you mean existing, not extant. The DNS is a bit new for any of its technology to become extinct or lost to the winds of history, unless that's a naïve comment because I'm insufficiently aware that I'm unaware of the things I'm unaware of. Or perhaps this is a trans-atlantic difference in vocabulary.

The principal challenges I have seen using a non-trivial transfer graph with NOTIFY and [AI]XFR are when a pair of candidate masters for a particular slave each receive a new revision of a particular zone at a different time. The one that receives it first sends NOTIFY messages to its slaves; if slaves don't use the NOTIFY source to select the master subsequently used to send SOA queries and [AI]XFR requests to (and it is normal not to in many/most/all widely-used implementations) there is a chance they will talk to the master that hasn't yet been updated and hence not retrieve the new revision of the zone. Presumably all masters update eventually, however, so normality will eventually be restored.

Whether this is a problem depends on your tolerance for delay in propagation of new zone data to the slaves, and the potential for equivalent masters (equivalent to the slave) to be carrying different revisions of the zone for a long time. It can work as-is; if it doesn't meet the requirements then you have to think a little harder about how you synchronise your masters or what client behaviour can be arranged when they receive NOTIFY messages.

It's nice when it works as-is, because a pure DNS solution avoids the kind ofshared fate and monitoring/troubleshooting overhead that things like load balancers and network filesystems can introduce. Every time I have seen a failure directly caused by a problem in something that was intended to provide high availability, it has made everybody involved very sad.

The state involved in processing NOTIFY messages, especially when there's loss in a path somewhere which triggers timeouts, back-off and retransfer, can be painful to manage at scale. I have seen some abuse of the specification in the form of fire-and-forget NOTIFYs to avoid the state problem (discard NOTIFY responses received and never attempt to retry sending, assuming that if ours didn't get through a NOTIFY from another master probably will).

Davey Song described a scenario that played out in the Yeti project where multiple masters each serving zones that were equivalent but different caused problems to downstream IXFR clients. The zones were equivalent in the sense that they only differed in RRSIGs -- each server used a local ZSK, corresponding properly to an RR in the DNSKEY RRSet, to sign all the non-DNSKEY RRSets -- but different in the sense that the RRSIGs were different, so really they were different zones with the same origin. IXFR clients got confused, which I think in retrospect was probably to be expected. This feels like the kind of scenario that Tony was talking about in his earlier message in this thread.

> when i had to feed hundreds of zones to large populations, i used MZ (see http://family.redbarn.org/~vixie/mz.pdf and perhaps also dotat.at/prog/nsnotifyd/metazone.5.pdf).

I used an earlier version of that back in the day, as you might remember. Pretty much everything I've been involved in since then has used out-of-band provisioning of zone targets though, using devopsy automation tools (ansible, chef, puppet) or provisioning databases that were replicated independently of the DNS. One notable exception used supermasters in PowerDNS, which you could call in-band (but still a different approach from metazones): <https://doc.powerdns.com/md/authoritative/modes-of-operation/#supermaster-automatic-provisioning-of-slaves>.

Joe

---

From: Paul Vixie <[email-masked]>
Date: Fri, 24 Nov 2017 15:51:52 -0800

Joe Abley wrote:
> Hi Paul,
> ...
> I presume you mean existing, not extant. ...

yes. thanks for understanding what i mean in spite of what i said.

> The principal challenges I have seen using a non-trivial transfer
> graph with NOTIFY and [AI]XFR are when a pair of candidate masters for a
> particular slave each receive a new revision of a particular zone at a
> different time. The one that receives it first sends NOTIFY messages to
> its slaves; if slaves don't use the NOTIFY source to select the master
> subsequently used to send SOA queries and [AI]XFR requests to (and it is
> normal not to in many/most/all widely-used implementations) there is a
> chance they will talk to the master that hasn't yet been updated and
> hence not retrieve the new revision of the zone. Presumably all masters
> update eventually, however, so normality will eventually be restored.

first let's clarify that this can't happen between the PM and its DM's,
because in my example, the PM was HA. i do know that not all NOTIFY
responders will try-first the server who sent them the NOTIFY, and that 
this can result in not learning a new serial number after a NOTIFY. the
BIND4 version of the NOTIFY code did this correctly, but i'm guessing
that RFC 1996 was not as clear on this topic as it might have been.

my preferred solution to this woe is to fix the NOTIFY responders, but 
as we know this is often outside our powers. my actual solutions to this 
have been to either delay all NOTIFY's for some amount of time that's 
long enough to get the DM's to all settle, and make sure that the 
settle-time is as short as i can make it. or better still, give the 
slave-operators only one address for "zone master", and anycast that one 
to get redundancy.

as an aside, the computed random delay-before-notify was not specified 
as a way to get a longer settling time; it was just to prevent all of 
the recipients of a NOTIFY-burst from doing their AXFR's in parallel. 
this was damnfoolishness related to BIND4's "fork on AXFR" logic, and i 
deeply regret it. 1996 was my first RFC and it shows.

> The state involved in processing NOTIFY messages, especially when
> there's loss in a path somewhere which triggers timeouts, back-off and
> retransfer, can be painful to manage at scale. I have seen some abuse of
> the specification in the form of fire-and-forget NOTIFYs to avoid the
> state problem (discard NOTIFY responses received and never attempt to
> retry sending, assuming that if ours didn't get through a NOTIFY from
> another master probably will).

alas, many operators solve only the problem they know they're having, 
without regard to the ones they don't know about, and without regard for 
the problems their solutions might cause for others. "monkey DNA".

> Davey Song described a scenario that played out in the Yeti project
> where multiple masters each serving zones that were equivalent but
> different caused problems to downstream IXFR clients. The zones were
> equivalent in the sense that they only differed in RRSIGs -- each server
> used a local ZSK, corresponding properly to an RR in the DNSKEY RRSet,
> to sign all the non-DNSKEY RRSets -- but different in the sense that the
> RRSIGs were different, so really they were different zones with the same
> origin. IXFR clients got confused, which I think in retrospect was
> probably to be expected. This feels like the kind of scenario that Tony
> was talking about in his earlier message in this thread.

yeti is probably a degenerate non-teaching corner case. since it deals
with the root zone, we had to politicize the design -- there are three
DM's whose only shared data is the IANA source zone they start with, and 
the KSK they sign ZSK's with. we had to avoid annointing any one party 
as having a special relationship to root-like zone data. this is 
unlikely to be a problem for anyone else or for any other zone.

>>see http://family.redbarn.org/~vixie/mz.pdf ...

> I used an earlier version of that back in the day, as you might
> remember. Pretty much everything I've been involved in since then has
> used out-of-band provisioning of zone targets though, using devopsy
> automation tools (ansible, chef, puppet) or provisioning databases that
> were replicated independently of the DNS.

that's what a lot of the professionals do. however, i won't recommend 
technology that hasn't been documented and tested for interoperability, 
portability, and lack of IPR encumbrance.

> One notable exception used
> supermasters in PowerDNS, which you could call in-band (but still a
> different approach from
> metazones)

MZ was influenced by BIND, and only implemented for BIND, but was meant 
to be name-server independent. i would like to see something like this 
gain broad industry traction, so that we can all spend our time solving 
things that somebody else hasn't already solved.

-- 
P Vixie

---

From: Tony Finch <[email-masked]>
Date: Mon, 27 Nov 2017 13:22:51 +0000

> available to serve precisely the same zone.

A primary master is wrt a zone not a server - a zone's primary master is
a server that's authoritative for a zone and which does not get the zone
contents via axfr/ixfr, but instead from a master file and/or UPDATE (or
a non-standard mechanism such as directly from a database).

The term says nothing about the downstream xfer topology. You can very
well having a primary master (plus backup(s)) feeding a cluster of
distribution servers which in turn feed the public authoritative servers.
The xfer topology doesn't have to be a tree, it doesn't even have to be an
acyclic graph.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
German Bight, Humber: West becoming cyclonic 5 to 7, increasing gale 8 or
severe gale 9 for a time. Moderate or rough, occasionally very rough. Rain or
squally showers. Moderate or good, occasionally poor.

---

From: P Vix <[email-masked]>
Date: Mon, 27 Nov 2017 13:29:06 +0000

Exactly.

On November 27, 2017 9:22:51 PM GMT+08:00, Tony Finch <[email-masked]> wrote:
>_______________________________________________
>DNSOP mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/dnsop

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Joe Abley <[email-masked]>
Date: Mon, 27 Nov 2017 11:34:11 -0500

Hi Tony,

> a non-standard mechanism such as directly from a database).

That's an alluringly clear definition, but I'm not sure it matches common understanding of the term, which I think has more to do with "single source of truth" than with the specifics of what transport is used to provision zone data in a server.

For example,

    W <------- A -------> X

Suppose A is a source of truth for a particular zone, and that W and X obtain zone data from A. Are you saying that if the mechanism represented by the arrows is [AI]XFR then A is a primary master and W and X are not, whereas if that mechanism is something else (perhaps it's rsync, with W, A and X all configured to be masters from local zone files) then W, A and X are all primary masters?

If A is not a nameserver but instead is a database, and the arrows represent database replication, then W and X are primary masters but A is not?

Joe

---

From: Tony Finch <[email-masked]>
Date: Mon, 27 Nov 2017 17:15:17 +0000

Joe Abley <[email-masked]> wrote:

> That's an alluringly clear definition, but I'm not sure it matches
> common understanding of the term, which I think has more to do with
> "single source of truth" than with the specifics of what transport is
> used to provision zone data in a server.

Hmm, I think I sort-of agree, but with caveats...

I prefer to make a clear distinction between the standard DNS as specified
in the RFCs versus any funky non-standard stuff that people might do with
the DNS.

The standard DNS protocols are closely fitted to a particular distributed
architecture, in which a zone has a single source of truth that we call
the primary master. You can, of course, implement a similar architecture
using non-standard protocols, but if you do you should take care to make
it clear how you are diverging from the standard - and be careful about
how you use standard DNS terminology when talking about your non-standard
system.

> For example,
>
>     W <------- A -------> X
>
> Suppose A is a source of truth for a particular zone, and that W and X
> obtain zone data from A. Are you saying that if the mechanism
> represented by the arrows is [AI]XFR then A is a primary master and W
> and X are not, whereas if that mechanism is something else (perhaps it's
> rsync, with W, A and X all configured to be masters from local zone
> files) then W, A and X are all primary masters?

In this case I would say A is a primary master and W and X are secondaries
that use rsync instead of standard zone transfers.

The part of the standard architecture that you have replaced is the zone
transfer mechanism, so the primary master is architecturally unaffected,
so it's OK to use the same name.

(But you can't call rsync "ixfr" even though it is incrementally
transferring zones, because that would be unreasonably confusing.)

> If A is not a nameserver but instead is a database, and the arrows
> represent database replication, then W and X are primary masters but A
> is not?

In this case I would be very proud of my high-availability multi-master
system.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
Shannon, Rockall, Malin, Hebrides, Bailey: North or northwest 5 to 7,
occasionally gale 8 except in Shannon. Rough or very rough, occasionally
moderate. Squally showers. Good.