Subject: Re: [dnsext] Fwd: New Version Notification for draft-ah-dnsext-rfc1995bis-ixfr-02

From: Ondřej Surý <[email-masked]>
Date: Thu, 16 Jun 2011 09:50:21 +0200

Hi,

since I didn't receive any comments I guess:

a) it's not exciting
b) it's so good that it doesn't need any improvement [which I doubt]

I would like to ask the WG for adoption of this document, so we can move
on...  after all this is just a clarification document with few things
added (no name sameness I promise! :)).

O.

On 27.4.2011 14:47, Ondřej Surý wrote:
> Diff is here:
> 
> https://tools.ietf.org/rfcdiff?url1=draft-ah-dnsext-rfc1995bis-ixfr-01&difftype=--html&submit=Go!&url2=draft-ah-dnsext-rfc1995bis-ixfr-02
> 
> 
> Sorry it took me so long to prepare -02, which we believe is as good as
> it gets.  Many thanks for Alfred for doing most of the work.

-- 
 Ondřej Surý
 vedoucí výzkumu/Head of R&D department
 -------------------------------------------
 CZ.NIC, z.s.p.o.    --    Laboratoře CZ.NIC
 Americka 23, 120 00 Praha 2, Czech Republic
 mailto:[email-masked]    http://nic.cz/
 tel:+420.222745110       fax:+420.222745112
 -------------------------------------------
_______________________________________________
dnsext mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/dnsext

---

From: Edward Lewis <[email-masked]>
Date: Thu, 16 Jun 2011 15:22:03 -0400

>b) it's so good that it doesn't need any improvement [which I doubt]
  c) was unaware of the document's existence despite thinking "it would be
     about time to tackle this"

;)

I began reading the draft and started listing a 
lot of comments.  It would take me a long time to 
complete, so I thought I'd raise two things here.

1. The document talks a lot about the context of 
an IXFR exchange.  Can this be removed and stick 
to just the actions of an IXFR client and server? 
I mean, it doesn't matter why the IXFR is being 
kicked off.  (BTW, the statement that IXFRs are 
mostly kicked off by NOTIFY isn't really true, we 
oversee a lot of zones that do not change thus do 
not cause NOTIFY's to fly.  This factoid applies 
to different places but in the end, doesn't 
really matter when you are describing the 
protocol.)

2. For the first time I have thought abotu 
IXFR-only and I don't get it at all.  The reason 
is I don't accept that "If this is not possible, 
in the case of bare IXFR, the server falls back 
to AXFR, i.e. it provides the full zone content." 
Yes, there's a AXFR-style IXFR (all in one UDP 
message) but I don't think that is what is meant.

When a server indicates that IXFR can't be used, 
the server can not switch to AXFR.  The switch 
that is observed in operations comes from the 
client deciding to switch.  If the reason for 
IXFR-only is that a client, having failed to IXFR 
from IP address #1 will try AXFR instead of 
trying IXFR from IP address #2, then my response 
is that this is an implementation issue, not a 
protocol issue.

RFC 1995 never specifies that a failed IXFR will 
cause an AXFR to happen.  Maybe that's an 
accident of history, because implementations do 
this that we think that is what is supposed to 
happen.  It's not in the specification that way, 
as far as I can find.

The closest passage is in section 2 of RFC 1995:

    Thus, a client should first make an IXFR query using UDP.  If the
    query type is not recognized by the server, an AXFR (preceded by a
    UDP SOA query) should be tried, ensuring backward compatibility.

Note that this is if the IXFR QTYPE is not 
recognized, not a refusal to use IXFR.  And note 
that this is for backward compatibility.

Thinking in another vein, if a client chooses not 
to ever use AXFR, then what happens when IXFR 
requests can't be satisfied?  Let the zone 
EXPIRE?  Is a dead server better than a server 
that is taking time to sync up?  (This calls out 
the motivation for IXFR-only.)

---

As I mentioned, there is a need to refresh this 
RFC - well, maybe.  I want to suggest specifying 
that a IXFR client will verify the RRSIG of the 
first SOA returned.  This will close one more 
place a forger might be able to insert into or 
remove records from a DNSSEC signed zone without 
detection (assuming there's no VPN, TSIG, TKEY or 
similar protection).

-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis             
NeuStar                    You can leave a voice message at +1-571-434-5468

I'm overly entertained.

---

From: Brian Dickson <[email-masked]>
Date: Thu, 16 Jun 2011 16:48:21 -0400

On Thu, Jun 16, 2011 at 3:22 PM, Edward Lewis <[email-masked]> wrote:

> 2. For the first time I have thought abotu IXFR-only and I don't get it at
> all.  The reason is I don't accept that "If this is not possible, in the
> case of bare IXFR, the server falls back to AXFR, i.e. it provides the full
> zone content."
>
[snip]
>
> RFC 1995 never specifies that a failed IXFR will cause an AXFR to happen.

Actually, it does, right at the beginning of section 4. Quoting the text:

4. Response Format

   If incremental zone transfer is not available, the entire zone is
   returned.  The first and the last RR of the response is the SOA
   record of the zone.  I.e. the behavior is the same as an AXFR
   response except the query type is IXFR.

> Thinking in another vein, if a client chooses not to ever use AXFR, then
> what happens when IXFR requests can't be satisfied?  Let the zone EXPIRE?
>  Is a dead server better than a server that is taking time to sync up?
>  (This calls out the motivation for IXFR-only.)

Those fall under the scope of operator, rather than implementer, issues.
Presumably the operator will act sensibly, trying all available
sources for IXFR-ONLY,
before giving up and doing an explicit AXFR.

The need for IXFR-ONLY is the desire to have interoperable implementations.

Suggesting that non-interoperable implementations of IXFR, to support IXFR-ONLY,
goes against common sense.

It is reasonable to expect zone administrators to operate multiple
authority servers for their zones.
It is reasonable to expect that zone administrators will want to have
zones instances be identical.
It is reasonable to expect that zone administrators may want to
operate servers from multiple vendors.

When you combine "multiple vendors" with "zone instances identical",
and add in "efficient transport",
you get the requirement for inter-vendor, interoperable IXFR, quite
possibly with IXFR-ONLY as a desired
capability.

The problem is, there is no current work-around, and adding this
without -bis standardization, isn't
necessarily interoperable. Better to update the standard. It's a no-brainer.

(End of flogging of the deceased quadroped.)

Brian

---

From: Mark Andrews <[email-masked]>
Date: Fri, 17 Jun 2011 10:17:40 +1000

In message <[email-masked]>, Brian Dickson 
writes:
> > all. =A0The reason is I don't accept that "If this is not possible, in the
> > case of bare IXFR, the server falls back to AXFR, i.e. it provides the fu=
> ll
> > zone content."
> >
> [snip]
> >
> > what happens when IXFR requests can't be satisfied? =A0Let the zone EXPIR=
> E?
> > =A0Is a dead server better than a server that is taking time to sync up?
> > =A0(This calls out the motivation for IXFR-only.)
> 
> Suggesting that non-interoperable implementations of IXFR, to support IXFR-=
> ONLY,

Ed,
    the problem is that in some ixfr implementations there is no
way to turn off, the optional, delta consolidation.  If your then
have a server being fed from two+ intermediate servers (for redundancy)
that are in turn being fed servers that consolidate deltas you get
lots of AXFR style responses when you switch between intermediate
servers.

With this setup the intermediate servers don't have all the changes
that have occurred to the zone so they can't always generate a delta
style response as they don't have the starting point.

This exact senario was warned about in Section 6 of RFC 1995.

   But, this feature may not be so useful if an IXFR client has access
   to two IXFR servers: A and B, with inconsistent condensation results.
   The current version of the IXFR client, received from server A, may
   be unknown to server B. In such a case, server B can not provide
   incremental data from the unknown version and a full zone transfer is
   necessary.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]
_______________________________________________
dnsext mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/dnsext

---

From: Masataka Ohta <[email-masked]>
Date: Fri, 17 Jun 2011 10:07:24 +0900

Mark Andrews wrote:

> way to turn off, the optional, delta consolidation.

That's not a problem. However, if the implementations blindly
consolidate ignoring the paragraph:

   Information about older versions should be purged if the total length
   of an IXFR response would be longer than that of an AXFR response.
   Given that the purpose of IXFR is to reduce AXFR overhead, this
   strategy is quite reasonable.  The strategy assures that the amount
   of storage required is at most twice that of the current zone
   information.

they are poor implementations to be avoided by rational operators,
unless there are fair reasons of the ignorance.

Consolidations following the paragraph above may still cause
transient inefficiency. But, does that matter and worth protocol
extensions and additional operational complexities?

						Masataka Ohta

---

From: Mark Andrews <[email-masked]>
Date: Fri, 17 Jun 2011 11:28:27 +1000

> extensions and additional operational complexities?

Ohta san, you totally missed the point.

Sending a AXFR style response when you have the requested starting
serial is *not* a problem.  The problem is sending a AXFR style
response when you don't have the requested starting serial but do
have a potential starting serial that is before the requested
starting serial.

Mark

---

From: Masataka Ohta <[email-masked]>
Date: Fri, 17 Jun 2011 10:44:01 +0900

Mark Andrews wrote:

> The problem is sending a AXFR style

						Masataka Ohta

---

From: Edward Lewis <[email-masked]>
Date: Fri, 17 Jun 2011 09:29:02 -0400

>    response except the query type is IXFR.

No, that is not a fallback to AXFR.  The spec is not saying that an 
unsatisfactory IXFR session leads to an AXFR session (with the same 
remote address).

What is confusing is the phrase "is the same as."

If the query type is IXFR, the response will be an IXFR response that 
contains all of the records in one (UDP) datagram, which functionally 
is the same as a AXFR having all of the records appear in multiple 
DNS messages over a stream - the same for only small to very small 
zones.  The response is limited to 512 bytes or whatever is in the 
EDNS0 buffer size of the requestor.

Think of it this way.  If I send an IXFR request over UDP, how could 
a response be an AXFR?  AXFR can only be sent on TCP.

In a way, the net result of an AXFR-Style IXFR (as I've heard that 
called) and an AXFR is that you get the entire zone in one shot. 
Functionally the result is the same.  But the delivery mechanism is 
much different.  No spec ever said that an implementation had to back 
up IXFR with AXFR (except for backwards compatibility - the passage I 
quoted).

>sources for IXFR-ONLY, before giving up and doing an explicit AXFR.

Operators generally use the tools implementers give them.

>The need for IXFR-ONLY is the desire to have interoperable implementations.

I said this in another context earlier this week, the desire to have 
interoperable implementations is subsidiary to having features that 
make sense.

>and add in "efficient transport", you get the requirement for
>inter-vendor, interoperable IXFR,

That's obvious, but, wait for it...

>quite possibly with IXFR-ONLY as a desired capability.

Note here you drop to "quite possibly" - but after reading up on 
IXFR-only I wouldn't go along with even that.  My beef here is that 
IXFR-only isn't needed (in the protocol) not that there shouldn't be 
a standard spec for it.  IXFR-all-sources-before-AXFR makes more 
sense than IXFR-only.  And even that is what I would consider an 
implementation optimization.

I'm overly entertained.

---

From: Edward Lewis <[email-masked]>
Date: Fri, 17 Jun 2011 09:53:57 -0400

>>>  extensions and additional operational complexities?

I'm arguing the same as Ohta-san.

I think RFC 1995's description is pretty clear, and whatever 
IXFR-only is trying to solve is just to address what implementations 
have done at the cost of muddying up the protocol further.

I had been thinking of requesting IXFR be updated for DNSSEC, but 
even that idea is really just about hardening implementations, not 
changing the IXFR protocol at all.
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis
NeuStar                    You can leave a voice message at +1-571-434-5468

I'm overly entertained.

---

From: Ondřej Surý <[email-masked]>
Date: Fri, 17 Jun 2011 16:15:15 +0200

On 17.6.2011 15:29, Edward Lewis wrote:
> contains all of the records in one (UDP) datagram, which functionally is
> the same as a AXFR having all of the records appear in multiple DNS
> messages over a stream - the same for only small to very small zones. 
> The response is limited to 512 bytes or whatever is in the EDNS0 buffer
> size of the requestor.
> 
> Think of it this way.  If I send an IXFR request over UDP, how could a
> response be an AXFR?  AXFR can only be sent on TCP.
> 
> Functionally the result is the same.  But the delivery mechanism is much
> different.  No spec ever said that an implementation had to back up IXFR
> with AXFR (except for backwards compatibility - the passage I quoted).

But the fact is that you get a full zone.

Also the new draft says:

   In case of fallback to AXFR, the answer contains the same RRs (and is
   subject to the same ordering rules) as specified in Sections 2.2
   through 3 of RFC 5936, with the single difference being the echoed
   QCODE (i.e., IXFR) in the Question section.

The fact is that the it's the behaviour which is "AXFR" (that is what we
talk about), but the QCODE/RCODE is not "AXFR", but IXFR.

> interoperable implementations is subsidiary to having features that make
> sense.

The IXFR-only is backwards compatible both ways in Client+, Server- and
Client-,Server+ scenarios.

Client+,Server-

C: Query IXFR-ONLY
S: NotImpl
C: Query IXFR
S: ...

Client-,Server+

C: Query IXFR
S: ...

> IXFR-all-sources-before-AXFR makes more sense
> than IXFR-only.  And even that is what I would consider an

It is optimization (look into archive, we talked about that before and
it should be written in Motivations for IXFR-only section of the I-D).

And you cannot do IXFR-all-sources-before-AXFR without IXFR-only.

O.
P.S.: I'll reply to your first mail later, but bear in mind that it's
written in context of AXFR clarify.

---

From: Edward Lewis <[email-masked]>
Date: Fri, 17 Jun 2011 10:50:15 -0400

>talk about), but the QCODE/RCODE is not "AXFR", but IXFR.

My impression is that the issue surrounding 
IXFR-only is about load on processing and network 
latency.  I fail to see that the problem is that 
you get all of the zone at once. The problem is 
that you have to do a lot of work to get all of 
the records of an large zone when all you really 
need is a just a small increment.

Why else would IXFR-only be needed?

Is it just to discover all of the increments? 
That sounds like a poor reason.  The increments 
aren't important except to satisfy onward IXFR 
answers and even then, if the zone is small, why 
not just use AXFR-style IXFR?

>it should be written in Motivations for IXFR-only section of the I-D).

The archives are not searchable - have links?
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis
NeuStar                    You can leave a voice message at +1-571-434-5468

I'm overly entertained.

---

From: Brian Dickson <[email-masked]>
Date: Fri, 17 Jun 2011 11:02:05 -0400

On Fri, Jun 17, 2011 at 9:29 AM, Edward Lewis <[email-masked]> wrote:

> contains all of the records in one (UDP) datagram, which functionally is the
> same as a AXFR having all of the records appear in multiple DNS messages
> over a stream - the same for only small to very small zones.  The response
> is limited to 512 bytes or whatever is in the EDNS0 buffer size of the
> requestor.

IXFR does not need to be over UDP. In RFC 1995, it even describes falling back
to doing IXFR over TCP, if IXFR over UDP fails for anything other than
a few specific
RCODEs.

If the IXFR query is sent over TCP, the answer will either be an IXFR
(for real), or
an IXFR which includes the whole zone.

IXFR-ONLY is designed specifically to handle the latter case,
preventing the "or..."
from being done.

There are plenty of zones that require IXFR-ONLY in order to
efficiently and effectively
process zone updates.

The motivation and justification for this is clear, IMHO. If there is
no workaround possible,
which I assert is the case for some zones, then IXFR-ONLY is required.

I see no reason not to include it in the protocol specification, if it
is going to be implemented
by more than one implementation. And I assert that the latter is
certainly the case.

Brian

---

From: Edward Lewis <[email-masked]>
Date: Fri, 17 Jun 2011 11:41:24 -0400

At 11:02 -0400 6/17/11, Brian Dickson wrote:

>IXFR does not need to be over UDP.

The point is that IXFR can be over UDP, not that it could be over TCP.

>(for real), or an IXFR which includes the whole zone.

I don't think there is an understanding of IXFR.  I'd suggest reading 
1995 again and checking it against your code.  Perhaps your code 
missed something.

>efficiently and effectively process zone updates.
>

Can you give me an example how your implementation is less 
"efficient" without IXFR-only and how you would alter your 
implementation to make use of it.

That would help establish a justification...
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis
NeuStar                    You can leave a voice message at +1-571-434-5468

I'm overly entertained.

---

From: Brian Dickson <[email-masked]>
Date: Fri, 17 Jun 2011 14:05:04 -0400

On Fri, Jun 17, 2011 at 11:41 AM, Edward Lewis <[email-masked]> wrote:
> I don't think there is an understanding of IXFR.  I'd suggest reading 1995
> again and checking it against your code.  Perhaps your code missed
> something.

I am pretty sure I didn't miss anything.

The point is, that there could be a situation, for a large zone, such that:
- IXFR over UDP is never possible (too much flux for the capacity of UDP)
- "real" IXFR might or might not be possible due to purging
- AXFR is orders of magnitude bigger (e.g. 10^4 or more) than IXFR

The "clever" trick of "AXFR" (IXFR fallback) being too big for UDP,
can't be used.

Either a protocol change (non-standard, or standards-based) between
the client is needed,
or the client is forced to be very unfriendly, and abort the IXFR if
it doesn't see its own Serial
as the second RR of the IXFR. Closing the TCP socket is the only
mechanism it would have to do that.

> Can you give me an example how your implementation is less "efficient"
> without IXFR-only and how you would alter your implementation to make use of
> it.
>

I'll do my best.

First, let's presume there's a non-trivial amount of IXFR needed, and
a non-trivial number of servers involved.

The smallest non-trivial arrangement that comes to mind is the following:
- 1 hidden master (source of truth, originator of all zone data)
- a set of distribution servers (hidden again, feeding non-hidden servers)
- authority servers to the world, who get their zones via XFR from the
distribution servers

This can be scaled upward by adding additional layers of distribution,
if needed.
Redundancy across multiple servers at each layer is presumed. (The
redundancy on the hidden master itself is out of scope.)

Now, presume some kind of command-and-control infrastructure,
directing/coordinating the activities and functions of the servers.
Call it the MCP (master control program).

And also presume some kind of mini-command-and-control scripting on
each of the non-hidden authority servers, i.e. for when they boot up.
Call it the bcp (boot control program).

Here's an example design:
The MCP keeps track of the current Hidden Master Serial Number (HMSN).
The MCP also keeps track of the Distribution Server Serial Numbers
(DSSN), along with the DS state (up/down/booting/whatever).
And the MCP keeps track of the Authority Server Serial Numbers (ASSN),
along with AS state (up/down/booting/whatever).

The MCP can tell each DS to purge its IXFR database.
The MCP keeps track of what DSSN's are available for IXFR on a
server-by-server basis.
Every time the HMSN changes, the MCP cycles through the currently "up"
DS's, telling all but the "next" one to purge.
Thus, if there are "X" number of DS's up, there are as many as X-1
different IXFR's possible, based on the respective DSSN's.

If an Authority Server is down for a relatively brief period of time
(crash/reboot), there is a good chance it can get itself up to date by
an IXFR-ONLY.
It "goes around the circle", in the reverse order, looking for the DS
that can give it an IXFR-ONLY to bring it current.
Once it goes once around the circle without succeeding, it falls back
to an AXFR.
If it takes less time to reboot than it takes for "X-1" serial number
changes, the IXFR-ONLY will be possible.

Similarly, the HM->DS updates can also be via IXFR-ONLY, possibly
involving the use of DS->DS IXFR-ONLY for scalability.

The overall objective is to reduce the probability of requiring an
AXFR to near zero, and to also limit the overhead in terms of IXFR
state needed to achieve this.
Removing AXFR from anywhere but the edge, under normal operational
conditions, is important for big zones that change frequently.

If it were not for IXFR-ONLY, there would be no "clean" way to do this
- since IXFR fallback to pseudo-AXFR would interfere with the
attempts.

Observation:
A query for "what SNs do you have available for IXFR" would be the
equivalent", or similarly, an "can you IXFR me SN foo" query would
also do the trick.

However, both of these are non-atomic and introduce race conditions.
An IXFR-ONLY is atomic - if it is possible, the IXFR occurs; if not,
it says so.

Doing two queries (with non-trivial (network + cpu) delay between a
"can you IXFR me SN foo", and an "IXFR foo"), means a "yes" followed
by an unexpected AXFR, is the race condition.

Brian

---

From: Masataka Ohta <[email-masked]>
Date: Sat, 18 Jun 2011 21:37:52 +0900

> along with AS state (up/down/booting/whatever).

If you want to have complex solutions for a simple problem,
have a command-and-control infrastructure which can tell ASes
from which DS to IXFR.

If you don't,

> The MCP can tell each DS to purge its IXFR database.

never purge purposelessly.

						Masataka Ohta

---

From: Brian Dickson <[email-masked]>
Date: Sat, 18 Jun 2011 20:24:54 -0400

> have a command-and-control infrastructure which can tell ASes

You are in fact correct.

The only problem is that, telling an AS to IXFR from a DS, does not
necessarily mean that the DS is able to provide the requested IXFR.
Yes, most of the time it should be able to, and will.

However, the rare time that it can't, IXFR-ONLY is important to have,
to avoid fallback.

> never purge purposelessly.

Exactly. Thank you for demonstrating that you understand what I wrote.
(I realize that understand is not synonymous with agree.)
_______________________________________________
dnsext mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/dnsext

---

From: "W.C.A. Wijngaards" <[email-masked]>
Date: Mon, 20 Jun 2011 09:50:54 +0200

Hi Brian,

> mechanism it would have to do that.

What is wrong with closing the TCP socket?

Best regards,
   Wouter

---

From: Shane Kerr <[email-masked]>
Date: Mon, 20 Jun 2011 14:11:01 +0200

Ed,

On Fri, 2011-06-17 at 10:50 -0400, Edward Lewis wrote:
> At 16:15 +0200 6/17/11, OndÞej Sur wrote:
> 

Yes, the problem is CPU, memory, network bandwidth, and latency.

If you have multiple masters, and one is missing a particular version of
a zone for whatever reason (for example if the server needs to be
restarted and has a long fsck), then if your slave is unlucky and sends
IXFR to that master, then it will get the entire version of the zone,
even if other masters can happily supply just the differences in the
zone. If your zone is large or your connections are slow, this can be a
problem.

The reason this came up is that two operators (Ondrej and me) both
encountered the problem in the real world and it adversely affected our
operations.

IXFR-only specifies that you do not want the behavior to fall back to an
AXFR-style transfer, to avoid this issue. That's all.

In my mind, it turns IXFR into "Do What I Said" instead of "Do What You
Think Might Be Helpful". :)

--
Shane

---

From: Shane Kerr <[email-masked]>
Date: Mon, 20 Jun 2011 14:14:07 +0200

Wouter, 

> What is wrong with closing the TCP socket?

There's nothing especially wrong with that, and IIRC we did consider
this option when discussing IXFR-only over beers several years ago.

While you will not get the entire zone, you'll likely still get a lot of
extra data. Your operating system will be happily filling its TCP buffer
until your application notices that it is getting a AXFR-style transfer
and then closes the connection.

Granted this is probably only a few tens of MB at most, but I don't
think the changes on the slave side are much more complicated for
IXFR-only, and solve the problem.

--
Shane

---

From: Edward Lewis <[email-masked]>
Date: Mon, 20 Jun 2011 08:28:57 -0400

>encountered the problem in the real world and it adversely affected our

I get what you are saying.  But was this caused by the way the tools 
available worked or was there something wrong with the protocol that 
forced the tools to react this way?  That's my central question.

>AXFR-style transfer, to avoid this issue. That's all.

What I'm contending is that the fall back is done by an impementation 
and not part of the protocol.

>Think Might Be Helpful". :)

What I began to think about, because of this thread, is that there's 
no reason why server B - acting as a NOTIFY server - has to request 
an IXFR from server A - acting as a NOTIFY client - but collectively 
we have thought of it this way.  Not that NOTIFY is the only lead 
into an IXFR or AXFR, it's just one of the ways to illustrate this.

If server A tells server B "I have a new zone", the protocol does not 
tell B it has to go to A for an update.  It's conceivable that server 
B will try to get the update from C.  B can send AXFR/IXFR requests 
to C without a NOTIFY (timer expiration would be one reason now).

It sounds to me that IXFR-only is more like 
try-all-possible-sources-for-IXFR-first.  And that can be done in 
code.

NOTIFY/IXFR/AXFR are all distinct and independent mechanisms within 
the DNS architecture.  They aren't necessarily linked but 
implementers have linked them.  It's natural to do so - I'm not 
saying implementers are the cause of all evil here (although they 
generally are) - but again this is is maybe just a mass adoption of a 
bad assumption.

I'm overly entertained.

---

From: Edward Lewis <[email-masked]>
Date: Mon, 20 Jun 2011 08:35:01 -0400

>and then closes the connection.

I really think there's a misunderstanding of the AXFR-style IXFR.

Even if IXFR is on TCP, it's the same protocol that runs over UDP.  I 
once tried to write up an AXFR over UDP and in writing the draft 
learned that AXFR is fundamentally unable to run over UDP.  AXFR and 
IXFR responses are different.  You don't get an AXFR response from an 
IXFR query.

If you have an open TCP connection and see a IXFR query lead to an 
AXFR response, you have to see an IXFR response and AXFR query in 
there too - and also an SOA query/response.  Of else something was 
mis-reporting the AXFR-style IXFR.  Or maybe the IXFR server was 
buggy.

I'm overly entertained.

---

From: Mark Andrews <[email-masked]>
Date: Mon, 20 Jun 2011 22:54:20 +1000

> IXFR responses are different.  You don't get an AXFR response from an 

They differ by the QTYPE is the question section of the response otherwise
they are identical.

> mis-reporting the AXFR-style IXFR.  Or maybe the IXFR server was 

The following is legal with IXFR and results in a AXFR style IXFR response.

Query:	1 message
	Question:
	example.net IN IXFR 
	Answer:
	example.net IN SOA 1000 ....

Response:  1+ messages.
	Question:
	example.net IN IXFR 
	Answer:
	example.net IN SOA 1001 ....
	example.net IN NS ....

	Answer:
	....

	Answer:
	.....
	example.net IN SOA 1001 ....

---

From: Mark Andrews <[email-masked]>
Date: Mon, 20 Jun 2011 23:19:59 +1000

> forced the tools to react this way?  That's my central question.

It's a mixture of a known limitation of a optional extension of the
protocol, configuration and how the tools work (i.e. always doing the
extension).
 
> and not part of the protocol.

Fallback is part of the protocol.  There are 4 possible responses to a IXFR
request.
1. "up to date" or "switch to TCP".
2. a delta response over UDP/TCP with optional compression.
3. a "up to date" response over TCP (response serial  <= query serial).
4. a "AXFR style" response over TCP.
 
> into an IXFR or AXFR, it's just one of the ways to illustrate this.

The problem still exists even if the client only talks to the server
that has sent the notify.

> try-all-possible-sources-for-IXFR-first.  And that can be done in 

Every thing is done in code.  If you meant by the client, then no
it can't be done by the client cleanly as the client doesn't have
access to the data to do so.  The server needs to make the decision
on the clients behalf.

---

From: Edward Lewis <[email-masked]>
Date: Mon, 20 Jun 2011 09:22:07 -0400

>The following is legal with IXFR and results in a AXFR style IXFR response.

My assumption is that IXFR is a single query, single response message 
mechanism, so I would disagree with the example shown.  But I can see 
why there's a disagreement here, based on this passage from RFC 1995.

    Transport of a query may be by either UDP or TCP.  If an IXFR query
    is via UDP, the IXFR server may attempt to reply using UDP if the
    entire response can be contained in a single DNS packet.  If the UDP
    reply does not fit, the query is responded to with a single SOA
    record of the server's current version to inform the client that a
    TCP query should be initiated.

    Thus, a client should first make an IXFR query using UDP.  If the
    query type is not recognized by the server, an AXFR (preceded by a
    UDP SOA query) should be tried, ensuring backward compatibility.  If
    the query response is a single packet with the entire new zone, or if
    the server does not have a newer version than the client, everything
    is done.  Otherwise, a TCP IXFR query should be tried.

When reading the first paragraph I get that IXFR is one message 
query, one message response ("single DNS packet").  However, the "may 
attempt to reply" is not correct, the RFC should read "and will reply 
with one of three kinds of responses."  One being the "no IXFR 
possible", the other being the ordinary IXFR, and then the AXFR-style 
IXFR.  BTW, the latter is not a term used in the RFC.

 From that I have always assumed that the IXFR response is one DNS 
message.  Even on TCP.

The second paragraph adds to the confusion over fallback.  It talks 
about trying AXFR, but only if the IXFR is not understood.  And the 
has one line tacked on to the end of the paragraph of TCP IXFR being 
tried.

What needs to be fixed is how IXFR behaves on TCP.  It should be 
limited in its resource consumption.  That's what I'd go after.

I suppose I never assumed an implementation try IXFR over TCP. 
That's why I've never seen this issue in code or operations.

I think that the change here is not to cut off AXFR-stype IXFRs but 
to limit what a IXFR server considers an acceptable IXFR response. 
On TCP though you don't have a forced limit (512, EDNS0).

I'm overly entertained.

---

From: "W.C.A. Wijngaards" <[email-masked]>
Date: Mon, 20 Jun 2011 15:36:08 +0200

Hi Shane,

> this option when discussing IXFR-only over beers several years ago.

So, maybe this can solve the problem.  No RFC needed.

> and then closes the connection.

Your fast-update-slave will look at this very quickly, the first bytes
hold the first two records.  So it'll stop the transfer after a couple
msec.  At 10 Mbyte per second, that is about 64k or so that was sent?

> IXFR-only, and solve the problem.

It is larger than the notify-only option, but 64k hardly bursts the
traffic and time limits?  Since it it about fast update, the close TCP
socket is just as fast as ixfr-only, in ping-time, since it has to
decide when the response gets back.  It does have more traffic.

I was asking in case the close was 'unfriendly' in an uninteroperable
way.  The ixfr-only option is there to look pretty?

Best regards,
   Wouter

---

From: Josh Littlefield <[email-masked]>
Date: Mon, 20 Jun 2011 14:31:25 -0400

> why there's a disagreement here, based on this passage from RFC 1995.

That assumption is incorrect over TCP.  Whether communicating deltas, or
communicating an "AXFR-style" full zone response, IXFR can be multiple
packets over TCP.  The "framing" of the response data into packets for
IXFR and AXFR is the same.  And the difference between IXFR's use of UDP
and TCP is just the ability to use message framing in TCP.

Unfortunately RFC 1995 spells that out no more clearly than RFC 1034,
but we've had interoperable implementations that understood the proper
framing of IXFR for more than a decade.

>
> I think that the change here is not to cut off AXFR-stype IXFRs but to
> limit what a IXFR server considers an acceptable IXFR response. On TCP
> though you don't have a forced limit (512, EDNS0).

I don't see why you would limit anything.  TCP framing does have a
per-message limit of 16K, due to the 2-byte length field.  But IXFR
already easily supports sending the full zone response, and there is
nothing wrong with that.

The issue addressed by the draft in question is that you might sometimes
prefer not to get that response (when the zone has 1M RRs) until you've
exhausted other potential sources of changesets.

Since you can tell the difference between a delta response and a full
response by the second answer section record, it is currently possible
to just shut the TCP connection if the response is not the type you
prefer.  That's a bit ugly, and the server sees it as an aborted IXFR.

-josh

-- 
Josh Littlefield                                  Cisco Systems, Inc.
[email-masked]                             1414 Massachusetts Avenue
Phone: +1 978-936-0001                     Boxborough, MA  01719-2205

---

From: Josh Littlefield <[email-masked]>
Date: Mon, 20 Jun 2011 14:44:07 -0400

> already easily supports sending the full zone response, and there is

FYI: Of course, I meant 64K above.

-josh

---

From: Paul Vixie <[email-masked]>
Date: Mon, 20 Jun 2011 21:44:15 +0000

> Date: Mon, 20 Jun 2011 14:44:07 -0400
> From: Josh Littlefield <[email-masked]>
> 

in fairness to you, anything above 16K is useless, since compression
pointers are only 14 bits in size.  the change in header amortization
if you go above 16K is negligible compared to the loss of compression
reach.  so while you didn't mean 16K, it's actually a good TCP/53 limit
for things like IXFR/AXFR.
_______________________________________________
dnsext mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/dnsext

---

From: Edward Lewis <[email-masked]>
Date: Mon, 20 Jun 2011 18:08:40 -0400

>framing of IXFR for more than a decade.

Well, that clears things up.

I then prefer leaving the IXFR-only proposal as is in the draft. 
Slamming the connection closed seems wasteful, especially in the 
cases when you may have already received the whole zone while 
deciding to look elsewhere.

I don't think there's text that needs fixing.  What got me was that I 
see IXFR as a UDP thing and think of it in constrained environments. 
The last paragraph of section 5 talks more (than RFC 1995 did) about 
IXFR over TCP.

I'm overly entertained.

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 21 Jun 2011 12:10:11 +0900

> However, the rare time that it can't, IXFR-ONLY is important to have,

OK.

> (I realize that understand is not synonymous with agree.)

Are you saying that:

   1) plain IXFR is fine

   2) some operators can't properly operate a little complicated
      IXFR with a dial to tweak purge behavior, which rarely causes
      bandwidth inefficient IXFR.

   3) to help the poor operators, yet another dial to tweak IXFR
      behavior MUST be added

?

Shane Kerr wrote:

> encountered the problem in the real world and it adversely affected
> our operations.

Are your examples essentially different from Brian Dickson's?

							Masataka Ohta

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 21 Jun 2011 12:10:36 +0900

Josh Littlefield wrote:

> The "framing" of the response data into packets for

It was intentional, because I didn't want to be involved in a
discussion on how AXFR should be clarified only to delay
publication of IXFR.

> framing of IXFR for more than a decade.

With RFC1995, anyone who can implement AXFR should be able to
implement IXFR, which should be just enough as an incremental
specification. :-)

						Masataka Ohta

---

From: Florian Weimer <[email-masked]>
Date: Tue, 21 Jun 2011 11:48:51 +0000

> and then closes the connection.

Is it possible to reach that conclusion early during the connection?
Would the window be fully opened at that point?

Anyway, if you close a socket while there is pending data to read,
modern TCP implementations are expected to abort the connection.  So all
that data will be discarded, and the sender can be notified before the
data currently in flight is acknowledged.

-- 
Florian Weimer                <[email-masked]>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99
_______________________________________________
dnsext mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/dnsext

---

From: Edward Lewis <[email-masked]>
Date: Tue, 21 Jun 2011 09:52:31 -0400

>    3) to help the poor operators, yet another dial to tweak IXFR

Ohta-san,

Times have changed since the days of RFC 1995, the requirements have 
changed.  Although the network has seen an increase in capacity (bits 
per second) the distance between endpoints has actually increased (we 
send zones further and further from home).

In addition the size of zones has grown greatly and the number of 
large zones has grown greatly.  I know that 15 years ago, even COM 
was under a million records.  Now a million records is an average 
sized TLD.

And thanks to dynamic update (which was published after RFC 1995), 
the role of IXFR has grown to surpass AXFR as the standard transfer 
mechanism.

It makes sense for a server, when told there is a new version, to do 
what it can to efficiently seek a new version.  It makes sense that a 
server would, when it knows that there is a new version to be had, 
contact all the places where it can get just the update before 
locking on to an AXFR.

 From a recovery perspective, what if the master server loses it's 
increment memory and cannot do AXFR?  Each slave might be called on 
to do AXFRs when it would be more efficient for one slave to do the 
AXFR and then answer IXFR requests until the system stabilizes.

I'm overly entertained.

---

From: Masataka Ohta <[email-masked]>
Date: Wed, 22 Jun 2011 00:08:34 +0900

Edward Lewis wrote:

> Times have changed since

See 2) to see an example of the current situation.

> And thanks to dynamic update (which was published after RFC 1995), the
> role of IXFR has grown to surpass AXFR as the standard transfer mechanism.

I'm afraid you don't know the history of DNSIND.

> increment memory and cannot do AXFR? Each slave might be called on to do 
> AXFRs when it would be more efficient for one slave to do the AXFR and 
> then answer IXFR requests until the system stabilizes.

What if what?

							Masataka Ohta

---

From: Edward Lewis <[email-masked]>
Date: Tue, 21 Jun 2011 11:33:49 -0400

>See 2) to see an example of the current situation.

It would be a real shame to make something operationally smooth, eh?

>I'm afraid you don't know the history of DNSIND.

RFC 1995 - August 1996
RFC 2136 - April 1997.

Didn't say that they weren't discussed closely in time, just 
referenced the publish date.

>>  then answer IXFR requests until the system stabilizes.

"Each slave might be called on to do AXFRs when it would be more 
efficient for one slave to do the AXFR and then answer IXFR requests 
until the system stabilizes."

Which is what happens today.  The desire is to make it better.

I'm overly entertained.

---

From: Masataka Ohta <[email-masked]>
Date: Wed, 22 Jun 2011 06:47:04 +0900

Edward Lewis wrote:

> Didn't say that they weren't discussed closely in time, just referenced 
> the publish date.

Irrelevant.

> until the system stabilizes."

Wrong.

What you are assuming is a crash of the master database along
with its local copies, which is a very serious but classical
problem with a classical solution.

First, note that it should not occur and occurs very rarely,
if the database is important and operated with reasonable
care. But, yes, it still occurs.

The most important thing to do, then, is restoration of
recent updates, not past history. You can't assume the
recent most data, which is most fragile, is magically
retained.

Classically, important database is equipped with restoration
mechanism with snapshots and journal, which happen to be
information stored in secondaries through IXFR.

But, because of IXFR delay, the recent most updates may be
retained only in dynamic update clients.

So, operators first restore recent data from a secondary with
the recent most version number. Rational operators should and
will restore not only the current most database but also
all the history, which is not absolutely necessary but makes
IXFR-only unnecessary.

Then, as much journal information as possible is collected from
dynamic update clients, sorted by time and used for restoration,
which is a manual operation, only after which, the master server
start operating again.

During the restoration of the master, secondaries have enough
time to synchronize there databases with efficient IXFR that
they and the master should have the same version, which means
there is no room for IXFR-only after the restoration of the
master, even if history of the master is, quite irrationally,
not restored.

> Which is what happens today.

You are saying DNS operators know nothing about the very basic
(thus, classical) points of database maintenance.

I hope its your imaginary example not happening anywhere in the
real world today.

						Masataka Ohta

---

From: Edward Lewis <[email-masked]>
Date: Tue, 21 Jun 2011 20:34:24 -0400

>(thus, classical) points of database maintenance.

You have a vivid imagination.

What I am saying is that operators, other than me, have proposed 
updating the tired old definition of IXFR based on what is see in 
practice today.  And I am agreeing that there is a reason to do so. 
Times have changed and older definitions sometimes need to be brought 
up to current standards.

In addition, you yourself said yesterday that parts of RFC 1995 were 
left somewhat unfinished so as to not have to redefine AXFR.  Well, 
now AXFR has been redefined, so the time has come to complete IXFR's 
documentation.  Just to refresh...

>discussion on how AXFR should be clarified only to delay

So, it seems that you have laid down a justification to now update RFC 1995.
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis
NeuStar                    You can leave a voice message at +1-571-434-5468

I'm overly entertained.

---

From: Masataka Ohta <[email-masked]>
Date: Mon, 27 Jun 2011 22:58:15 +0900

Edward Lewis wrote:

> practice today.

Operators may find problems not properly handled by the current
protocol and may also propose a solution.

However, it does not mean that the problems are real nor that the
solution is the best one.

Thus, we are having an operational discussion, here.

> And I am agreeing that there is a reason to do so.

You may or may not join the discussion.

However, a purely imaginary example of yours with unrealistic
assumptions is merely harmful.

> Times 
> have changed and older definitions sometimes need to be brought up to 
> current standards.

Sure. If only there were convincing real world examples...

> left somewhat unfinished

No.

						Masataka Ohta