Subject: Re: [DNSOP] Mirja Kühlewind's Discuss on draft-ietf-dnsop-session-signal-12: (with DISCUSS and COMMENT)

From: Stuart Cheshire <[email-masked]>
Date: Wed, 01 Aug 2018 00:48:29 -0700

On 30 Jul 2018, at 13:19, Mirja Kühlewind <[email-masked]> wrote:

> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------

I’m responding to the “DISCUSS” items right now. I’ll get to the “COMMENT” items shortly.

> 1) In addition to the bullet point in the 6.2 that was flagged by Spencer, I
> would like to discuss the content of section 5.4.  (DSO Response Generation). I
> understand the desire to optimize for the case where the application knows that
> no data will be sent as reply to a certain message, however, TCP does not have
> a notion of message boundaries and therefore cannot and should not act based on
> the reception of a certain message. Indicating to the TCP that an ACK can be
> set immediately in an specific situation is also problematic as ACK processing
> is part of the TCP's internal machinery. However, why it is important at all
> that an TCP-level ACK is send out fast than the delayed ACK timer? The ACK
> receiver does not expose the information when an ACK is received to the
> application and the delayed ACK timer only expires if no further data is
> received/send by the ACK-receiver, therefore this optimization should not have
> any impact in the application performance. I would just recommend to remove
> this section and any additional discussion about delayed ACKs.
> 
> Please note that the problem described in [NagleDA] only occurs for
> request-response protocols where no further request can be sent before the
> response is received. This is not the case in this protocol (as pipelining is
> supported).

The problem here is not further requests, it’s further responses. Consider a client that subscribes for mDNS relay service <https://tools.ietf.org/html/draft-ietf-dnssd-mdns-relay-01>.

If the server gets an mDNS packet and relays it, Nagle blocks relaying of a further mDNS packet until an ack is received. On a campus GigE backbone with sub-millisecond round-trip times, this potentially delays the relaying of a subsequent mDNS packet for up to 200 ms. That’s a long time on a sub-millisecond network. If the client were to send a reply to the first relayed mDNS packet, then TCP would piggyback its ack on that data packet, and Nagle would then free the server to relay the next mDNS packet.

The optimization advocated here is the observation that if a networking API were to allow the server to explicitly indicate an empty reply, then that lets the TCP stack know that it doesn’t need to wait 200 ms in the hope that it can piggyback its ack on an outbound data packet.

Without this, people are tempted to set TCP_NODELAY, which is worse overall for the network.

> 2) Further regarding keep-alives:
> in sec 6.5.2: "For example, a hypothetical keepalive interval
>   value of 100ms would result in a continuous stream of at least ten
>   messages per second, in both directions, to keep the DSO Session
>   alive."
> 
> This does not seems correct. There should be at max one keep-alives message in
> flight. Thus the keep-laives timer should only be restarted after the
> keep-alive reply was received.

On a campus GigE backbone with sub-millisecond round-trip times, even a hypothetical keepalive interval value of 100ms would still have only one keep-alive message in flight at a time. But it would still be an unreasonable keepalive interval.

>   And, in this extreme example, a single packet loss and
>   retransmission over a long path could introduce a momentary pause in
>   the stream of messages, long enough to cause the server to
>   overzealously abort the connection."
> 
> This doesn't really make sense to me: As I said, TCP will retransmit and the
> keep-alive timer should not be running until the reply is received. If you want
> to abort the connection based on keep-alives quickly before the TCP connection
> indicates you a failure, you need to wait at minimum for an interval that is
> larger than the TCP RTO (with is uaually 3 RTTs) which means you basically need
> to know the RTT.

The point of this text is to illustrate that a keepalive interval value of 100ms would be unreasonable. I think you would agree with that. This is to support why the immediately following text mandates a minimum keepalive interval of ten seconds.

> Also sec 7.1: "If the client does not generate the
>      mandated keepalive traffic, then after twice this interval the
>      server will forcibly abort the connection."
> Why must the server terminate the connection at all if the client refuses to
> send keep-alives? Isn't that what the inactivity timer is meant for? Usually
> only the endpoint that initiates the keep-alive should terminate the connection
> if no response is received.

A client cannot refuse to send keep-alives. A connection with an active mDNS relay subscription is never considered “inactive”, but a server may still require reasonable keep-alives to verify that the client is still there.

> 3) There is another contraction regarding the inactive timer:
> Sec 6.2 say
>   "A shorter inactivity timeout with a longer keepalive interval signals
>   to the client that it should not speculatively keep an inactive DSO
>   Session open for very long without reason, but when it does have an
>   active reason to keep a DSO Session open, it doesn't need to be
>   sending an aggressive level of keepalive traffic to maintain that
>   session."
> which indicates that the client may leave the session open longer than
> indicated by the inactive timer of the server. However section 7.1.1 say that
> the client MUST close the connection when the timer is expired.

A connection with an active mDNS relay subscription is never considered “inactive”, because there is still active client/server state, even if no traffic is flowing. A server may still require reasonable keep-alives to verify that the client is still there.

Stuart Cheshire

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Wed, 1 Aug 2018 14:02:05 +0200

Hi Stuart,

please see below.

> Am 01.08.2018 um 09:48 schrieb Stuart Cheshire <[email-masked]>:
> 

I unterstand the point, I just don’t really think that this is a problem that is specific to this use case and therefore should not be necessarily discussed in tis document (given the problem is quite complex). However, I guess that could be good input to taps, given taps is working on a message-based interface on top of TCP.

> Without this, people are tempted to set TCP_NODELAY, which is worse overall for the network.

Not sure. In the described scenarios this might actually not be a bad think to do.

> On a campus GigE backbone with sub-millisecond round-trip times, even a hypothetical keepalive interval value of 100ms would still have only one keep-alive message in flight at a time. But it would…

Not sure if that is a unreasonable keep-alive in a GigE backbone. I would actually hope that you don’t need a keep-alive mechanism at all in those scenarios but it depends if there are any middleboxes and how quickly their state expires. Given you have a campus network, you might know what the timeout are and set the keep-alive interval respectively. Maybe that’a better advise to give.

My problem with the example text above it that is seems to indicate that you just send a keep-alives very x time units while you need to wait for the response before restarting the timer. This needs to be clarified. However as I said, I’m not certain about the actual value fo this section as all, as it does not seems the right document to discuss these more general issue.

> The point of this text is to illustrate that a keepalive interval value of 100ms would be unreasonable. I think you would agree with that.

Yes. I understood that, however, for me this illustration was rather confusing. For me something like "the keep-alives interval should not be chosen to low to reduce network load and must be sufficiently larger than the RTT to avoid server termination if the keep-alive gets lost and needs to be retransmitted“ would be enough.

> This is to support why the immediately following text mandates a minimum keepalive interval of ten seconds.
> 

Ah, thanks I don’t think this case was explain in the text (or I missed it) please clarify. May also provide in general more reasoning why and that the client is required to send the keep-alives as requested. At the beginning of the doc it seemed more that the keep-alive interval is more a recommendation than a requirement and it is important to understand that to correctly understand the rest of the doc.

> A connection with an active mDNS relay subscription is never considered “inactive”, because there is still active client/server state, even if no traffic is flowing. A server may still require reas…

However, the cited text above says 
"should not speculatively keep an inactive DSO Session open for very long without reason, but when it does have an active reason to keep a DSO Session open,“
which explicitly talks about keeping INACTIVE session open speculatively for a longer time than the inactivity timeout.

Mirja

---

From: Ted Lemon <[email-masked]>
Date: Thu, 2 Aug 2018 01:39:04 -0400

On Wed, Aug 1, 2018 at 8:02 AM, Mirja Kuehlewind (IETF) <[email-masked]
> wrote:

> > The problem here is not further requests, it’s further responses.
> Consider a client that subscribes for mDNS relay service <
> https://tools.ietf.org/html/draft-ietf-dnssd-mdns-relay-01>.
> >
> > If the server gets an mDNS packet and relays it, Nagle blocks relaying
> of a further mDNS packet until an ack is received. On a campus GigE
> backbone with sub-millisecond round-trip times, this potentially delays the
> relaying of a subsequent mDNS packet for up to 200 ms. That’s a long time
> on a sub-millisecond network. If the client were to send a reply to the
> first relayed mDNS packet, then TCP would piggyback its ack on that data
> packet, and Nagle would then free the server to relay the next mDNS packet.
> >
> > The optimization advocated here is the observation that if a networking
> API were to allow the server to explicitly indicate an empty reply, then
> that lets the TCP stack know that it doesn’t need to wait 200 ms in the
> hope that it can piggyback its ack on an outbound data packet.
>
> I unterstand the point, I just don’t really think that this is a problem
> that is specific to this use case and therefore should not be necessarily
> discussed in tis document (given the problem is quite complex). However, I
> guess that could be good input to taps, given taps is working on a
> message-based interface on top of TCP.
>
> >

The problem is that we can't give good advice on this—in some scenarios,
TCP_NODELAY probably won't cause any harm.   My personal opinion on this
(which Stuart has not confirmed) is that indeed this is a general problem,
and that it does make sense to raise it in taps rather than trying to solve
it here.   Consequently, I've updated the section as follows:

## Flow Control Considerations

Because unacknowledged DSO messages do not generate an immediate response
from the responder, if
there is no other traffic flowing from the responder to the initiator, this
can result in a
200ms delay before the TCP acknowledgment is sent to the initiator
{{NagleDA}}.  If the
initiator has another message pending, but has not yet filled its output
buffer, this can delay
the delivery of that message by more than 200ms.  In many cases, this will
make no difference.
However, implementors should be aware of this issue.  Some operating
systems offer ways to
disable the 200ms TCP acknowledgment delay; this may be useful for
relatively low-traffic
sessions, or sessions with bursty traffic flows.

> > On a campus GigE backbone with sub-millisecond round-trip times, even a
> hypothetical keepalive interval value of 100ms would still have only one
> keep-alive message in flight at a time. But it would still be an
> unreasonable keepalive interval.
>
> Not sure if that is a unreasonable keep-alive in a GigE backbone. I would
> actually hope that you don’t need a keep-alive mechanism at all in those
> scenarios but it depends if there are any middleboxes and how quickly their
> state expires. Given you have a campus network, you might know what the
> timeout are and set the keep-alive interval respectively. Maybe that’a
> better advise to give.
>

I think this text (currently in the document) addresses this case:

A corporate DNS server that knows it is serving only clients on the
internal
network, with no intervening NAT gateways or firewalls, can impose a higher
keepalive interval, because frequent DSO keepalive traffic is not required.

Indeed, in this case I agree that a fairly long number could be used.
 However, bear in mind that keepalive not only catches lost state in the
network, but also lost state on the endpoint, so getting rid of it entirely
means that a connection might appear open for a very long time with no
traffic flowing and no attempt to reconnect.   Sending regular keepalives
ensures that if state is lost on either endpoint, we get a timely TCP RST
when the next keepalive arrives and finds no open connection waiting for it.

However, I think that 100ms is pretty clearly too short for this
application—we aren't trying to do VRRP, and a packet that would produce a
user-visible delay would also trigger the TCP reset for the broken
connection.

My problem with the example text above it that is seems to indicate that
> you just send a keep-alives very x time units while you need to wait for
> the response before restarting the timer. This needs to be clarified.
> However as I said, I’m not certain about the actual value fo this section
> as all, as it does not seems the right document to discuss these more
> general issue.
>

I think I see where the confusion is.   The current text says that when a
client receives a keepalive timeout TLV, it resets the keepalive timer.
 This only makes sense when the server is sending the client an
unacknowledged Keepalive message.   I've made the following change, which I
think should clear up the confusion:

In the case of the keepalive timer, the handling of the received value is
straightforward. When
a client receives a server-initiated message with the Keepalive TLV as its
primary TLV, it resets the
keepalive timer.  Whenever it receives a Keepalive TLV from the server,
either in a server-initiated
message or a reply to its own client-initiated Keepalive message, it
updates the keepalive interval
for the DSO Session. The new keepalive
interval indicates the maximum time that may elapse before another message
must be sent or
received on this DSO Session, if the DSO Session is to remain alive.
If the client receives a response to a keepalive message that specifies a
keepalive interval
shorter than the current keepalive timer, the client MUST immediately send
a Keepalive message.
However, this should not normally happen in practice: it would require that
Keepalive interval
the server be shorter than the round-trip time of the connection.

> > The point of this text is to illustrate that a keepalive interval value
> of 100ms would be unreasonable. I think you would agree with that.
>
> Yes. I understood that, however, for me this illustration was rather
> confusing. For me something like "the keep-alives interval should not be
> chosen to low to reduce network load and must be sufficiently larger than
> the RTT to avoid server termination if the keep-alive gets lost and needs
> to be retransmitted“ would be enough.
>

I think the confusion here is rooted in the same problem that I (hope) I
fixed with the above text.

> > A client cannot refuse to send keep-alives. A connection with an active
> mDNS relay subscription is never considered “inactive”, but a server may
> still require reasonable keep-alives to verify that the client is still
> there.
>
> Ah, thanks I don’t think this case was explain in the text (or I missed
> it) please clarify. May also provide in general more reasoning why and that
> the client is required to send the keep-alives as requested. At the
> beginning of the doc it seemed more that the keep-alive interval is more a
> recommendation than a requirement and it is important to understand that to
> correctly understand the rest of the doc.
>

This is the current text that covers this:

If a client disconnects from the network abruptly,
without cleanly closing its DSO Session,
perhaps leaving a long-lived operation uncancelled,
the server learns of this after failing to
receive the required DSO keepalive traffic from that client.
If, at any time during the life of the DSO Session,
twice the keepalive interval value (i.e., 30 seconds by default) elapses
without any DNS messages being sent or received on a DSO Session,
the server SHOULD consider the client delinquent,
and SHOULD forcibly abort the DSO Session.

I think that this means that if a non-infinite keepalive interval has been
negotiated, the client has to send keepalives.   Of course, the spec does
allow the client and server to negotiate an infinite keepalive interval, in
which case no keepalives are ever sent; in this sense, keepalives are
indeed optional.

> > A connection with an active mDNS relay subscription is never considered
> “inactive”, because there is still active client/server state, even if no
> traffic is flowing. A server may still require reasonable keep-alives to
> verify that the client is still there.
>
> "should not speculatively keep an inactive DSO Session open for very long
> without reason, but when it does have an active reason to keep a DSO
> Session open,“
> which explicitly talks about keeping INACTIVE session open speculatively
> for a longer time than the inactivity timeout.
>

I added the following text after the text you quoted.

An example of this would be a client that has subscribed to DNS Push
notifications: in this case, the client is not sending any traffic to the
server, but the session is not inactive, because there is a pending request
to the server to receive push notifications.

I have not finished reviewing all your comments, but I wanted to
particularly review your conversation with Stuart before going through the
stuff you and Stuart didn't discuss, so that I'd have that context.   I
will send a second response to your message that covers the issues I
haven't yet covered here.

---

From: Ted Lemon <[email-masked]>
Date: Thu, 2 Aug 2018 02:35:41 -0400

On Mon, Jul 30, 2018 at 4:19 PM, Mirja Kühlewind <[email-masked]>
wrote:

[Skipping the DISCUSS, which I hope we have already addressed.]

1) sec 3: I really find it a bit strange that there is normative language
> about
> error handling (as well as in the "same service instance" definition part)
> in
> the terminology section. Maybe move those paragraphs somewhere else...?
> Also
> the part about "long-lived operations" and messages types provides far more
> information than just terminology and I would recommend to also move it
> into an
> own section or maybe just have it as part of the intro.
>

I happen to agree with this criticism, and this issue was discussed, but we
decided not to make this change, because it would have been a fairly
massive overhaul of the document.

> 2) Maybe call section 5 "Protocol specification" instead of "Protocol
> details"...?
>

That would imply that you might not have to read the rest of the document.

> 3) Sec 5.1: "DSO messages MUST be carried in only protocols and in
> environments
>    where a session may be established according to the definition given
>    above in the Terminology section (Section 3)."
> I don't get this. Which part of section 3? Given section 3 is on
> terminology
> and this is a normative statement, I would recommend to spell out here
> explicitly what is meant. Do you mean the protocol must be
> connection-oriented,
> reliable, and providing in-order delivery? Any thing else? However, given
> that
> you say two paragraphs onwards that this spec is only applicable for the
> use
> with TCP and TLS/TCP, do you even need to specify these requirements
> normatively?
>

We've since updated this text, but it didn't address your concern.   I've
updated it further as follows:

DSO messages MUST NOT be carried in protocols and
environments where a session can't be established.   For example,
DNS over plain UDP {{?RFC0768}} is not appropriate since it does not provide
in-order message delivery, and, in the presence of NAT gateways and
firewalls
with short UDP timeouts, it cannot provide a persistent bi-directional
communication channel unless an excessive amount of DSO keepalive traffic
is used.
UDP also doesn't provide a way to mark the start of a session and the end
of a session.

> 4) sec 5.1 "It is a common
>    convention that protocols specified to run over TLS are given IANA
>    service type names ending in "-tls"."
> Not sure this is true. Isn't it usually just an "s" at the end? Or with
> registry are you talking about?
>

https://www.iana.org/assignments/service-names-port-numbers/
service-names-port-numbers.xml

I think the "s" suffix is more for URIs and /etc/services (which is
informal).   I've added a reference to the registry.

> 5) sec 5.1: "In some environments it may be known in advance by external
> means
>    that both client and server support DSO, ..."
> I guess the client and server also need to know if TLS is supported or not.
> Maybe spell this out as well.
>

This is specified in DNS-over-TLS.   Additionally, for example
draft-ietf-dnssd-mdns-relay explicitly requires TLS.   It's also covered in
the paragraph before this one.  This paragraph is actually a separate
topic: whether the client can assume the server supports DSO and vice
versa, as opposed to negotiating and falling back.

> 6) sec 5.1: "... therefore either
>    client or server may be the initiator of a message."
> Maybe s/initiator of a message/initiator of a message exchange/
>

It's not necessarily an exchange.

> 7) sec 5.1.2: "Having initiated a connection to a server, possibly using
> zero
> round-
>    trip TCP Fast Open and/or zero round-trip TLS 1.3, a client MAY send
>    multiple response-requiring DSO request messages to the server in
>    succession without having to wait for a response to the first request
>    message to confirm successful establishment of a DSO session."
> Why is the ability to send more than one request related to TCP Fast
> Open/TLS1.3 0-RTT? These are two independent mechanisms to speed up
> processing.
> Mentioning TCP Fast Open/TLS1.3 0-RTT here is rather confusing.
> Respectively I
> also don't think that the sentence: "Similarly, DSO supports zero
> round-trip
> operation." is describing quite the same.
>
> 8) Further please provide references to TCP Fast Open and TLS1.3 and maybe
> rephrase this paragraph to use normative language: "Caution must be taken
> to
> ensure that DSO messages sent before the
>    first round-trip is completed are idempotent, or are otherwise immune
>    to any problems that could be result from the inadvertent replay that
>    can occur with zero round-trip operation."
> Maybe just:
> "DSO messages sent with TLS1.3 0-RTT before the TLS handshake is completed
> or
> in TCP SYN data with use of TCP Fast Open MUST be idempotent." However,
> this is
> actually already required by TLS1.3 and TFO, so there is after all no need
> to
> just rephrase this requirement here (at least not normatively). I think it
> would be more useful for every DSO message type to specify if it can be
> sent in
> 0-RTT or not and require this for specification of future TLVs.
>

I agree the text is a bit awkward: this section is only talking about 0rtt
sessions.   I've updated it to say this:

DSO permits zero round-trip operation
using TCP Fast Open {{?RFC7413}}
and TLS 1.3 {{?I-D.ietf-tls-tls13}}
to reduce or eliminate
round trips in session establishment.

A client MAY send multiple response-requiring DSO messages using TCP fast
open or TLS 1.3 early data,
without having to wait for a response to the first request message
to confirm successful establishment of a DSO session.

Also, I added the following to security considerations based on ekr's
related comment:

9) sec 5.1.3: "In cases where a DSO session is terminated on one side of a
>    middlebox, and then some session is opened on the other side of the
>    middlebox in order to satisfy requests sent over the first DSO
>    session, any such session MUST be treated as a separate session."
> This sentence seems a bit non-sensical, which probably isn't great for a
> normative sentence. If a session is terminated and open of the other end,
> doesn't that mean that you have two sessions?
>

The point is to exclude the implementation doing something clever, which
would indeed be nonsensical.

> 10) sec 5.1.3: "A middlebox that is not doing a strict pass-through will
> have
> no way to
>    know on which connection to forward a DSO message, and therefore will
>    not be able to behave incorrectly."
> I'm not sure I understand this sentence. Can you clarify?
>

We added this in response to a similar comment:

To illustrate the above, consider a network where a middlebox
terminates one or more TCP connections from clients and multiplexes the
queries therein over a single TCP connection to an upstream server.
The DSO messages and any associated state are specific to the individual
TCP connections.  A DSO-aware middlebox MAY in some circumstances be
able to retain associated state and pass it between the client and
server (or vice versa) but this would be highly TLV-specific.  For
example, the middlebox may be able to maintain a list of which clients
have made Push Notification subscriptions {{?I-D.ietf-dnssd-push}} and
make its own subscription(s) on their behalf, relaying any subsequent
notifications to the client (or clients) that have subscribed to that
particular notification.

> 11) As already briefly mentioned by Ben, there is quite some redundant
> text in
> sec 5 (with 5.2) for handling of message IDs and TLVs. Given this text is
> normative, I would really recommend to only specify it clearly once. Please
> also check the rest of the doc further things that are specified
> normatively
> multiple times. It usually makes it must clearer to specify it only once,
> at
> least normatively, at the appropriate position in the doc.
>

I think you may be seeing two different places where the message id must be
zero as saying the same thing, but they are not.   Or maybe I'm just
confused, but I don't see a change to make here.

> 12) sec 5.3.1: "When a DSO unacknowledged message is unsuccessful for some
> reason, .." What does unsuccessful mean here? Can you clarify?
>

This has been updated in response to a previous comment:

When an unacknowledged DSO message type is received (MESSAGE ID field is
zero), the receiver SHOULD already be expecting this DSO message type.
{{unrecognized}} describes the handling of unknown DSO message types.
Parsing
errors MUST also result in the receiver aborting the connection.
When an unacknowledged DSO
message of an unexpected type is received, the receiver should abort the
connection.
Other internal errors processing the unacknowledged DSO message are
implementation dependent as to whether the connection should be aborted
according to the severity of the error.

13) sec 6.5.2: "A corporate DNS server that knows it is serving only
> clients on
> the
>    internal network, with no intervening NAT gateways or firewalls, can
>    impose a higher keepalive interval, because frequent keepalive
>    traffic is not required."
> I guess in this scenario it is probably most appropriate to not send any
> keep-alives…
>

I think I already addressed this in my response to your DISCUSS—please let
me know if you still see an issue here.

> 14) sec 6.6: "   o  The server application software terminates unexpectedly
> (perhaps
>       due to a bug that makes it crash)."
> This bullet point does not really make sense to me because at that time
> when
> the app is crashed there is no way for the server anymore to perform any
> actions.
>

Right, the connection is terminated as a result of the server crashing, not
as the result of the server noticing that it crashed and deciding to
terminate the connection. :)

> 15) sec 7.1: "When a client is sending its second and subsequent Keepalive
> DSO
>    requests to the server, the client SHOULD continue to request its
>    preferred values each time. "
> I don't understand the SHOULD here.. what else should be client put in
> these
> field instead...?
>

It could repeat back to the server what the server sent it.

> 16) sec 7.1.2: "Once a DSO Session has been established, if either
>    client or server receives a DNS message over the DSO Session that
>    contains an EDNS(0) TCP Keepalive option, this is a fatal error and
>    the receiver of the EDNS(0) TCP Keepalive option MUST forcibly abort
>    the connection immediately."
> This is normatively specified multiple time (3?) in the doc. Please
> consider to
> only specify it once where most appropriate (probably section 7.1.2)
>

It appears in two places, not three, and in both places it makes sense to
say it.   The requirements are consistent in both places.   I would rather
say it in both places than have someone skim the document and miss it.   If
we had two places in the document where that were all that was said, I'd
agree, but both places where this is specified are contextually
appropriate—I don't agree that 7.1.2 is better.

> 16) sec 7.1: "The Keepalive TLV is not used as an Additional TLV."
> This is redundant with the normative sentence in the next paragraph. Maybe
> just
> remove this sentence...?
>

WFM!

> 17) +1 to Ben's discuss regarding the reconnection of clients. A TCP RST
> can be
> sent for many reasons and waiting for an hour seems not appropriate. I
> would
> rather recommend to log an error and directly try to reconnect.
>

 I haven't seen Ben's response to what Tom said about this, but the context
for this is that the client aborted the connection because it detected an
implementation error on the server.   Waiting an hour is actually pretty
optimistic.