Subject: statement regarding keepalives

From: Kent Watsen <[email-masked]>
Date: Fri, 13 Jul 2018 00:37:59 +0000

Dear TSVAREA,

The folks working with the BBF asked the NETMOD WG to consider modifying draft-ietf-netconf-netconf-client-server to support TCP keepalives [1].  However, it is unclear what IETF's position is on the use of keepalives, especially with regards to keepalives provided in protocol stacks (e.g., <some-app> over HTTP over TLS over TCP).

After some discussion with Transport ADs (Spencer and Mijra) and the TLS ADs (Eric and Ben), the following draft statement has been crafted.  Spencer and Mijra have requested TSVAREA critique it before, perhaps, developing a consensus document around it in TSVWG.

It would be greatly appreciated if folks here could review and provide comments on the draft statement below.  The scope of the statement can be increased or reduced as deemed appropriate. 

[1] https://mailarchive.ietf.org/arch/msg/netconf/MOzcZKp2rSxPVMTGdmmrVInwx2M 

Thanks,
Kent (and Mahesh) // NETCONF chairs

===== STATEMENT =====

When the initiator of a networking session needs to maintain a persistent connection [1], it is necessary for it to periodically test the aliveness of the remote peer.  In such cases, it is RECOMMENDED that the aliveness check happens at the highest protocol layer possible that is most meaningful to the application, to maximize the depth of the aliveness check.  

E.g., for an HTTPS connection to a simple webserver, HTTP-level keepalives would test more aliveness than TLS-level keepalives.  However, for a webserver that is accessed via a load-balancer that terminates TLS connections, TLS-level aliveness checks may be the most meaningful check that could be performed.

In order to ensure aliveness checks can always occur at the highest protocol layer, it is RECOMMENDED that protocol designers always include an aliveness check mechanism in the protocol and, for client/server protocols, that the aliveness check can be initiated from either peer, as sometimes the "server" is the initiator of the underlying networking connection (e.g., RFC 8071).

Some protocol stacks have a secure transport protocol layer (e.g., TLS, SSH, DTLS) that sits on top of a cleartext protocol layer (e.g., TCP, UDP).  In such cases, it is RECOMMENDED that the aliveness check occurs within protection envelope afforded by the secure transport protocol layer.  In such cases, the aliveness checks SHOULD NOT occur via the cleartext protocol layer, as an adversary can block aliveness check messages in either direction and send fake aliveness check messages in either direction.

[1] While reasons may vary for why the initiator of a networking session feels compelled to maintain a persistent connection.  If the session is primarily quiet, and the use case can cope with the additional latency of starting a new connection, it is RECOMMENDED to use short-lived connections, instead of maintaining a long-lived persistent connection using aliveness checks.

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 12 Jul 2018 21:05:36 -0400

Hi Kent, I agree with the spirit of the statement / guidance you've drafted.

You might want to tweak some of the wording, e.g. "test more aliveness" 
could be "test more layers of functionality" or something like that, but 
this is just a nit.

I think the footnote recommending short-lived connections should be more 
clear about why that's the recommendation.  What is the risk/danger/etc 
of longer-lived connections.  That recommendation seems a bit naked as 
currently described, and actually should probably be more than just a 
footnote.

---

From: "Black, David" <[email-masked]>
Date: Fri, 13 Jul 2018 14:37:02 +0000

+1 on Wes's comments, especially that "layers of functionality" is better than "aliveness" ;-).

Thanks, --David

> -----Original Message-----
> From: tsv-area [mailto:[email-masked]] On Behalf Of Wesley
> Eddy
> Sent: Thursday, July 12, 2018 9:06 PM
> To: [email-masked]
> Subject: Re: statement regarding keepalives
> 
> > The folks working with the BBF asked the NETMOD WG to consider
> modifying draft-ietf-netconf-netconf-client-server to support TCP keepalives
> [1].  However, it is unclear what IETF's position is on the use of keepalives,
> especially with regards to keepalives provided in protocol stacks (e.g.,
> <some-app> over HTTP over TLS over TCP).
> >
> > After some discussion with Transport ADs (Spencer and Mijra) and the TLS
> ADs (Eric and Ben), the following draft statement has been crafted.  Spencer
> and Mijra have requested TSVAREA critique it before, perhaps, developing a
> consensus document around it in TSVWG.
> >
> > It would be greatly appreciated if folks here could review and provide
> comments on the draft statement below.  The scope of the statement can
> be increased or reduced as deemed appropriate.
> >
> > [1]
> https://mailarchive.ietf.org/arch/msg/netconf/MOzcZKp2rSxPVMTGdmmrVI
> nwx2M
> >
> > Thanks,
> > When the initiator of a networking session needs to maintain a persistent
> connection [1], it is necessary for it to periodically test the aliveness of the
> remote peer.  In such cases, it is RECOMMENDED that the aliveness check
> happens at the highest protocol layer possible that is most meaningful to the
> application, to maximize the depth of the aliveness check.
> >
> > E.g., for an HTTPS connection to a simple webserver, HTTP-level keepalives
> would test more aliveness than TLS-level keepalives.  However, for a
> webserver that is accessed via a load-balancer that terminates TLS
> connections, TLS-level aliveness checks may be the most meaningful check
> that could be performed.
> >
> > In order to ensure aliveness checks can always occur at the highest protocol
> layer, it is RECOMMENDED that protocol designers always include an
> aliveness check mechanism in the protocol and, for client/server protocols,
> that the aliveness check can be initiated from either peer, as sometimes the
> "server" is the initiator of the underlying networking connection (e.g., RFC
> 8071).
> >
> > Some protocol stacks have a secure transport protocol layer (e.g., TLS, SSH,
> DTLS) that sits on top of a cleartext protocol layer (e.g., TCP, UDP).  In such
> cases, it is RECOMMENDED that the aliveness check occurs within protection
> envelope afforded by the secure transport protocol layer.  In such cases, the
> aliveness checks SHOULD NOT occur via the cleartext protocol layer, as an
> adversary can block aliveness check messages in either direction and send
> fake aliveness check messages in either direction.
> >
> > [1] While reasons may vary for why the initiator of a networking session
> feels compelled to maintain a persistent connection.  If the session is
> primarily quiet, and the use case can cope with the additional latency of
> starting a new connection, it is RECOMMENDED to use short-lived
> connections, instead of maintaining a long-lived persistent connection using
> aliveness checks.
> >
> >

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 19 Jul 2018 16:06:52 -0500

Dear All,

Top-posting ... thank yous to David and Wes for your feedback.

I'd like to report back to the SEC ADs about our discussion tomorrow when
the IESG and IAB meet (immediately after the final meeting session).

If you have any other comments, they will be appreciated at any point in
time, but would be most useful if they arrive before then.

Thanks,

Spencer

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 20 Jul 2018 13:47:45 +0200 (CEST)

Hi,

While I agree with the sentiment here, I have personally been in positions 
where application programmers were unable to (in a timely manner) modify 
whatever was running, to implement a keepalive protocol. In that case, 
turning on TCP keepalives was a very easy thing to do that immediately 
would yield operational benefits.

So I'd like to see in the text that we recommend to do it as "high up" in 
the stack as possible, but still don't put off people turning on TCP 
keepalives "because the IETF doesn't recommend that", and thus they do 
nothing at all and the problem just persists.

Also, should we talk about recommendations for what these timers should 
be? In my experience, it's typically in tens of seconds up to 5-10 minutes 
that makes sense for Internet use. Shorter than that might interrupt the 
connection prematurely, longer than that causes things to take too long to 
detect a problem. Of course it's up to the application/environment to 
choose the best value for each use-case, but some text on this might be 
worthwhile to have there?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 20 Jul 2018 08:39:47 -0500

Hi, Mikael,

> worthwhile to have there?

This is exactly the kind of feedback I'd like to reflect in whatever
guidance we give, in whatever form. Thank you for that.

Other thoughts?

Spencer

(And, yes, there's a tension between "why would I tear down a perfectly
good idle connection because it might not work the next time I try to use
it? for real" and "OMG, I need to send a message to the other side, but my
idle connection is now timing out, so I have to set up a new connection,
secure it, and do whatever else I need to do with the other side before I
can send a message!!!". That's a good thing to include in our advice)

---

From: Tom Herbert <[email-masked]>
Date: Fri, 20 Jul 2018 07:52:28 -0600

> guidance we give, in whatever form. Thank you for that.

Spencer,

When I saw the subject I was wondering if this was going to be the proposal
to deprecate keepalives! With vast numbers of connections performing them,
they at some point start to create congestion. This is especially
problematic if somehow keepalives timers become sychronized. IIRC Google
calendar was brought down at one point a while back because of a keepalive
avalanche. I would assume that keepalives are primarily needed to maintain
NAT state, so if NAT were deprecated (like conceptually promised by  IPv6)
then maybe keepalives could go away.

In any case, I think the text here probably would be good in the draft that
examines current state of keepalives and makes recommendations.

Tom

---

From: Joe Touch <[email-masked]>
Date: Fri, 20 Jul 2018 07:47:57 -0700

> On Jul 20, 2018, at 4:47 AM, Mikael Abrahamsson <[email-masked]> wrote:
> 
> So I'd like to see in the text that we recommend to do it as "high up" in the stack as possible, but still don't put off people turning on TCP keepalives "because the IETF doesn't recommend that", and thus they do nothing at all and the problem just persists.

Agreed. Further, I don’t see the problem with having keepalives at all levels of the stack - at lower levels, they can be suppressed as long as higher levels are performing that function, but it’s still always useful to have them on at lower levels as a “backup”.

So I would hesitate to say “do this at the highest level”. My advice would be “do this at ALL levels that benefit, and be sure to suppress independent keepalives at a given level if there is activity at higher levels that suffices”.

Joe

---

From: Kent Watsen <[email-masked]>
Date: Fri, 20 Jul 2018 15:14:47 +0000

> ...but still don't put off people turning on TCP keepalives "because
> the IETF doesn't recommend that", and thus they do nothing at all and
> the problem just persists.

No disagreement with what you and others have written, but note that 
the proposed statement only recommends not using TCP keepalives in
the presence of a crypto layer on top of the TCP-layer.

Perhaps the statement could be refined, something along the lines 
of, in cases when there is a crypto layer, to recommend not using,
or at least relying on, TCP keepalives, *unless* higher-level
keepalives have stopped working.

To be clear, the statement as written, though not stated explicitly,
recommends TCP keepalives, in cases where they make sense.

Kent

---

From: Joe Touch <[email-masked]>
Date: Fri, 20 Jul 2018 10:35:27 -0700

Hi, Kent, 

For keepalives, I think there are some more useful recommendations than
are currently proposed. Here's my cut: 

- keepalives SHOULD be used at *every* protocol layer where it is
important to retain state (connectivity) in the absence of traffic from
higher layers 

- keepalives at a layer SHOULD be suppressed in the presence of
sufficient traffic from higher layers  

- keepalives at a layer SHOULD NOT be interpreted as implying state at
any other layer (this was the error that BGP once made, i.e., assuming
that if TCP could not keep state then the route should be dropped) 

IMO, these are necessary and sufficient.  

I don't think there should be any other statement about the relative
timing of keepalives at different layers or that keepalives should be
driven by the highest layer possible. That's just silly - e.g., if the
app layer doesn't care about keeping state but the lower layer benefits,
there's no reason to rewrite the app just to keep the function out of
the transport layer(s). 

Joe