Subject: [TLS] TLS 1.3 and TCP interactions

From: David Benjamin <[email-masked]>
Date: Fri, 29 May 2020 16:59:57 -0400

Hi all,

As we’ve been using TLS 1.3 in more scenarios, we’ve encountered some
interesting interactions with TCP. We thought we’d document these and send
a note here. In general, we've found that TLS implementations need to be
wary of post-handshake messages and “unexpected” transport writes. This
unfortunately also includes some server handshake alerts.

TLS APIs

First, some background on APIs for TLS libraries. TLS is often deployed
“transparently” underneath a TCP-based protocol. HTTPS sandwiches TLS
between HTTP and TCP, etc. By and large, reads and writes over TLS are
one-to-one with reads and writes over TCP.

TLS APIs and callers can subtly rely on this. Some libraries expose an
interface like the POSIX sockets API, including non-blocking behavior. If
the transport is blocked on I/O, this is surfaced as an error for the
caller to retry later. Importantly, the library cannot drive transport I/O
on its own. The caller must drive the operation to completion. Other APIs
transform bytes and leave I/O to the application. Any TCP writes triggered
by TLS reads and vice versa are even more directly part of the API surface.
In contrast, sometimes the TLS library can drive I/O itself. For example, a
Go TLS implementation can do background work in a goroutine.

Also note that libraries may predate TLS 1.3, but now enable TLS 1.3 by
default. Those libraries must ensure callers written against TLS 1.2 work
in TLS 1.3.

Post-handshake messages and flow control

TLS 1.2 and TLS 1.3 both have post-handshake messages, but TLS 1.2 only
uses them for renegotiation, which is rare and often disabled. TLS 1.3 has
post-handshake NewSessionTickets. A server will typically send tickets
immediately after the handshake

We initially treated NewSessionTicket as an extra flight in the server
handshake. After receiving the client Finished, the server would write
NewSessionTicket and then signal handshake completion. This kept tickets
working in unmodified server callers.

However, this can lead to a deadlock in some cases. A typical HTTP/1.1
client will first write its request and, only when this is complete, read
the response. If the write exceeds the transport buffer, it will not
complete, and thus the client will not read, until after the server starts
reading. An HTTP/1.1 server caller knows to read first, but only after the
handshake completes. If NewSessionTicket also exceeds the transport buffer,
this strategy means the server won’t complete the handshake until the
client starts reading. Thus the connection deadlocks.

While these messages usually fit in transport buffers, we don’t like
systems with invisible cliffs, particularly deadlocks. Some TLS
implementations embed client certificates in tickets, which can make them
large. Additionally, mock transports in tests sometimes use artificially
small buffers.

Recommendation: We switched to deferring NewSessionTicket to the first
application write by default. Server callers which wish to flush them
earlier may, but they should not block normal I/O on it. TLS
implementations which can drive transport I/O themselves may be able to
instead write them in the background after the handshake. Note, however,
the discussion on “Client-write-only protocols” below.

Separately, in case the server does not do this, TLS 1.3 client
implementations should eagerly read from the socket after the handshake,
even if the caller isn’t expecting application data. However, this is only
possible at a layer which can drive I/O itself. We implement this in
Chromium’s abstractions over BoringSSL, but cannot do so in BoringSSL
itself.

Likewise, while they are unlikely to exceed the transport buffer, TLS
libraries should defer KeyUpdate acknowledgements to the next application
write, possible from the KeyUpdate tweaks
<https://mailarchive.ietf.org/arch/msg/tls/cfw4paCGxI7Fj8QNmj6k1I66VII/>
made early on.

0-RTT and flow control

There is a similar effect in 0-RTT. The client writes the ClientHello and
early data. The server responds with ServerHello..Finished. Depending on
I/O strategy, implementations may hit a similar deadlock if the client
won’t read the ServerHello flight until it has written its early data, but
the server won’t read early data until it has written the ServerHello
flight.

Some factors make this deadlock less of a concern than NewSessionTicket:

   -

   0-RTT is new as of TLS 1.3 and should not be enabled by default. 0-RTT
   clients are already expected to handle extra cases such as 0-RTT rejects
   and replayability. That means libraries can impose extra requirements or
   introduce APIs for these I/O patterns.
   -

   The ServerHello flight does not contain a certificate. It is more likely
   to fit in transport buffers and has more-or-less fixed size.
   -

   For HTTP, RFC8470 only sends GETs over early data by default, which are
   smaller than POSTs and more likely to fit in transport buffers. But this is
   another invisible cliff in the system.

Recommendation: 0-RTT clients should eagerly read from the connection, even
if the application isn’t expecting data yet. This avoids this deadlock and
opportunistically confirms the handshake sooner, so more data is sent over
1-RTT. Note this must be done at a layer which can drive transport I/O
itself.

Client certificate errors and TCP resets

If a server rejects a client certificate, it should end an alert, so the
client can react accordingly. The client may display an error to the user,
clear a cache of certificate decisions, or prompt the user to select a
different certificate.

TLS 1.2 has a two round-trip handshake:

      ClientHello        -------->

                                            ServerHello

                                           Certificate*

                                     ServerKeyExchange*

                                    CertificateRequest*

                         <--------      ServerHelloDone

      Certificate*

      ClientKeyExchange

      CertificateVerify*

      [ChangeCipherSpec]

      Finished           -------->

                                     [ChangeCipherSpec]

                         <--------             Finished

The server has a handshake flight after the client certificate. If it
rejects the certificate, the TLS implementation will write an alert and
then report failure, at which point the caller will close the socket and
discard the connection. From the client’s perspective, this alert comes
instead of ChangeCipherSpec/Finished, during the handshake. It processes
the alert and cleanly fails the handshake, before any application data
flows.

TLS 1.3 reduces the handshake to one round-trip:

       ClientHello       -------->

                                            ServerHello

                                  {EncryptedExtensions}

                                  {CertificateRequest*}

                                         {Certificate*}

                                   {CertificateVerify*}

                                             {Finished}

                         <--------  [Application Data*]

       {Certificate*}

       {CertificateVerify*}

       {Finished}        -------->

There is no server flight after the client certificate. If the server
rejects it, it will again write an alert and report failure. However, now
the client receives it instead of the first application data record. This
is a behavior change to callers, who now must handle client certificate
errors out of read as well as connect.

Moreover, in a client-speaks-first protocol, the error now comes after the
client has already sent its request. This is not only a behavior change but
makes it unreliable over TCP. TCP sees:

   1.

   Client: write(ClientHello);
   2.

   Server: read(ClientHello); write(ServerHello..Finished);
   3.

   Client: read(ServerHello..Finished); write(Certificate..Finished);
   4.

   Server: read(Certificate..Finished); write(bad_certificate); close();
   5.

   Client: write(“GET / ...”); read(???);

Note (4) and (5) happen in parallel. Ideally ??? would be a bad_certificate
alert, but it is sometimes a TCP reset. I’m not a TCP expert, but I believe
this is because the client writes data (“GET / ...”) the server never
consumes. If it arrives at the server TCP stack before close(), the socket
is closed with unread data. If it arrives after close(), the socket
receives data after close(). TCP appears to consider either condition an
application protocol error and triggers a reset shortly after sending the
alert. If the client consumes the alert before its TCP stack sees the
reset, the alert gets through. Otherwise, TCP will not reliably deliver the
alert. Receive buffers may be cleared, data isn’t retransmitted, etc. This
is particularly pronounced on loopback.

Note, if TCP did not reset, we’d deadlock other scenarios. Suppose the
client request did not fit in transport buffers. The client would not read
until that is flushed, but the server will never ACK it. The client would
then never progress to the alert and get stuck. By resetting, TCP
interrupts large client writes with some error, albeit the wrong one.

Recommendation: We do not have a good answer here. The deadlock scenario
means we cannot hope to reliably deliver alerts unless the client eagerly
reads as above. But servers cannot rely on clients to do this, and this is
not sufficient because of the TCP reset.

It seems the only fix is for the server to keep the connection alive for
some time after the failure, maybe draining some bytes from the
application, with some limit before giving up and resetting if the client
seems to be writing a lot of data without ever reading. This would need to
be quite up the stack. We have not implemented this.

TLS False Start (RFC7918) exposes much of the same issues, but, in TLS 1.3,
this flow is not optional. Clients cannot even choose to pay a round-trip
to restore the TLS 1.2 flow because, in the successful case, there is
nothing to wait for. One could imagine an extension that adds an optional
server flight, but a round-trip to fix an error condition is an
unsatisfying trade-off.

Clients could also consider TCP resets to be potential client certificate
errors. This is also unsatisfying as TCP resets are unauthenticated and may
have other causes.

Client-write-only protocols

Edge cases may have other unexpected writes. Consider a protocol where the
server never writes, and thus the client never reads. TLS 1.3 introduces
server NewSessionTicket messages, so we again trigger the deadlock above.
If the client further shuts down the read half of the connection, the
NewSessionTicket message will also trigger the TCP reset behavior above.

Recommendation: Don’t do this. If you must, either the client must read
anyway to pick up the ticket, or the server must not send tickets. A TLS
server library probably should default to deferring tickets to application
write, which would do the latter. Note this means such protocols don’t get
resumption.

Half-RTT data

We haven’t done much with half-RTT data outside of 0-RTT connections, but
half-RTT may risk similar issues. Half-RTT data in a client certificate
connection is sent before the server learns the client identity. That means
the connection is writable before all its properties are established, which
is an awkward API.

One might think to avoid this state by configuring half-RTT data ahead of
time for the library to write during the handshake, immediately after
ServerHello..Finished. A half-RTT HTTP/2 SETTINGS frame doesn’t need a
streaming API, and this avoids exposing the incomplete state to the caller.

However, this also risks flow control issues, depending on sizes and I/O
patterns. If both half-RTT data and client Certificate..Finished are too
large, this design has another flow control deadlock: the server will not
read client Certificate..Finished until writing half-RTT data, and the
client will not read half-RTT data until it has written its flight.

Recommendation: Sadly, it seems half-RTT APIs need to be more complicated
than this.

Hopefully this is helpful to folks.

David

---

From: Jeremy Harris <[email-masked]>
Date: Fri, 29 May 2020 22:36:42 +0100

>    Server: read(Certificate..Finished); write(bad_certificate); close();

Rather than close(), the server should shutdown(SHUT_WR)

> consumes.

... because, having solely notified that it will send no more, the
server is free to continue to read from the socket.

 If it arrives at the server TCP stack before close(), the socket
> alert.

-- 
Cheers,
  Jeremy

---

From: David Benjamin <[email-masked]>
Date: Fri, 29 May 2020 17:52:43 -0400

> server is free to continue to read from the socket.

The server's application logic has no reason to continue to read from the
socket. This behavior typically comes from layers above the stack. From the
server's perspective, the TLS library just said the handshake failed. It
required a valid client certificate during the handshake, and the client
did not produce one. This is fatal to the connection, so the server has no
reason to keep it around and no application data to read or write.

The issue is that we've tweaked the I/O patterns in TLS 1.3 such that the
resulting overall behavior interacts badly with TCP. The server could
shutdown(SHUT_WR) and intentionally hold on to the socket and drain data,

This, however, has DoS implications for the server and is a very different
API shape from many current TLS libraries.

David

---

From: Watson Ladd <[email-masked]>
Date: Fri, 29 May 2020 18:35:58 -0400

On Fri, May 29, 2020 at 5:01 PM David Benjamin <[email-masked]> wrote:
>
> Hi all,
>
>
> As we’ve been using TLS 1.3 in more scenarios, we’ve encountered some interesting interactions with TCP. We thought we’d document these and send a note here. In general, we've found that TLS implementations need to be wary of post-handshake messages and “unexpected” transport writes. This unfortunately also includes some server handshake alerts.
>
>
> TLS APIs
>
> First, some background on APIs for TLS libraries. TLS is often deployed “transparently” underneath a TCP-based protocol. HTTPS sandwiches TLS between HTTP and TCP, etc. By and large, reads and writes over TLS are one-to-one with reads and writes over TCP.
>
>
> TLS APIs and callers can subtly rely on this. Some libraries expose an interface like the POSIX sockets API, including non-blocking behavior. If the transport is blocked on I/O, this is surfaced as an error for the caller to retry later. Importantly, the library cannot drive transport I/O on its own. The caller must drive the operation to completion. Other APIs transform bytes and leave I/O to the application.. Any TCP writes triggered by TLS reads and vice versa are even more directly part of the API surface. In contrast, sometimes the TLS library can drive I/O itself. For example, a Go TLS implementation can do background work in a goroutine.

In my experience the issues are thorniest when dealing with blocking
sockets. Libraries using nonblocking sockets have to signal to the
application that they want IO to happen during the handshake, and can
use that same mechanism at later times, particularly for rekeying.
Libraries with blocking behavior are unfortunately in a difficult
position if they imitate a POSIX API and have no means to drive I/O.

One possible dirty trick is to set nonblocking on an owned socket and
translate the blocking call into a select or poll based loop that
issues both writes and reads until enough is read or written. Note
that the real corner case is unanticipated needs to read from the
socket: the library has control when it needs to write.

---

From: Nico Williams <[email-masked]>
Date: Fri, 29 May 2020 18:28:22 -0500

> socket: the library has control when it needs to write.

Indeed.

Another is to start a worker thread to do all (async) I/O on the
connection and use inter-thread communications primitives on the
blocking I/O API side.  Because the worker thread needs to do async I/O
anyways, it might as well service multiple connections to reduce the
amount of resources needed for the whole thing.

Nico
--

---

From: Jeremy Harris <[email-masked]>
Date: Sat, 30 May 2020 01:06:23 +0100

On 29/05/2020 22:52, David Benjamin wrote:
> socket.

Perhaps not, though I suggest it is valuable to it for debug and
development to be able to see the client's reaction to the shutdown.

However, you ignored the value of the shutdown() action not resulting
in a RST.  That means that the client view becomes deterministic;
your "read(???)" in the close/RST scenario becomes
"read(error, ALERT:bad cert)".
-- 
Cheers,
  Jeremy

-- 
Cheers,
  Jeremy

---

From: Viktor Dukhovni <[email-masked]>
Date: Fri, 29 May 2020 21:12:48 -0400

> > this is because the client writes data (“GET / ...”) the server never

Fortunately, at least for SMTP, the issues David raises are not likely
to be seen because:

    - The protocol is not unidirectional, client and server take turns.

    - The server sends the first AppData (220 greeting), so the
      NewTicket message is read by the client, or any complaints
      about the client certificate...

However, if the server has something to send during the data transfer
phase (e.g. attempts to rekey the connection), then the client could
perhaps not be polling the server socket for incoming traffic, and
a deadlock may be possible.

Such mid-transfer traffic from the server is rather unlikely in
practice, so I don't think there are immediate concerns for Exim or
Postfix, but keeping an eye on the server socket for incoming data may
be prudent.  If a zero-length SSL_read() when the socket appears
readable takes care of any unexpected non-application-data, then
it should be fairly easy to incorporate into the I/O loop.

    - If the remote socket is readable, but you're trying to write
      and don't yet expect to read, do a zero-byte read, but since
      this may not drain the read queue of unexpected app-data,
      don't poll for read again unless the write side has made
      some progress.

    - If you have something to write, and the socket is writable, write it.

    - If you are expecting to read something, read a chunk of data,
      paying attention also to SSL_ERROR_WANT_WRITE.

At least in Postfix, the socket is non-blocking once TLS is enabled and
we're relying on SSL_ERROR_WANT_READ/SSL_ERROR_WANT_WRITE to provide
appropriate retry signals, and then wait for the corresponding
condition.

Handling out-of-turn app-data (mid-data-transfer server errors) requires
a bit more care, but the server is supposed to wait until the client
sends '.'.  Early SMTP error responses are unreliable even without TLS.

-- 
    Viktor.

---

From: Ilari Liusvaara <[email-masked]>
Date: Sat, 30 May 2020 10:41:04 +0300

> streaming API, and this avoids exposing the incomplete state to the caller.

There are other issues with half-RTT HTTP/2 SETTINGS as well. I had a
HTTP/2 implementation that used to send HTTP/2 SETTINGS as half-RTT if
there was no client certificate[1]. I later changed it to not use
half-RTT for HTTP/2 because some TLS clients puked with
unexpected_message alert if one sent SETTINGS in half-RTT[2]. I do not
know which clients did that[3].

[1] Not doing that with client certificate was caused by API issues
with the handoff. The TLS library native API would have allowed sending
data in half-RTT even with client certificate pending (unlike say false
start, which is the TLS implementation just prohibits).

[2] The unexpected_message actually came after the handshake
completed (the client did send correct Finished MAC). 

(Fixing the issue was quite nasty shotgun debugging, and it took
long time to verify issue was fixed, as normally there would only be
one or two failures per day.)

[3] Some terse user reports I heard reported things like the site
not working in Firefox on Windows host, but working in Firefox on
Linux VM on that host and site not working in Chrome (or Chromium?)
but working in Firefox on the same system.

(I did get one user report that the problem was fixed about a
week after I did the change that fixed it.)

-Ilari

---

From: David Benjamin <[email-masked]>
Date: Sat, 30 May 2020 10:58:54 -0400

On Fri, May 29, 2020 at 8:06 PM Jeremy Harris <[email-masked]> wrote:

> development to be able to see the client's reaction to the shutdown.\
>

I have never seen a production server do this.

> "read(error, ALERT:bad cert)".

Yes, I said exactly this twice now. :-P Except the shutdown() isn't the
deciding factor. It's the lack of close(). shutdown(SHUT_WR) doesn't hurt,
but the alert already signals to the client that the connection is done.

However, there are two problems here. First, the server needs to call
close() eventually to release resources. This is a failed connection. The
natural thing to do is to tear down the connection state which, ultimately,
will destroy the socket in the natural implementation. In
particular, connection management is often outside the low-level TLS
library. The HTTP server is simply doing something like:

   if (tls_server_handshake(connection->tls) != TLS_HANDSHAKE_SUCCESS) {
      // Release all resources associated with connection. It's not needed
anymore.
      // This, among other work, needs to call close(connection->fd). The
caller here
      // is not a TLS expert and reasonably assumes that the TLS library
has sent
      // any data it needs to send. But the TLS library doesn't own the fd
and can't
      // prevent the close.
      delete_connection(connection);
  }

If the HTTP server software is aware of this mess, it can certainly get the
right behavior by leaving the socket open for longer, releasing it later
after a timeout or so. This is not obvious, hence the note here, so folks
would be aware of this interaction. Moreover, simply closing the socket on
error worked just fine in TLS 1.2, so most existing HTTP server software
are not going to keep their connections alive after error for the sake of
it.

Additionally, as noted in the original email, leaving the socket open is
insufficient if the client's first write exceeds the transport buffers.
Either the client needs to eagerly read (not an obvious expectation, nor
something that all clients do, hence the email), or the server must drain
the socket (an *especially* non-obvious expectation).

David

---

From: David Benjamin <[email-masked]>
Date: Sat, 30 May 2020 11:27:28 -0400

> > socket: the library has control when it needs to write.

The signaling for nonblocking sockets is somewhat interesting. I don't
think it's as straightforward as that because these non-blocking APIs and
how callers use them becomes sequential anyway. Consider the
NewSessionTicket deadlock issue and a TLS implementation which wishes to
send it immediately after the handshake, rather than deferring to the
first write. Signaling to the application that I/O needs to happen during
handshake, read, and write may look something like...

enum tls_io_result {
    tls_io_success,
    tls_io_error,
    tls_io_want_read,
    tls_io_want_write,
};

tls_io_result tls_handshake(...);
tls_io_result tls_read(...)
tls_io_result tls_write(...)

An HTTP/1.1 server may then:

1. tls_handshake(). On tls_io_want_*, pop back to the select() loop and
retry until tls_io_success.
2. tls_read("GET HTTP/1.1 ..."). On tls_io_want_*, pop back to the select()
loop and retry until tls_io_success.
3. tls_write("HTTP/1.1 200 OK ...")). On tls_io_want_*, pop back to the
select() loop and retry until tls_io_success.

Conversely, an HTTP/1.1 client may tls_handshake(), tls_write(), and
tls_read(), in that order. It likewise would only advance to the next
TLS-level function when the previous one succeeded. You generally do
not read or write before the connection is established, and HTTP/1.1,
unlike HTTP/2, never reads and writes concurrently.

Now the TLS library implements TLS 1.3 and wishes the server to send
NewSessionTicket as soon as possible. It may think to send it right at the
end of tls_handshake(). That gets it out independent of the caller's I/O
patterns. If NewSessionTicket hits EWOULDBLOCK, it then returns
tls_io_want_write out of tls_handshake(). But now we have our deadlock.
This HTTP server will not proceed to tls_read() and thus not consume the
(potentially large) HTTP request until *after* NewSessionTicket is written.
Conversely, this client will not consume the NewSessionTicket via
tls_read() until *after* it has finished writing the HTTP request.

Deferring to the first write makes it all work out, but that is because it
removes the "out-of-turn" write altogether.

> amount of resources needed for the whole thing.

Right, what is happening here, blocking or non-blocking, is that a
non-deferred NewSessionTicket must not be in sequence with the handshake or
TLS read. A deferred one can be in sequence with TLS write since you're
"naturally" blocking on transport write anyway.

If your core TLS protocol implementation is at the same layer as the
component that can drive new sequences, that all works fine. If not, you
need the upper layer to drive that background work, which means this is
part of the TLS protocol implementation's API contract. (Often the layers
above have only a basic understanding of the TLS protocol, since the point
of the TLS library is to abstract that.)

Or, in the case of NewSessionTicket, just defer them to TLS writes because
it's so much simpler. :-)

David

---

From: Nico Williams <[email-masked]>
Date: Sun, 31 May 2020 13:47:30 -0500

I wonder if an extension could be used to signal the client's and
server's ability to handle unexpected alerts.

So, for example, if a client TLS implementation and the application
using it can't handle new tickets, then don't send them.

Of course, implementors should get advice about these problems.

Nico
--