Subject: [TLS] Record-level ACKs in DTLS 1.3

From: Hanno Becker <[email-masked]>
Date: Fri, 28 Feb 2020 11:50:16 +0000

Hi,

TL;DR
This is all about various aspects of how ACKs work in DTLS 1.3:
- The DTLS 1.3 specification requires clarification regarding
  when ACKs should be sent.
- Record-level ACKs make efficient implementations for IoT
  devices harder. I argue that handshake-level ACKs reduce
  implementation complexity and allow for optimized
  implementations.

Details:
To illustrate, consider the following flight exchange, where the
second and third message in the second flight get reordered:

      Client                                  Server
   +----------+
   | Seq No 1 |-------------------------->   Received
   | Rec No a |
   +----------+
                                         +---------------+
    Received    <------------------------|   Seq No 1    |
                                         |   Rec No b    |
                                         +---------------+
                                         +---------------+
                           +-------------|   Seq No 2    |
                           |             |   Rec No c    |
                           |             +---------------+
                           |
                           |             +---------------+
    Received    <------------------------|   Seq No 3    |
                           |             |   Rec No d    |
                           |             +---------------+
  +-----------+            |
  |    ACK    |------------------------->
  | RecNo ??? |            |
  +-----------+            |
                           |
                           |
                           |
    Received    <----------+

The specification recommends that the client SHOULD send an ACK
when it receives out-of-order handshake message - here message (3,d)
while awaiting (2,c). However:

Question:
Which records does the client acknowledge in the ACK?

This is in fact implementation-dependent:
The client must only acknowledge record (3,d) if it has buffered it.
Implementations which don't implement out of order buffering of handshake
messages (to save RAM and ROM) must not acknowledge messages that weren't
buffered. The reason is that otherwise the server is mislead in which messages
need resending and which don't.

Conclusion 1:
If ACKs must only be sent for records which contain handshake messages
which were actually fed into the flight buffering / reassembly module
of the implementation, the specification should clearly say so.
I consider this point to be prone to misinterpretation, potentially
leading to incompatible implementations, because ACKs acknowledge
receipt at record granularity, and yet they must be sent only under
some assumptions on how their content was processed.

Next, given that the generation of record-ACKs is therefore content-specific,
I wonder why ACKs acknowledge records in the first place, and not triples
of (handshake seq nr, fragment offset, fragment length). It appears to me
that the latter matches their semantics much closer and allows for more
simpler and more efficient implementations. I'll elaborate on that in
the following.

As I understand, the simplest implementation of the retransmission state
machine using record-level ACKs works by buffering copies of records sent
in a flight and retransmits those that don't get acknowledged. The record
sequence numbers of the original and retransmitted records are internally
associated with the opaque record content, so that ACKs for both the original
and the retransmitted records can be recognized.

In this approach, the record contents are opaque to the sender's retransmission
state machine, which simplifies the implementation. However, it has major drawbacks:

(1) It requires buffering the entire flight, which incurs high memory
    overhead that can be significant on constrained devices.

    Implementations should be allowed to not buffer handshake
    messages but re-generate them on the fly through a callback whenever
    retransmission is needed.

    Two examples:
    - Consider the Certificate message: The raw certificate
      (chain) must reside in RAM/ROM already, and the Certificate message
      could easily be re-regenerated from that. Buffering handshake messages
      instead creates significant and unnecessary overhead.

    - This is of increasing importance with the advent of post-quantum
      cryptography, which comes with significantly larger key material.

(2) It doesn't allow switching the MTU for retransmission.

    Once the MTU may change for retransmission, the sender cannot keep
    track anymore of a single set of record sequence numbers per message such
    that an ACK for any of them confirms receipt of the message.

Of course, no implementation is strictly _forced_ to follow the above approach,
but the current record-level ACKs design significantly hardens any other approach:

For example, retransmission through callbacks or support for MTU switching
requires maintaining the tuples (record seq nr, lists of handshake fragments)
for all transmissions of the last flight. (Note that, in particular, just
remembering the mapping for the last transmission isn't enough, since an
acknowledgement for an earlier transmission might arrive late.)
The _sender_ would need to maintain less state, and would have an easier time
figuring out what to retransmit, if ACKs acknowledged handshake fragments
directly. On the _receiver_ side, there's no benefit of ACKing at the record level,
either, because -- as we have just seen -- it is _not_ the case that it can be
implemented as an automated mechanism at the record level, but needs to be triggered
from the handshake layer, at a point where all information about the handshake content
are available.

Implementations following the above buffering-based approach don't increase
in complexity through a switch to handshake level acknowledgements instead
of record level acknowledgements: They can just treat the acknowledged handshake
metadata as an _opaque_ identifier, replacing what was previously the record
sequence number.

Conclusion 2:
Handshake-level ACKs simplify the retransmission state machine
for non-buffering implementations on constrained devices, while
not hardening buffering implementations using record-level ACKs
so far. I'm aware of the lateness of this proposal, but would be
happy if the group would discuss and consider handshake-level ACKs
instead of record-level ACKs.

Let me know what you think,
Cheers,
Hanno
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 28 Feb 2020 06:44:07 -0800

Hanno,

Thanks for your note. I don't think your proposal will be an
improvement. It destroys information which could otherwise be used for
improve round-trip and loss estimation (cf. the difference between
QUIC and TCP ACKs). Second, it prevents the receiver from saying
some non-sensical things like acknowledging part of a received packet.
It's of course true that the current design allows you to ACK
non-received packets, but that's much more straightforward to
detect.

I agree that the current design requires keeping somewhat more
state during the period when no ACKs have been received. However,
your proposal actually requires retaining a simila data structure
once they have been received.

Some detailed comments below.

> second and third message in the second flight get reordered:

Yes, I agree that you need to only ACK data which you have processed.
I would be happy to take a PR to clarify this point.

>     retransmission is needed.

Note that many handshake messages cannot be re-generated without extensive
caching (for instance, ServerHello). However, as long as the messages
are generated consistently (a requirement in any case), then you in
fact need-not buffer them in memory; you merely keep the record -> offset
mappings as an edit list.

> acknowledgement for an earlier transmission might arrive late.)

Well, sort of. Nothing forbids duplicate transmission, so just remembering
the last transmission is mostly just inefficient.

> instead of record-level ACKs.

As noted above, I do not believe we should make this change.

-Ekr