Subject: Re: [TLS] Record-level ACKs in DTLS 1.3

From: Hanno Becker <[email-masked]>
Date: Wed, 4 Mar 2020 10:53:05 +0000

Hi Ekr,

thanks for your prompt reply.

> Thanks for your note. I don't think your proposal will be an
> improvement. It destroys information which could otherwise be used for
> improve round-trip and loss estimation (cf. the difference between
> QUIC and TCP ACKs).

What information is destroyed that was there before? You can still
use handshake metadata in place of record sequence numbers and
do the same analysis as before.
Also, with QUIC and TCP, we're mainly talking about throughput optimization
at the application stage, where DTLS doesn't ACK anyway.

> Second, it prevents the receiver from saying
> some non-sensical things like acknowledging part of a received packet.
> It's of course true that the current design allows you to ACK
>non-received packets, but that's much more straightforward to detect.

I don't think this is actually a complication: In both cases you
have a list of messages you've sent which are tagged somehow -- either
by a record sequence number or handshake metadata which at this point
can be treated opaque -- and if you receive an ACK for a non-existent
tag, you ignore it or fail, whatever the policy.

Beyond that, I consider the more fine-grained ACKing at the handshake
level a benefit, because it allows to acknowledge parts of records
containing multiple handshake messages only some of which could be
processed. Example:

    +---------------------------------+
    | Seq No 1                        |---------------> Received
    | Fragment 0..a                   |
    +---------------------------------+

    +---------------------------------+     lost
    | Seq No 1                        |----------x
    | Fragment a+1..b                 |
    +---------------------------------+

    +-----------------+----------+
    | Seq No 1        | Seq no 2 |-------------------> Received
    | Fragment b+1..N | complete |
    +-----------------+----------+

​If this happens with an implementation which only supports contiguous
reassembly (I think GnuTLS does it this way, or at least used to), it
will drop the second fragment, but the second message might still be
buffered and hence ACK'ed. With record-level ACKs, such fine-grained
ACKing isn't possible, and HS Seq No 2 will need to be resent even
though it has been received.

>  I don't think your proposal will be an improvement.

I uphold this: ACKing at the handshake level does not lead to complications
while (a) easing memory efficient implementations for IoT devices and
(b) increasing conceptual clarity by avoiding the current semantic ambiguity
of record level ACK with dependency on implementation-specific handshake
level details like buffering and reassembly strategy.

> Yes, I agree that you need to only ACK data which you have processed.
> I would be happy to take a PR to clarify this point.

It's actually a MUST only, not a need only, which is an important point
because it means that ACKing can _not_ be implemented as an automatic
mechanism at the record layer.
I'll file a PR once the discussion has concluded.

Best,
Hanno

________________________________
From: Eric Rescorla <[email-masked]>
Sent: Friday, February 28, 2020 2:44 PM
To: Hanno Becker <[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: Re: [TLS] Record-level ACKs in DTLS 1.3

Hanno,

Thanks for your note. I don't think your proposal will be an
improvement. It destroys information which could otherwise be used for
improve round-trip and loss estimation (cf. the difference between
QUIC and TCP ACKs). Second, it prevents the receiver from saying
some non-sensical things like acknowledging part of a received packet.
It's of course true that the current design allows you to ACK
non-received packets, but that's much more straightforward to
detect.

I agree that the current design requires keeping somewhat more
state during the period when no ACKs have been received. However,
your proposal actually requires retaining a similar data structure
once they have been received.

Some detailed comments below.

> Hi,
>
> TL;DR
> This is all about various aspects of how ACKs work in DTLS 1.3:
> - The DTLS 1.3 specification requires clarification regarding
>   when ACKs should be sent.
> - Record-level ACKs make efficient implementations for IoT
>   devices harder. I argue that handshake-level ACKs reduce
>   implementation complexity and allow for optimized
>   implementations.
>
> Details:
> To illustrate, consider the following flight exchange, where the
> second and third message in the second flight get reordered:
>
>       Client                                  Server
>    +----------+
>    | Seq No 1 |-------------------------->   Received
>    | Rec No a |
>    +----------+
>                                          +---------------+
>     Received    <------------------------|   Seq No 1    |
>                                          |   Rec No b    |
>                                          +---------------+
>                                          +---------------+
>                            +-------------|   Seq No 2    |
>                            |             |   Rec No c    |
>                            |             +---------------+
>                            |
>                            |             +---------------+
>     Received    <------------------------|   Seq No 3    |
>                            |             |   Rec No d    |
>                            |             +---------------+
>   +-----------+            |
>   |    ACK    |------------------------->
>   | RecNo ??? |            |
>   +-----------+            |
>                            |
>                            |
>                            |
>     Received    <----------+
>
> The specification recommends that the client SHOULD send an ACK
> when it receives out-of-order handshake message - here message (3,d)
> while awaiting (2,c). However:
>
> Question:
> Which records does the client acknowledge in the ACK?
>
> This is in fact implementation-dependent:
> The client must only acknowledge record (3,d) if it has buffered it.
> Implementations which don't implement out of order buffering of handshake
> messages (to save RAM and ROM) must not acknowledge messages that weren't
> buffered. The reason is that otherwise the server is mislead in which messages
> need resending and which don't.
>
> Conclusion 1:
> If ACKs must only be sent for records which contain handshake messages
> which were actually fed into the flight buffering / reassembly module
> of the implementation, the specification should clearly say so.
> I consider this point to be prone to misinterpretation, potentially
> leading to incompatible implementations, because ACKs acknowledge
> receipt at record granularity, and yet they must be sent only under
> some assumptions on how their content was processed.

Yes, I agree that you need to only ACK data which you have processed.
I would be happy to take a PR to clarify this point.

> As I understand, the simplest implementation of the retransmission state
> machine using record-level ACKs works by buffering copies of records sent
> in a flight and retransmits those that don't get acknowledged. The record
> sequence numbers of the original and retransmitted records are internally
> associated with the opaque record content, so that ACKs for both the original
> and the retransmitted records can be recognized.
>
> In this approach, the record contents are opaque to the sender's retransmission
> state machine, which simplifies the implementation. However, it has major drawbacks:
>
> (1) It requires buffering the entire flight, which incurs high memory
>     overhead that can be significant on constrained devices.
>
>     Implementations should be allowed to not buffer handshake
>     messages but re-generate them on the fly through a callback whenever
>     retransmission is needed.

Note that many handshake messages cannot be re-generated without extensive
caching (for instance, ServerHello). However, as long as the messages
are generated consistently (a requirement in any case), then you in
fact need-not buffer them in memory; you merely keep the record -> offset
mappings as an edit list.

>     Two examples:
>     - Consider the Certificate message: The raw certificate
>       (chain) must reside in RAM/ROM already, and the Certificate message
>       could easily be re-regenerated from that. Buffering handshake messages
>       instead creates significant and unnecessary overhead.
>
>     - This is of increasing importance with the advent of post-quantum
>       cryptography, which comes with significantly larger key material.
>
> (2) It doesn't allow switching the MTU for retransmission.
>
>     Once the MTU may change for retransmission, the sender cannot keep
>     track anymore of a single set of record sequence numbers per message such
>     that an ACK for any of them confirms receipt of the message.
>
> Of course, no implementation is strictly _forced_ to follow the above approach,
> but the current record-level ACKs design significantly hardens any other approach:

> For example, retransmission through callbacks or support for MTU switching
> requires maintaining the tuples (record seq nr, lists of handshake fragments)
> for all transmissions of the last flight. (Note that, in particular, just
> remembering the mapping for the last transmission isn't enough, since an
> acknowledgement for an earlier transmission might arrive late.)

Well, sort of. Nothing forbids duplicate transmission, so just remembering
the last transmission is mostly just inefficient.

> Conclusion 2:
> Handshake-level ACKs simplify the retransmission state machine
> for non-buffering implementations on constrained devices, while
> not hardening buffering implementations using record-level ACKs
> so far. I'm aware of the lateness of this proposal, but would be
> happy if the group would discuss and consider handshake-level ACKs
> instead of record-level ACKs.

As noted above, I do not believe we should make this change.

-Ekr

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 5 Mar 2020 06:38:40 -0800

Hanno,

> at the application stage, where DTLS doesn't ACK anyway.

If you retransmit a record with exactly the same contents,
it is not possible which version was received. It's true that

> tag, you ignore it or fail, whatever the policy.

Sure, but now you just have some new artificial rule
that you can't process part of a handshake message,
even though you can actually say that.

> If this happens with an implementation which only supports contiguous

Yes, I agree that this is true, however, those implementations
have already decided that they favor implementation simplicity
over high performance, so I don't think this consideration
is relevant here.

> level details like buffering and reassembly strategy.

I don't really agree with (b) for the reasons above. It introduces new
complications. As for (a) I believe that in practice the state that
must be kept is quite small (in general, there will only be no
retransmission at all and so you will only need one or slightly more
extant flight). The TLS handshake already requires you to store quite
a bit of state (hashes, etc.)  so I'm not persuaded by this argument.

I think we're down to a difference of technical judgement now, so
I'll leave it to the chairs to determine how to address this comment.

-Ekr

---

From: Hanno Becker <[email-masked]>
Date: Thu, 5 Mar 2020 15:28:38 +0000

Hi Ekr,

> a bit of state (hashes, etc.)  so I'm not persuaded by this argument.

DTLS 1.3 should suit the case of IoT devices and lossy IoT networks,
together with the long-term prospect of post-quantum cryptography
and the large cryptographic material that goes along with it. In such
context, it is / will be important to be able to deal with excessive
fragmentation and retransmission efficiently.

Moreover, note that in the face of a PMTU change, retransmission on the
basis of buffered opaque record content without knowledge of the underlying
handshake structure doesn't work, and will require retransmission of the entire flight.
This is avoided with ACKing at the handshake level, which makes retransmissions
agnostic to PMTU changes.

> I'll leave it to the chairs to determine how to address this comment.

It would be nice to hear some more opinions on this here, too.

Thanks for the discussion,

Cheers,
Hanno

Sent: Thursday, March 5, 2020 2:38 PM

Hanno,

-Ekr

---

From: "Christopher Wood" <[email-masked]>
Date: Wed, 11 Mar 2020 06:37:07 -0700

> Thanks for the discussion,

There are two primary issues raised in this thread:

1. Lack of clarification regarding when ACKs should be sent.
2. Record-level ACKs make efficient implementations for IoT devices 
harder. (draft

The latest version of the specification addresses (1) with the following 
text:

~~~
record_numbers:  a list of the records containing handshake messages
       in the current flight which the endpoint has received and either
       processed or buffered, in numerically increasing order.

    Implementations MUST NOT acknowledge records containing non-
    duplicative handshake messages or fragments which have not been
    processed or buffered.  Otherwise, deadlock can ensue.

    During the handshake, ACKs only cover the current outstanding flight
    (this is possible because DTLS is generally a lockstep protocol).
    Thus, an ACK from the server would not cover both the ClientHello 
and
    the client's Certificate.  Implementations can accomplish this by
    clearing their ACK list upon receiving the start of the next flight.

    After the handshake, ACKs SHOULD be sent once for each received and
    processed record (potentially subject to some delay) and MAY cover
    more than one flight.
~~~

Regarding (2), given that record-level ACKs align with QUIC and also do 
not seem to be insurmountable implementation obstacles (e.g., stacks can 
re-generate large handshake messages on-demand instead of buffering 
them), the chairs do not think any change here is required.

Thanks,
Chris, on behalf of the chairs