Subject: [tsvwg] Comments on draft-tsvwg-quic-protocol-01

From: Martin Duke <[email-masked]>
Date: Fri, 8 Jan 2016 10:07:57 -0800

I've been toying with implementing parts of this draft while also trying to
interact with the existing Chromium implementation, and beyond the
obviously incomplete sections in the draft I think there are several items
that are unclear, suboptimal, or not conformant with Chromium.

This is a wonderful contribution with great promise, and many exceptional
design features. But:

1) Section 5.1: I suggest that "the CHLO MUST fit in one packet; if need
be, one that requires IP fragmentation." As ugly as that is, it's far
superior to forcing the server to maintain state on incomplete CHLO. In
another forum the authors have indicated that this is their intention. This
also requires an exception to the preface of Section 6, which says to avoid
IP fragmentation (though as I understand it we are a long way from CHLO
having to be that long).

2) Section 5.1: "The next time the client sends a CHLO, it can use the
cached credentials from the previous connection to immediately send
encrypted requests to the server."

I would appreciate wording that makes it clearer that, "upon receipt of an
authenticated REJ, the client MUST generate a new connection ID for
subsequent CHLO messages, with sequence number 1." This clarification is
also important to making servers stateless pre-SHLO.

3) Section 5.1: Without veering into the Crypto documents area, I think it
would be helpful to indicate the frames that MUST, SHOULD, or MAY be in
CHLO, REJ, and SHLO messages. Obviously there will be STREAM frames for
stream 1, but I can certainly imagine PADDING, WINDOW_UPDATE, and ACK
messages required as well. And I imagine CONNECTION_CLOSE etc. MAY be
present. But I'm not sure! The stream 1 fields may be in a crypto document
but the frame list should be here.

4) Section 6.1: There is a little encryption ambiguity in the packets that
the server receives from the client just after sending the SHLO. These
packets could either use the initial key or the symmetric key, and with
packet reordering they could be interleaved for while. As someone that
implements server-side stuff for a living, it's irritating to potentially
have to decrypt the packet twice, and to allocate space for two sets of
keys in the PCB.

There are two unused bits in the Public Flags. It appears that Chromium has
already taken one for a "multipath" flag, currently undocumented. I propose
that the second one be an indicator that the client has received the SHLO
and is using the symmetric key.

5) Section 6.1: The Chromium implementation sometimes puts 12
authentication bits between public and private header. It would be good to
make this explicit in the document, as the figure in Section 6.1 reasonably
reads as having the private flags immediately follow the sequence number.

6) Section 6.5: Please provide the correct format for the CADR tag.

7) Section 8.3: I have read this section several times and cannot discern
at all what the TCP timestamps are for, how many there should be, where
they come from, and what format they are in. Are these echoes of some
sender timestamp (and if, so where is that from) or are these local clock
measurements?

Obviously this has something to do with RTT calculation, but the mapping of
the ACK frame fields to this calculation is really unclear. I tried to
generate some psuedocode to generate an ACK frame and got nowhere.

Regards,
Martin

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 12 Jan 2016 12:01:18 -0800

Hi Martin,

Thanks so much for your comments and your interest! We're working on
updating the draft -- this is decidedly a -01 draft at the moment -- so
thank you for your text suggestions. Answers inline:

> IP fragmentation (though as I understand it we are a long way from CHLO

I've added text to say "Any CHLO packet must be sent as a single UDP
datagram." There's no normative language in the draft at the moment.

> also important to making servers stateless pre-SHLO.

This gets into stateless vs non-stateless rejects.  Though there is code to
support stateless rejects, the current intention is to support both
stateless and stateful rejects going forward.  The documentation needs more
information to properly describe stateless rejects, which will happen once
we have more experience with stateless rejects.

> but the frame list should be here.

Good suggestion. We'll work this in in a future revision.

> and is using the symmetric key.

This is an interesting idea. To be clear, this interleaving should only
really happen for ~1 roundtrip after the SHLO is sent, and soon after the
first forward secure packet is received, the server should be able to drop
the initial encryption key. At this point, we do trial decryption, and this
bit could be a nice optimization.

That said, since the path for the QUIC handshake is to integrate with
TLS1.3, we are not making optimizations to QUIC's handshake at this point.
If I remember correctly, TLS1.3 could benefit from such a bit; this may be
a good suggestion for TLS1.3.

And thanks for the note on the multipath bit -- we didn't have it when the
earlier version of the draft went out, but we've documented it in the
upcoming rev of the draft.

> reads as having the private flags immediately follow the sequence number.

This is the auth hash and is part of the 'NULL' encryption algorithm
formatting... effectively, you're looking at an encrypted packet which has
public_header followed by the auth hash, followed by the encrypted body.
This is not the QUIC wire format, but the output of the NULL encrypter...
the plaintext QUIC packet format still has private flags following the
sequence number. I don't think it makes sense to change the QUIC
(plaintext) packet format.

Currently this is for debug purposes only (comment added to this effect)
and is optional, and the formatting is somewhat awkward... it may make
sense to document a more standard version of this. Nevertheless, this
should probably be considered optional for now.

> Obviously this has something to do with RTT calculation, but the mapping
> of the ACK frame fields to this calculation is really unclear. I tried to

Apologies, that timestamp section was extremely confusing.  The wording has
been substantially updated, which hopefully make it clearer -- let us know
if you are still stuck, and we're happy to help!

Thanks again for your comments!
Ian and Jana

---

From: Joe Touch <[email-masked]>
Date: Thu, 21 Jan 2016 10:15:18 -0800

On 1/8/2016 10:07 AM, Martin Duke wrote:
> I've been toying with implementing parts of this draft while also trying
> to interact with the existing Chromium implementation, and beyond the
> obviously incomplete sections in the draft I think there are several
> items that are unclear, suboptimal, or not conformant with Chromium.
> 
> This is a wonderful contribution with great promise, and many
> exceptional design features. But:
> 
> superior to forcing the server to maintain state on incomplete CHLO. 

FWIW, I disagree. One feature of TCP is its intent that all segments fit
inside unfragmented IP packets - that's why it tracks MTU. IMO, TCP
signalling needs to support that feature, not subvert it for mere
convenience (TCP is stateful anyway).

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 21 Jan 2016 10:19:20 -0800

> convenience (TCP is stateful anyway).

To clarify:

QUIC is stateful. If intended to be used where TCP is used, it should
follow TCP's principles when it comes to MTUs - stay inside the detected
MTU and address multi-segment state the same way it manages any other state.

Any protocol that deliberately sends messages larger than the known MTU
- except for detecting MTUs itself - is asking for trouble.

Joe

---

From: "Eggert, Lars" <[email-masked]>
Date: Fri, 22 Jan 2016 09:05:23 +0000

> - except for detecting MTUs itself - is asking for trouble.

+1

We also recommend such message size limit for general UDP use in RFC5405 and 5405bis.

Lars

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 22 Jan 2016 10:14:01 -0800

Yes, agreed. It is absolutely the intention (and current practice) that all
QUIC packets, including handshake packets, fit within a PMTU to avoid
fragmentation.

---

From: Tom Herbert <[email-masked]>
Date: Fri, 22 Jan 2016 10:21:51 -0800

> QUIC packets, including handshake packets, fit within a PMTU to avoid
PMTU discovery should work on UDP sockets (Linux at least). The udp
man page describes it.

Tom

---

From: Joe Touch <[email-masked]>
Date: Fri, 22 Jan 2016 10:23:06 -0800

OK, then I'm confused about the note about the particular protocol
message that won't then...

Joe

On 1/22/2016 10:14 AM, Jana Iyengar wrote:
> Yes, agreed. It is absolutely the intention (and current practice) that
> all QUIC packets, including handshake packets, fit within a PMTU to
> avoid fragmentation.
> 
> 
> On Fri, Jan 22, 2016 at 1:05 AM, Eggert, Lars <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     On 2016-01-21, at 19:19, Joe Touch <[email-masked]
>     <mailto:[email-masked]>> wrote:

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 23 Jan 2016 07:54:18 +1300

> PMTU discovery should work on UDP sockets (Linux at least). The udp

You are joking, right?

PMTUD is notoriously unreliable in the actual Internet in the actual real world,
(which is why we have RFC 4821) and not all computers run Linux.

In any case, as others have said, relying on fragmentation is poor design.

   Brian

---

From: Joe Touch <[email-masked]>
Date: Fri, 22 Jan 2016 11:00:35 -0800

> (which is why we have RFC 4821) and not all computers run Linux.

+1. QUIC should be using PLMTUD.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Fri, 22 Jan 2016 11:16:21 -0800

>> PMTU discovery should work on UDP sockets (Linux at least). The udp
No, and I am not advocating that as the solution. My point is that
QUIC running on Linux (which I assume is not an insignificant number
of hosts) is already probably using PMTUD and is subject to whatever
problems or benefits that entails. That is not to say it is good or
bad, but QUIC developers should be aware of this. If you get an
EMSGSIZE error on a UDP write operation there is a good chance it is
because of PMTUD.

---

From: Joe Touch <[email-masked]>
Date: Fri, 22 Jan 2016 11:20:17 -0800

> EMSGSIZE error on a UDP write operation there is a good chance it is

Certainly. But you are also vulnerable to black holing, where the use of
larger messages might result in a silent drop - forever. Unless the
layer above UDP adapts to that, the blackhole will persist.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Fri, 22 Jan 2016 12:07:16 -0800

> layer above UDP adapts to that, the blackhole will persist.
Right, that's why we also have TCP probing (RFC4821). Presumably, that
can be implemented for QUIC, but since QUIC is contained in the
application it may require more work. For instance, PMTU is a path
property that we can discover on the basis of packets in one flow. A
userspace application is restricted in it's interaction with L3
(routing) so there's no way it can inform other applications or the
networking stack about the PMTU, so discovery may need to be done
multiple times to the same destination. PTB error is something we can
apply to the cached path characterisitcs though.

---

From: Martin Duke <[email-masked]>
Date: Wed, 27 Jan 2016 12:51:47 -0800

Hello Ian and Jana,

I appreciate your responses to my comments.In the cases where you've
declined my advice, I think the reasoning is sound and I withdraw the
comment. In particular I bow to the wisdom of the crowd on fragmentation.

Regarding the timestamp issue, are you referring to the text in -02? I
agree that you've clarified the wire spec a bit, so that I can actually
extract the values or encode them. But I'm still not sure about how the
sender fills the fields and how the receiver uses them.

As I see no timestamp with data, I gather this is not a RFC 1323-like
echo-based mechanism. So what is the *data sender* going to do with the
time differential between connection start and the arrival time of any
given packet at the data receiver? It's not sufficient, in itself, to
determine the RTT, so what other timing mechanism is implied?

Is the data receiver supposed to return one timestamp per received packet,
or at some other rate?

Martin

On Tue, Jan 12, 2016 at 12:01 PM, Jana Iyengar <[email-masked]> wrote:

>> I would appreciate wording that makes it clearer that, "upon receipt of
>> an authenticated REJ, the client MUST generate a new connection ID for
> This gets into stateless vs non-stateless rejects.  Though there is code
> to support stateless rejects, the current intention is to support both
>> 4) Section 6.1: There is a little encryption ambiguity in the packets
>> that the server receives from the client just after sending the SHLO. These
> There are two unused bits in the Public Flags. It appears that Chromium
>> has already taken one for a "multipath" flag, currently undocumented. I
>> propose that the second one be an indicator that the client has received
>> the SHLO and is using the symmetric key.
>>
>
> Apologies, that timestamp section was extremely confusing.  The wording
> has been substantially updated, which hopefully make it clearer -- let us
> know if you are still stuck, and we're happy to help!
>