Subject: [QUIC] 2 issues for the hopper: terminology of packet vs sequence # and byte order

From: Patrick McManus <[email-masked]>
Date: Fri, 4 Nov 2016 17:01:17 -0400

Hi All - two more things to raise.. one I hope editorial and one design -
when we start doing github, I'll open issues to track there.

First, editorially draft-hamilton-quic-transport-protocol-01 generally
refers to packet numbers instead of sequence numbers. There seem to be two
exceptions to that - in 6.2 and the identifier
SEQUENCE_NUMBER_LIMIT_REACHED. OTOH
https://tools.ietf.org/html/draft-thomson-quic-tls-01 almost always refers
to sequence numbers (with one exception). I think SN is not defined, but PN
is and it should standardize on the PN language.

Second, draft-hamilton-quic-transport-protocol-01 boldly uses little endian
encoding. Ya gotta admire that chutzpah in an IETF draft :) From a
processor alignment pov, it makes perfect sense and I would support it on a
clean slate or on a well layered protocol. quic however, by design, is
neither of those things.

So we get into a bit of a mess - the packet number is little endian when it
appears in transport fields, but in the TLS document the AEAD nonce is
formed by padding the big endian representation of the same number. That
smells like a bug waiting to happen. Likewise a Stream ID in a transport
STREAM frame would be big endian, but when that same ID is encoded via 7540
rules from the h2-mapping document (e.g. in a PUSH_PROMISE) then it needs
to be big endian. Again - seems like a footgun. So while I salute the
valiant effort to go little endian, I think we ought to bow to the
constraints this effort faces and go the traditional all network byte order
path.

-Patrick

---

From: "Eggert, Lars" <[email-masked]>
Date: Sat, 5 Nov 2016 08:09:00 +0000

On 2016-11-04, at 22:01, Patrick McManus <[email-masked]> wrote:
> Second, draft-hamilton-quic-transport-protocol-01 boldly uses little endian encoding. Ya gotta admire that chutzpah in an IETF draft :) From a processor alignment pov, it makes perfect sense and I would support it on a clean slate or on a well layered protocol. quic however, by design, is neither of those things.

Chair hat off: That totally threw me when I started to implement, because apparently my mind is trained to automatically insert htonl() etc., and it took me a minute or so to understand why my code was producing incorrectly laid out packets.

Lars

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 5 Nov 2016 09:57:37 +0100

> On 5 Nov 2016, at 09:09, Eggert, Lars <[email-masked]> wrote:
> 
I guess if you really want to write a portable implementation you need to insert functions,
just not hton[sl](). You need to write htole[sl] and use them...

Best regards
Michael

---

From: Brian Trammell <[email-masked]>
Date: Sat, 5 Nov 2016 10:23:12 +0100

hi Patrick, all,

> On 04 Nov 2016, at 22:01, Patrick McManus <[email-masked]> wrote:
> 
> Hi All - two more things to raise.. one I hope editorial and one design - when we start doing github, I'll open issues to track there.
> 
> First, editorially draft-hamilton-quic-transport-protocol-01 generally refers to packet numbers instead of sequence numbers. There seem to be two exceptions to that - in 6.2 and the identifier SEQUENCE_NUMBER_LIMIT_REACHED. OTOH https://tools.ietf.org/html/draft-thomson-quic-tls-01 almost always refers to sequence numbers (with one exception). I think SN is not defined, but PN is and it should standardize on the PN language.
> 
> So we get into a bit of a mess - the packet number is little endian when it appears in transport fields, but in the TLS document the AEAD nonce is formed by padding the big endian representation of the same number. That smells like a bug waiting to happen. Likewise a Stream ID in a transport STREAM frame would be big endian, but when that same ID is encoded via 7540 rules from the h2-mapping document (e.g. in a PUSH_PROMISE) then it needs to be big endian. Again - seems like a footgun. So while I salute the valiant effort to go little endian, I think we ought to bow to the constraints this effort faces and go the traditional all network byte order path.

Either +1 or +16777216 to this, depending on your native byte order.

Even without the interactions with TLS and H2, a QUIC header merged with a UDP header would have two byte orders (big in the ports and length, little everywhere else).

The currently dominant processor architectures may be little-endian (though, given that the byte-order conversion calls pretty much always operate on cache, they're basically free) but we've spent decades training everyone that the network is big endian, and this decision itself also has a rationale grounded in transparency and diagnosability for humans looking at packet dumps: most of our notations for dealing with numbers are big-endian. So going little-endian seems like an esoteric choice with a whole lot more downside than upside.

Cheers,

Brian

---

From: Martin Thomson <[email-masked]>
Date: Mon, 7 Nov 2016 14:35:17 +1100

> is and it should standardize on the PN language.

Thanks Patrick.  They are sequence numbers in TLS, so it's easy to be
lazy.  We should talk about packet numbers when talking about QUIC:

https://github.com/martinthomson/quic-tls/issues/13

---

From: "Christian Huitema" <[email-masked]>
Date: Sun, 6 Nov 2016 22:16:13 -0800

On Sunday, November 6, 2016 7:35 PM, Martin Thomson wrote
> https://github.com/martinthomson/quic-tls/issues/13

Also, the sequence number of TLS are probably different from the packet numbers used in QUIC -- a TLS record can span multiple QUIC packets, the QUIC packet number also counts the 0-RTT packets on other streams beside stream 1, etc.

There is a lot of text in the TLS draft that concerns itself with the QUIC packet layer encryption, including discussion of retransmissions. Are we sure that this is the right place for this text? What if someone decided to use a version of QUIC that uses something else than TLS to establish keys?

-- Christian Huitema

---

From: Martin Thomson <[email-masked]>
Date: Mon, 7 Nov 2016 17:31:44 +1100

On 7 November 2016 at 17:16, Christian Huitema <[email-masked]> wrote:
>
> There is a lot of text in the TLS draft that concerns itself with the QUIC packet layer encryption, including discussion of retransmissions. Are we sure that this is the right place for this text?

Yes.  Though I think that merging this with the main draft would make
some of that easier (and some of it harder).

> What if someone decided to use a version of QUIC that uses something else than TLS to establish keys?

They would have to solve all the same problems.
QUIC-as-it-is-deployed, makes different decisions in some places
(trial decryption, for example), but the basic set of problems is the
same.

---

From: "Christian Huitema" <[email-masked]>
Date: Mon, 7 Nov 2016 08:08:41 -0800

On Sunday, November 6, 2016 10:32 PM, Martin Thomson wrote:

>> There is a lot of text in the TLS draft that concerns itself with the QUIC packet layer 
>> encryption, including discussion of retransmissions. Are we sure that this is the right 
>> place for this text?
>

The integration of encryption and transport is a key element of the QUIC design. The problems that is poses are independent of the particular security session protocol running on stream 1. I am trying to think of a negotiation-agnostic API for the packet layer. I like this kind of API, because they allow for modular code design and testing.

I like your notation of epochs, @A, @B, @C, etc. It allows us to speak about the different phases, clear-text, 0-RTT, 1-RTT, key renewal, and to discuss the interaction between epoch transition and packet retransmission. The security layer will provide the 0-RTT and 1-RTT keys to the packet layer. These are API events that are independent of the security negotiation mechanisms, and can be used in the packet layer design.

Loss detection is independent of the stream mechanisms, but retransmission is not. We may want to do a specific retransmission rule for stream 1 during epochs A and B.

-- Christian Huitema

---

From: Martin Thomson <[email-masked]>
Date: Tue, 8 Nov 2016 07:52:50 +1100

> Loss detection is independent of the stream mechanisms, but retransmission is not. We may want to do a specific retransmission rule for stream 1 during epochs A and B.

Agreed.  I've tried to capture that, but it definitely needs refinement.

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 7 Nov 2016 15:30:12 -0800

Hi Patrick,

On Fri, Nov 4, 2016 at 2:01 PM, Patrick McManus <[email-masked]>
wrote:

> SEQUENCE_NUMBER_LIMIT_REACHED. OTOH https://tools.ietf.org/html/
> draft-thomson-quic-tls-01 almost always refers to sequence numbers (with
> one exception). I think SN is not defined, but PN is and it should
> standardize on the PN language.
>

This should all be Packet Number, and we should fix it all. The history is
that we used to call these Sequence Numbers, until we decided that the
overlap with TCP terminology was unnecessarily confusing, since there are
clear semantic differences between sequence number in TCP and packet number
in QUIC. (Specifically, sequencing is used in TCP to order received bytes
at the receiver, whereas Sequence Numbers in QUIC have only to do with the
order in which packets may have been sent. There's also freedom at a sender
to skip some numbers while sending, to avoid optimistic ack attacks, which
means that packets sent in sequence may have discontiguous numbers
assigned.)

> Second, draft-hamilton-quic-transport-protocol-01 boldly uses little
> endian encoding. Ya gotta admire that chutzpah in an IETF draft :)
>

More chutzpah than redefining HTTP transport with a 3-part non-layered
protocol at the IETF? :-)

>From a processor alignment pov, it makes perfect sense and I would support
> it on a clean slate or on a well layered protocol. quic however, by design,
> is neither of those things.
>

(I now proceed into this bikeshed. It was good to know you all.)
I think disagree. I definitely think we have a clean-slate, at least as far
as *this* protocol is concerned (modulo the running code.) I would argue
that we have a well-layered protocol, but it is not "simply-layered". See
more below.

> constraints this effort faces and go the traditional all network byte order

So, to be clear, as long as the QUIC framer (as against the TLS or the HTTP
framer) is the thing that's writing numbers on one side, and the QUIC
framer is the thing reading numbers on the other side, there's no issue
with endianness. In addition, if it's a non-QUIC framer writing numbers on
one side, and a non-QUIC framer reading numbers on the other side, QUIC
doesn't introduce any issues with endianness.
On how the bits are used by other components which interact with the QUIC
framer, that's a matter of API.

So, if the TLS nonce is sent and delivered over Stream 1. QUIC transports
and delivers bytes over Stream 1. QUIC doesn't care about the endianness of
the nonce just as the nonce's endianness would have nothing to do with
TCP's rules about endianness in TCP fields.

Similarly,  a Stream ID in a PUSH_PROMISE frame, which is sent as an H2
frame is basically opaque data on the wire to QUIC. In the mapping doc, a
PUSH_PROMISE frame is sent over the Headers stream, which is entirely an
HTTP-mapping concept. From QUIC's pov, it's a stream as any other. When an
HTTP client receives this frame over QUIC, it will probably "reserve" this
stream ID through an API call down into QUIC.

So yes, while not simply-layered, there's very clear encapsulation and
separation of functions and concerns. Non-QUIC entities MUST NOT put things
in the QUIC frame format, and MUST NOT read things directly from the frame
format.

That said, I understand that there's baggage in terms of what the IETF has
always done. The argument against it is that when engineering for the
common case, little endian makes much more sense in today's world.

(ducking for cover)
- jana

---

From: Martin Thomson <[email-masked]>
Date: Tue, 8 Nov 2016 10:35:43 +1100

I'm going to recommend that the big-endian and little-endian
proponents find themselves a room in Seoul.  I plan to mute any thread
that continues in this fashion.  It's not a rathole, it's the Sarlacc.

On 8 November 2016 at 10:30, Jana Iyengar <[email-masked]> wrote:
> in QUIC. (Specifically, sequencing is used in TCP to order received bytes at
> the receiver, whereas Sequence Numbers in QUIC have only to do with the
>> So we get into a bit of a mess - the packet number is little endian when
>> it appears in transport fields, but in the TLS document the AEAD nonce is
>> rules from the h2-mapping document (e.g. in a PUSH_PROMISE) then it needs to
>> be big endian. Again - seems like a footgun. So while I salute the valiant
>> effort to go little endian, I think we ought to bow to the constraints this
>> effort faces and go the traditional all network byte order path.
>
>
> framer) is the thing that's writing numbers on one side, and the QUIC framer
> is the thing reading numbers on the other side, there's no issue with
> endianness. In addition, if it's a non-QUIC framer writing numbers on one
> side, and a non-QUIC framer reading numbers on the other side, QUIC doesn't
> introduce any issues with endianness.
> the nonce just as the nonce's endianness would have nothing to do with TCP's
> rules about endianness in TCP fields.
>
> always done. The argument against it is that when engineering for the common
> case, little endian makes much more sense in today's world.
>
> (ducking for cover)
> - jana
>
>>
>>
>> -Patrick
>>
>>
>>
>

---

From: Patrick McManus <[email-masked]>
Date: Mon, 7 Nov 2016 23:13:48 -0500

hey Jana,

On Mon, Nov 7, 2016 at 6:30 PM, Jana Iyengar <[email-masked]> wrote:

fwiw - I don't believe this to be a mere matter of preference, though some
may disagree. Its probably worth citing the spdy spec bug that got the
endian-ness of the Setting ID wrong as a motivator. Totally avoidable
category of problem.

> I think disagree. I definitely think we have a clean-slate, at least as
> far as *this* protocol is concerned (modulo the running code.) I would
> argue that we have a well-layered protocol, but it is not "simply-layered".
> See more below.
>

The layer boundaries here are not strong - and that's OK! That's the nature
of this design and the source of of its benefits. I should have said  not
strongly layered instead of not well layered which implied a judgment that
I don't even agree with. What we should be aiming for is strong alignment
and coherency, not strong boundaries.

I mentioned a couple ways these layers bleed through already (h2 frames
referencing transport frame ids, and the packet number in the aead nonce),
but there's also the transport windowing information carried in a tls
extension and the versioning information repeated in no less than 3 layers
(alt-svc, version, alpn) and no doubt more spots. All of this is for the
good - I'm just saying we need to look at the coherency question as the
integrated big picture. Putting things in the same byte order is just table
stakes.

wrt clean slate - my references were to the constraints of tls 1.3 and h2..
we're going to want to keep those as unchanged as possible as design goals;
so aligning with them is valuable and that's a constraint. Aligning is
important, as they aren't layers isolated from each other in the same way
TCP/TLS/7540-h2 are.

I may have been misunderstood to be referencing google-quic and I certainly
appreciate a willingness to break with that.

does that make more sense?

-P

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 7 Nov 2016 23:08:30 -0800

> endian-ness of the Setting ID wrong as a motivator. Totally avoidable

Fair enough. Apologies for being facetious.

> The layer boundaries here are not strong - and that's OK! That's the
> nature of this design and the source of of its benefits. I should have
> said  not strongly layered instead of not well layered which implied a
> judgment that I don't even agree with. What we should be aiming for is
> strong alignment and coherency, not strong boundaries.
>

+1.

> integrated big picture. Putting things in the same byte order is just table

I agree that we should look at the integrated picture. My argument was that
the wire format of QUIC is independent of wire formats of the others. In
general, I think that we can separate concerns about byte order for QUIC
from others. That said, your point is about reducing vulnerabilities, and
is well taken. I think you're making a compelling argument.

> wrt clean slate - my references were to the constraints of tls 1.3 and
> h2.. we're going to want to keep those as unchanged as possible as design
> goals; so aligning with them is valuable and that's a constraint. Aligning
> is important, as they aren't layers isolated from each other in the same
> way TCP/TLS/7540-h2 are.
>

As I pointed out in my earlier email, I think that these are close but
separable. Specifically, when it comes to wire format, I think it's
reasonable to separate the processing of the various headers, which is what
matters for endianness. I don't think that QUIC's choice of endianness
should or will affect the wire formats of either TLS or H2, because, as I
pointed out, neither TLS nor H2 will be processing the QUIC wire format
directly.

> I may have been misunderstood to be referencing google-quic and I
> certainly appreciate a willingness to break with that.
>

Thanks for the clarification.

> does that make more sense?

It does. I still think, perhaps naively, that these wire formats are
separable. If QUIC version shows up in 3 places, it obeys the wire format
of the 3 places where it shows up. I can see how different endian-nesses
can leak though, but my point was that these could be well-specified where
there's confusion. That said, I think your point is "avoid the confusion in
the first place."

This may be worth a consensus call, since the spec says little-endian
currently, and we'll have to decide what to do upon adoption.

- jana

---

From: Brian Trammell <[email-masked]>
Date: Wed, 9 Nov 2016 13:20:32 +0100

hi Jana, all,

> On 08 Nov 2016, at 08:08, Jana Iyengar <[email-masked]> wrote:
> 
> On Mon, Nov 7, 2016 at 8:13 PM, Patrick McManus <[email-masked]> wrote:
> hey Jana,
> 

<snip>

> It does. I still think, perhaps naively, that these wire formats are separable. If QUIC version shows up in 3 places, it obeys the wire format of the 3 places where it shows up. I can see how different endian-nesses can leak though, but my point was that these could be well-specified where there's confusion. That said, I think your point is "avoid the confusion in the first place."

This was my point, at least.

Question for the design on the Google side: was this design choice taken as self-evident, or is there benchmarking on the performance impact in-protocol? The tradeoff between performance and the impact on violation principle of least surprise on debugging effort would be easier to evaluate with numbers in at least one column. :)

> This may be worth a consensus call, since the spec says little-endian currently, and we'll have to decide what to do upon adoption.

Though I'm happily dancing around it, I agree with Martin about this being a sarlacc pit. I guess those of us who have an opinion should find a time before the WG meeting, then?

Cheers,

Brian

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 9 Nov 2016 09:00:12 -0800

On Wed, Nov 9, 2016 at 4:20 AM, Brian Trammell <[email-masked]> wrote:

> Question for the design on the Google side: was this design choice taken
> as self-evident, or is there benchmarking on the performance impact
> in-protocol? The tradeoff between performance and the impact on violation
> principle of least surprise on debugging effort would be easier to evaluate
> with numbers in at least one column. :)
>

Given that every packet is encrypted and MACed I think some pretty
extraordinary evidence
is going to be required that little-endian has any significant performance
improvement.

-Ekr

> Though I'm happily dancing around it, I agree with Martin about this being
> a sarlacc pit. I guess those of us who have an opinion should find a time
> before the WG meeting, then?
>
> Cheers,
>
> Brian
>

---

From: Jana Iyengar <[email-masked]>
Date: Wed, 9 Nov 2016 18:21:08 -0800

> is going to be required that little-endian has any significant performance

No performance impact was measured. This choice was based on common-sense
and the fact that Chrome runs only on little-endian platforms, which is, to
a reasonable approximation, "all of them".

>> Though I'm happily dancing around it, I agree with Martin about this
>> being a sarlacc pit. I guess those of us who have an opinion should find a
>> time before the WG meeting, then?
>>
>
I don't think this needs to become a sarlacc pit, since we aren't really
arguing what universal choice of endianness is good. I suspect that a
consensus call will show if more discussion is needed? Especially given
that this is a change to the current draft, we'll need a call if adopted
anyways.