Subject: RE: Packet number encryption

From: "Roni Even (A)" <[email-masked]>
Date: Thu, 1 Feb 2018 08:04:55 +0000

+1
I think that this is where we need to evaluate the security versus manageability. It is not about the complexity of the solution, the proposed encryption is not perfect so we should decide what is good enough solution that will address security concerns and allow for network manageability.
Roni

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Mirja Kühlewind
Sent: Wednesday, January 31, 2018 9:45 AM
To: Martin Thomson
Cc: Brian Trammell; Eric Rescorla; QUIC WG; Christian Huitema
Subject: Re: Packet number encryption

> Am 31.01.2018 um 04:08 schrieb Martin Thomson <[email-masked]>:
> 
> On Wed, Jan 31, 2018 at 8:00 AM, Christian Huitema <[email-masked]> wrote:
>>> (1) An unprivileged on-path device that sees a packet from a flow 
>>> that is migrated on purpose by an endpoint (i.e., due to connection 
>>> migration or
>>> multipath) should not be able to associate that packet to the prior flow.
>> 
>> Yes.
> 
> Agreed.  This remains the primary target.  I don't think that there 
> are any negatives with this design with respect to this use case (or 
> at least I found nothing in Brian's email on this point).

If that is the goal, I think the complexity of the proposed solution is not justified. Just select a new random offset when you migrate but increase the packet number monotonously otherwise to support manageability.

I’m really concerned about complexity here. Even though this scheme is not very complex, any complexity is a potential source for future implementation errors which can also led to restrictions in what changes can be deploy with future versions. Making a protocol as simple as possible and thereby the spec as easy understandable as possible is also key for deployment. If we add complexity here without making it clear what the goal or additional benefit is of the complexity, I think that is just wrong.

My 2c.
Mirja

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Thu, 1 Feb 2018 09:08:08 +0100

hi Jana, all,

> On 1 Feb 2018, at 03:46, Jana Iyengar <[email-masked]> wrote:
> 
> I like this proposal for the ossification reason that Roberto described, but I also like it because it really simplifies things from an implementation perspective.

There seems to be a largely unquestioned assumption that packet number encryption has utility for ossification prevention. I am not convinced of this.

Our experience with TLS greasing shows that placing random values in fields that would otherwise largely be static over the lifetime of the protocol improves the ability to use those fields later. Packet number encryption is greasing only by analogy. "Field X has value Y" is a fundamentally different observation than "The values of field X over time are governed by function F", and I'm not sure that intuition in the simpler case applies directly to the more complicated one.

Specifically, for packet number encryption, the assumption appears to be that in-network devices will deploy during the operational lifetime of QUIC v1 that will make use of the observation that packet numbers monotonically increase, in a way that will impair the deployment of QUIC vN > 1 that does not have packet numbers that monotonically increase, or packet numbers that appear at a different location in the header. In other words, these devices will fail on this new version of QUIC in such a way that it will impair service to endpoints using QUIC vN.

Analogies to the TCP sequence number here are a bit flawed. The most important difference is that the open nature of TCP's wire image means that usage of the sequence number isn't limited to observation: on-path devices can change it, and forge or delay ACKs based on it. The only thing an on-path device can do with a QUIC packet with a packet number it doesn't like is to drop it. This function might have some utility as part of an in-network QUIC garbage filter, as part of an anti-DDoS system, but any such device would need to track the version number as well (indeed, tracking version negotiation and handshaking would be a useful primitive here in its own right), and would therefore have enough information to not break on new versions it knows about.

(Note we specifically don't care about the case where devices making use of this observation stop working in a way that the end-to-end properties of QUIC vN are preserved: the invariants as of a given version are the public wire-image interface to the protocol, anything else is at-your-own-risk, no-user-servicable-parts-inside).

I suspect that similar arguments can be had about all but the simplest proposed applications of greasing to functions more complex than "the value of field X is always Y or Z". The example of packet number tracking points to an anti-ossification approach that I have much more confidence in, though:

The vigorous and meaningful exercise of the version negotiation and version coexistence systems built into QUIC would force agility on on-path devices. The earlier we do this, the more effective it would be. I've already said this in other contexts, but I'll explicitly suggest it here: let's do QUIC 2, and soon (say, July 2019?), and use version negotiation *itself* as a method to "grease" the version negotiation mechanism along with the rest of the header.

> We currently have packet number gaps to be used with new CIDs, but these are the same packet numbers on the wire as within the transport. So, when used with connection migration or after quiescence, the sender and receiver of ACKs has to deal with potentially large gaps... this is encodable on the wire, sure, but these gaps are unnecessary within the transport.
> 
> Encrypting packet numbers reduces complexity in that you get a packet number gap on the wire without having to explicitly model it inside the transport. The transport machinery starts packet numbers at 0, increments it by 1 irrespective of migration or quiescence, and the ACK frames similarly carry the plaintext packet number. Separating what's used within the transport and what's visible on the wire allows you to do things like use multiple CIDs within the same 5-tuple.

This is a more convincing argument for packet number encryption -- it does seem hard to get this elegance in the transport while having a non-trivially-linkable wire image without it.

> I see it as a clean separation of concerns, with consequent simplification in implementation and an additional degree of freedom.
> 
> I'm not convinced of the value of packet number in measuring reordering degree, since this can be trivially done by looking at the IP ID field for packets on the same 5-tuple.

For IPv4 packets, on implementations where IPID is not randomized, yes. I don't have numbers on how prevalent these are; in any case, where it isn't randomized, that's arguably a bug that should be fixed. Should the guidance in the manageability draft, then, be "use IPv4 if you care about loss and reordering detection"? I am not optimistic about the chances of that getting past the IESG. ;)

Cheers,

Brian

> On Wed, Jan 31, 2018 at 6:00 PM, Martin Thomson <[email-masked]> wrote:
> With some light editing for quoting purposes...
> 
> On Thu, Feb 1, 2018 at 12:02 PM, Roberto Peon <[email-masked]> wrote:
> >> Or do you mean CIDs 1-3 may be used on any of the 5-tuples A-C, for the same
> >> communicating peers  Alice'sPhone and Bob'sBankServer93?
> >
> > The latter assuming we allow the server to specify it.
> 
> I find "assuming we allow the server to specify *it*" to be strangely cryptic.
> 
> To be clear, my operating assumption is while that servers can provide
> multiple connection IDs, those connection IDs need to be usable on any
> 5-tuple.  Are you suggesting that a server might need to signal where
> a connection is usable?  (I can see how that might make sense, but it
> isn't possible with the protocol as specified.)  Or are you suggesting
> that there might need to be some signal that allows clients to
> distinguish between connection IDs that are for concurrent use? That
> is, as opposed to what you might assume to be sequential use based on
> the current specification.
> 
> FWIW, it should be possible to remove the sequencing field from
> NEW_CONNECTION_ID after making this change.  The only reason left for
> sequential connection ID use is to give the server the ability to know
> when certain connection IDs no longer need to be available.  A server
> might then be able to clean up associated state and even reuse them.
> 
> Of course, that likely only makes sense if the server is creating
> explicit mappings for each, which - as discussed at the interim -
> isn't super practical in many deployments.  Right now, I think that
> the assumption is an assumption only, and that relying on strictly
> linear use of connection IDs would be unwise.
> 
>

---

From: Roberto Peon <[email-masked]>
Date: Thu, 1 Feb 2018 08:38:25 +0000

I'm assuming a server smart enough to encrypt routing info (i.e. l7 instance to which the l4 should forward) with sufficient random salt that the CID looks completely random to someone else (other than l4 or l7).

I was assuming for now that a CID so given was valid on any path.  It would, however, be perfectly reasonable (though not yet spec'd) for "real" multipath to specify CID on a per dst tuple basis (so per day tuple or for any dst tuple could be different CIDs), in which case the server would need a protocol mechanism. Ideally such a mechanism would allow an automatic change of CID when the client is participating in the new 5-tuple/path, potentially allowing detection of explicit client action (client explicitly binds to new 5-tuple) vs nat rebinding (oh, I have a new address,oops/how curious!), etc.

I was also thinking about l4/l7 shared key material. In particular, to deal with key rotation for the shared key material used to encrypt the CID without making a new connection, a server must both be able to provide CIDs, and be able revoke them. It must revoke (or equivalent) so that a client will have a finite lifetime with the old CID, presumably using the old key material.

-=R

Sent via the Samsung Galaxy S7, an AT&T 4G LTE smartphone

---

From: Martin Duke <[email-masked]>
Date: Thu, 1 Feb 2018 16:47:17 -0800

Brian, I can come up with a middlebox design that is trying to be "helpful"
but ends up causing PN ossification, but as you suggest this requires the
box to be dumb about versions. So, meh.

The other benefit of this scheme is that acks can go back to 1 byte PNs in
ack frames for many connections, as the true packet number will start at 0
(boo!) or 1 (yay!).

On Feb 1, 2018 12:09 AM, "Brian Trammell (IETF)" <[email-masked]> wrote:

hi Jana, all,

> I like this proposal for the ossification reason that Roberto described,
but I also like it because it really simplifies things from an
implementation perspective.

There seems to be a largely unquestioned assumption that packet number
encryption has utility for ossification prevention. I am not convinced of
this.

Our experience with TLS greasing shows that placing random values in fields
that would otherwise largely be static over the lifetime of the protocol
improves the ability to use those fields later. Packet number encryption is
greasing only by analogy. "Field X has value Y" is a fundamentally
different observation than "The values of field X over time are governed by
function F", and I'm not sure that intuition in the simpler case applies
directly to the more complicated one.

Specifically, for packet number encryption, the assumption appears to be
that in-network devices will deploy during the operational lifetime of QUIC
v1 that will make use of the observation that packet numbers monotonically
increase, in a way that will impair the deployment of QUIC vN > 1 that does
not have packet numbers that monotonically increase, or packet numbers that
appear at a different location in the header. In other words, these devices
will fail on this new version of QUIC in such a way that it will impair
service to endpoints using QUIC vN.

Analogies to the TCP sequence number here are a bit flawed. The most
important difference is that the open nature of TCP's wire image means that
usage of the sequence number isn't limited to observation: on-path devices
can change it, and forge or delay ACKs based on it. The only thing an
on-path device can do with a QUIC packet with a packet number it doesn't
like is to drop it. This function might have some utility as part of an
in-network QUIC garbage filter, as part of an anti-DDoS system, but any
such device would need to track the version number as well (indeed,
tracking version negotiation and handshaking would be a useful primitive
here in its own right), and would therefore have enough information to not
break on new versions it knows about.

(Note we specifically don't care about the case where devices making use of
this observation stop working in a way that the end-to-end properties of
QUIC vN are preserved: the invariants as of a given version are the public
wire-image interface to the protocol, anything else is at-your-own-risk,
no-user-servicable-parts-inside).

I suspect that similar arguments can be had about all but the simplest
proposed applications of greasing to functions more complex than "the value
of field X is always Y or Z". The example of packet number tracking points
to an anti-ossification approach that I have much more confidence in,
though:

The vigorous and meaningful exercise of the version negotiation and version
coexistence systems built into QUIC would force agility on on-path devices.
The earlier we do this, the more effective it would be. I've already said
this in other contexts, but I'll explicitly suggest it here: let's do QUIC
2, and soon (say, July 2019?), and use version negotiation *itself* as a
method to "grease" the version negotiation mechanism along with the rest of
the header.

> We currently have packet number gaps to be used with new CIDs, but these
are the same packet numbers on the wire as within the transport. So, when
used with connection migration or after quiescence, the sender and receiver
of ACKs has to deal with potentially large gaps... this is encodable on the
wire, sure, but these gaps are unnecessary within the transport.
>
> Encrypting packet numbers reduces complexity in that you get a packet
number gap on the wire without having to explicitly model it inside the
transport. The transport machinery starts packet numbers at 0, increments
it by 1 irrespective of migration or quiescence, and the ACK frames
similarly carry the plaintext packet number. Separating what's used within
the transport and what's visible on the wire allows you to do things like
use multiple CIDs within the same 5-tuple.

This is a more convincing argument for packet number encryption -- it does
seem hard to get this elegance in the transport while having a
non-trivially-linkable wire image without it.

> I see it as a clean separation of concerns, with consequent
simplification in implementation and an additional degree of freedom.
>
> I'm not convinced of the value of packet number in measuring reordering
degree, since this can be trivially done by looking at the IP ID field for
packets on the same 5-tuple.

For IPv4 packets, on implementations where IPID is not randomized, yes. I
don't have numbers on how prevalent these are; in any case, where it isn't
randomized, that's arguably a bug that should be fixed. Should the guidance
in the manageability draft, then, be "use IPv4 if you care about loss and
reordering detection"? I am not optimistic about the chances of that
getting past the IESG. ;)

Cheers,

Brian

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 2 Feb 2018 09:56:07 +0900

2018-01-31 19:00 GMT+09:00 Eggert, Lars <[email-masked]>:
> On 2018-1-31, at 10:52, Gorry Fairhurst <[email-masked]> wrote:
>> +1 - Simply: This *is* complicated and seems to add little.
>
> So as an implementor (chair hat off), this adds very little to the overall complexity of the protocol.

+1

The algorithm defined in the PR is easy to implement.

When using OpenSSL what you need to do is:

* when switching to a new pn_key, call EVP_EncryptInit_ex to setup a
CTR cipher (e.g., EVP_aes_128_ctr, EVP_chacha20) along with the pn_key
passed as the argument.

* for each packet, call EVP_EncryptInit_ex to set the IV to the 16
bytes of the AEAD ciphertext, then call EVP_EncryptUpdate to obtain
the encrypted packet number

Decryption is exactly the same as encryption, since it is CTR mode.

To me it seems that using packet number encryption actually simplifies
the implementation, since we no longer need to care about randomizing
the initial packet number or implement special measures to avoid
linkability when switching to a new CID.

> Lars

-- 
Kazuho Oku

---

From: Christian Huitema <[email-masked]>
Date: Thu, 1 Feb 2018 16:49:08 -1000

> linkability when switching to a new CID.

+1.

By the way, I wrote a small test vector for the PN encryption using AES,
at
https://github.com/quicwg/base-drafts/wiki/Test-vector-for-AES-packet-number-encryption.

-- Christian Huitema

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Fri, 2 Feb 2018 10:35:41 +0100

Sent from my iPhone

> On 2 Feb 2018, at 01:47, Martin Duke <[email-masked]> wrote:
> 
> Brian, I can come up with a middlebox design that is trying to be "helpful" but ends up causing PN ossification, but as you suggest this requires the box to be dumb about versions. So, meh.

The insight hiding in my proposal about rapid versioning is exactly this: we should have a goal to minimize the deployability of boxes that are dumb about versions. Grease the version negotiation mechanism here is fake: it protects the codepoints but not the whole mechanism. As long as there is only one version you’ll get boxes that don’t need to care about tracking VN to know what’s going on. If version 2 follows on the heels of (or deploys simultaneously with) version 1, there will be fewer (none) of those boxes deployed.

> The other benefit of this scheme is that acks can go back to 1 byte PNs in ack frames for many connections, 

You get that with the random offset proposal as well.

As far as I can see the benefits of this proposal are (1) it lets us presuppose a single-sequence-number design for multipath in version 2, which has some nice properties; and (2) it lets us dispense with all the messiness of PN jumps on intentional migration. The first of these seems meh, the second is a little more convincing: packet number jumping is not pretty.

> as the true packet number will start at 0 (boo!) or 1 (yay!).

I actually have strong opinions on both sides of this argument, so I think I’ll go off in the corner and have a pointless fight with myself on this one.

Cheers,

Brian

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 2 Feb 2018 02:54:00 -0800

and (2) it lets us dispense with all the messiness of PN jumps on
intentional migration. The first of these seems meh, the second is a little
more convincing: packet number jumping is not pretty.

I have proposed a defence against optimistic ACK attack the avoids the use
of packet number gaps and which makes it mandatory for implementations to
support:
https://github.com/quicwg/base-drafts/issues/1092

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 2 Feb 2018 13:24:14 +0100

Please see below.

> Am 02.02.2018 um 01:56 schrieb Kazuho Oku <[email-masked]>:
> 

I didn’t say that it is too complex (I actually explicitly said in my original mail  it is not very complex) but it adds additional complexity to a protocol that is already complex and doesn’t provide much benefit from my point of view but can be an additional source for errors. After all it must also be possible to implement QUIC if you just read the final specs and have not participated in the wg design discussion or can easy ask a clarifying question on the list.

> When using OpenSSL what you need to do is:

If you don’t want to use OpenSSL or are in an environment where you can use such a library, it is definitely more complex than this.

> linkability when switching to a new CID.

Choosing a random number is definitely less complex than encrypting something. It might simply your current code because you are using a library that implements the complexity already. And special measures when switch to a new path are needed anyway for the CID.

Mirja

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 2 Feb 2018 13:27:33 +0100

Hi Jana,

one point below, just to clarify this explicitly.

> Am 01.02.2018 um 03:46 schrieb Jana Iyengar <[email-masked]>:
> 

You also have this with an random offset. You internally start with 0 or 1 and just add or remove the offset from the PN in the wire image. Done. I like this separation a lot and think that is a  great change as it does make implementation easier and reduces overhead. However, these are two different things. There is no need to encrypt the PM just for that.

Mirja

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 2 Feb 2018 22:13:47 +0900

2018-02-02 21:24 GMT+09:00 Mirja Kühlewind <[email-masked]>:
> I didn’t say that it is too complex (I actually explicitly said in my original mail  it is not very complex) but it adds additional complexity to a protocol that is already complex and doesn’t provide much benefit from my point of view but can be an additional source for errors.

Regarding CID migration, I would argue that the current draft is
already complex due to the use of HKDF based on a derived key to
calculate the gap. But it still has issues (please refer to
https://github.com/quicwg/base-drafts/issues/1034).

Switching to packet number encryption resolves those issues. You no
longer need to do special calculation of the packet number upon CID
migration.

Having a consistent logic to handle packet numbers is IMO a better
choice than requiring complex operations in a corner case.

> After all it must also be possible to implement QUIC if you just read the final specs and have not participated in the wg design discussion or can easy ask a clarifying question on the list.
>
>>

I agree that the mileage may vary depending on the crypto library you use.

However, I would also argue that it would be fairly natural to assume
that you have access to the underlying cipher of an AEAD cipher that
you would be using (e.g. AES-ECB (or AES-CTR) for AES-GCM, Chacha20
for Chacha20-Poly1305). Then, it is easy to construct the proposed
logic, though you might need to write code specific to each cipher.

> Choosing a random number is definitely less complex than encrypting something. It might simply your current code because you are using a library that implements the complexity already. And special…

I agree that calculating a random number is not a big deal. Regarding
the amount of the special measures in relation to CID migration,
please refer to my comment above.

-- 
Kazuho Oku

---

From: Dmitri Tikhonov <[email-masked]>
Date: Fri, 2 Feb 2018 10:08:13 -0500

On Fri, Feb 02, 2018 at 10:35:41AM +0100, Brian Trammell (IETF) wrote:
> I actually have strong opinions on both sides of this argument, so
> I think I’ll go off in the corner and have a pointless fight with
> myself on this one.

How strong is the preference for starting off the packet sequence with
zero?  Zero is such a convenient (and efficient) special value,
programming-wise.

  - Dmitri.

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Fri, 2 Feb 2018 16:13:54 +0100

> On 2 Feb 2018, at 16:08, Dmitri Tikhonov <[email-masked]> wrote:
> 

Yeah, in this case, starting at one is far preferable so as to use 0 as the special value, but counting from one in general just feels dirty to people who grew up with C.

(This bikeshed has been painted so often that it's made of paint, though, so I apologize for even referring to it in passing.)

---

From: "Fossati, Thomas (Nokia - GB/Cambridge, UK)" <[email-masked]>
Date: Fri, 2 Feb 2018 15:32:49 +0000

> overall complexity of the protocol.

This doesn't sound great.  It's a bit like saying that adding the Higgs
mechanism to the standard model's lagrangian doesn't make it much more
complicate :-)  (*)

More seriously though, I'd like to point out that it is not just about
implementation complexity, it's also the energy cost per packet that is
quite crucial.  I haven't done the math but ISTM that bringing in
another batch of non-optional crypto computation on a per packet basis
is not going to move the needle in the right direction for stacks that
run on low power (IoT).

This is not a catastrophe - we have CoAP and a (D)TLS profile - but
neither it's ideal, since cutting off the small things from the wider
ecosystem creates an artificial gap which then will need a middlebox to
bridge.  (Sure, we have specified the behaviour of a similar box
already, but it'd be really better if the extra translation logic could
be avoided in the first place.)

I guess what I'm saying is that PN encryption being a core mechanism
that can't be negotiated at handshake time reduces our ability to later
profile QUIC for the IoT, which would be a bit unlucky.

So the question is: would it be possible to make this property a
configurable knob instead?

(*) https://www.symmetrymagazine.org/article/the-deconstructed-standard-model-equation

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 2 Feb 2018 18:30:53 -0800

A few points as I catch up on this thread.

First, I'll remind folks that QUIC is an encrypted transport. I say this
because the cost of this operation is trivial in the context of encrypting
every packet. The cost is borne at the servers and not at middleboxes, so
the additional crypto cost is basically trivial.

Second, this simplifies and increases robustness of implementation.
Avoiding random PN jumps, as Kazuho points out, makes special-casing of
code unnecessary. Special code that is exercised occasionally is a strong
bug attractor, and I would strongly argue for as few of those as possible
in implementation.

Third, yes, ossification is a real concern. A simple example: using
increasing packet numbers as a signature for detecting QUIC. Even if you
argue against this example, there are innovative ways in which ossification
will happen. This is based on a true story: We had no idea how GQUIC's
flags field could get ossified, the value that was being used commonly
became used as a signature for QUIC traffic (see Section 7.5 in the SIGCOMM
QUIC paper
<https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46403.pdf>
).

There's a win here in terms of implementation complexity and several
implementers have said so. There's a win in terms of ossification and our
experience says so. There's a potential loss of manageability, in being
able to detect reordering. This is the trade-off, and I am still in favor
of encrypting packet numbers.

On Fri, Feb 2, 2018 at 7:32 AM, Fossati, Thomas (Nokia - GB/Cambridge, UK) <
[email-masked]> wrote:

> On 31/01/2018, 10:06, "QUIC on behalf of Eggert, Lars" <
> [email-masked] on behalf of [email-masked]> wrote:

---

From: Piotr Galecki <[email-masked]>
Date: Sat, 3 Feb 2018 02:59:52 +0000

What is the group’s proposal for allowing network monitoring tools to detect packet retransmission or packet reordering ?
These metrics are pretty straightforward to measure for TCP flows.
Tools like Wireshark can analyze packets and spot issues.
How can the network tools perform this kind of measurements for QUIC flows if PN field is encrypted?

From: QUIC [mailto:[email-masked]] On Behalf Of Jana Iyengar
Sent: Friday, February 02, 2018 9:31 PM
To: Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]>
Cc: Gorry Fairhust <[email-masked]>; Eric Rescorla <[email-masked]>; Mirja Kühlewind <[email-masked]>; Christian Huitema <[email-masked]>; Brian Trammell <[email-masked]>; Eggert, Lars <[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: Re: Packet number encryption

First, I'll remind folks that QUIC is an encrypted transport. I say this because the cost of this operation is trivial in the context of encrypting every packet. The cost is borne at the servers and not at middleboxes, so the additional crypto cost is basically trivial.

Second, this simplifies and increases robustness of implementation. Avoiding random PN jumps, as Kazuho points out, makes special-casing of code unnecessary. Special code that is exercised occasionally is a strong bug attractor, and I would strongly argue for as few of those as possible in implementation.

Third, yes, ossification is a real concern. A simple example: using increasing packet numbers as a signature for detecting QUIC. Even if you argue against this example, there are innovative ways in which ossification will happen. This is based on a true story: We had no idea how GQUIC's flags field could get ossified, the value that was being used commonly became used as a signature for QUIC traffic (see Section 7.5 in the SIGCOMM QUIC paper<https://static.googleusercontent.com/media/research.google.com/en/pubs/archive/46403.pdf>).

There's a win here in terms of implementation complexity and several implementers have said so. There's a win in terms of ossification and our experience says so. There's a potential loss of manageability, in being able to detect reordering. This is the trade-off, and I am still in favor of encrypting packet numbers.

On Fri, Feb 2, 2018 at 7:32 AM, Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]<mailto:[email-masked]>> wrote:
On 31/01/2018, 10:06, "QUIC on behalf of Eggert, Lars" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:
> On 2018-1-31, at 10:52, Gorry Fairhurst <[email-masked]<mailto:[email-masked]>> wrote:

---

From: <[email-masked]>
Date: Sat, 3 Feb 2018 13:16:55 +0100

On -10/01/-28163 20:59, Jana Iyengar wrote:
> experience says so. There's a potential loss of manageability, in being able to
> detect reordering. This is the trade-off, and I am still in favor of encrypting
> packet numbers.

Given this, what's your position re the 4-bit subsequence number in clear 
proposed earlier ? Is it okay, or has it got too much potential for a signature ?

-Alex

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Patrick McManus <[email-masked]>
Date: Sat, 3 Feb 2018 09:04:20 -0500

On Fri, Feb 2, 2018 at 9:59 PM, Piotr Galecki <
[email-masked]> wrote:

> What is the group’s proposal for allowing network monitoring tools to
> detect packet retransmission or packet reordering ?
>

quic retransmits data, it does not retransmit packets. I'm not trying to be
pedantic - that's the answer to your question because obviously data needs
keys. Cooperating parties need to cooperate to share keying material...
this is something wireshark and tls have worked out in the past.

> *From:* QUIC [mailto:[email-masked]] *On Behalf Of *Jana Iyengar
> *Sent:* Friday, February 02, 2018 9:31 PM
> *To:* Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]
> >
> *Cc:* Gorry Fairhust <[email-masked]>; Eric Rescorla <[email-masked]>;
> Mirja Kühlewind <[email-masked]>; Christian Huitema <
> [email-masked]>; Brian Trammell <[email-masked]>; Eggert, Lars <
> [email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <
> [email-masked]>
> *Subject:* Re: Packet number encryption
>
>
>
> <https://static.googleusercontent.com/media/research.google.com/en/pubs/archive/46403.pdf>
> ).
>
>
>
> On Fri, Feb 2, 2018 at 7:32 AM, Fossati, Thomas (Nokia - GB/Cambridge, UK)
> <[email-masked]> wrote:
>

---

From: "Fossati, Thomas (Nokia - GB/Cambridge)" <[email-masked]>
Date: Sat, 3 Feb 2018 15:04:00 +0000

On 03/02/2018, 02:31, "Jana Iyengar" <[email-masked]> wrote:
> First, I'll remind folks that QUIC is an encrypted transport. I say
> this because the cost of this operation is trivial in the context of
> encrypting every packet. The cost is borne at the servers and not at
> middleboxes, so the additional crypto cost is basically trivial.

"QUIC is an encrypted transport" is bit vague a definition.  And it is
exactly that vagueness that we are discussing here, I think, that is:
deciding when grey becomes one of white or black.

Going back for a second to IoT: imagine using short header with no CID &
tiny payloads - e.g., the typical 2-4 bytes periodic sensor reading.  In
this case, PN encryption would increase the crypto compute per packet by
something around 50%, which can make a substantial difference on
low-power, battery operated devices.

This is a rather extreme case, but it serves to illustrate the point
that when moving in that greyish space, it makes sense to maybe allow
run-time choices instead of forcing a specific policy, which has the
potential to alienate an entire class of use cases, at design time.

Let endpoints negotiate the grey feature and make it black or white
depending on their cost-benefit judgement.

In particular, from a security/privacy perspective, PN encryption looks
entirely like a client decision.

And then there is the manageability issue that Roni and Piotr have
raised, which is very important, and should be weighted in the feature
selection process carefully.  It's not great for people sitting in NOC
rooms to have their toolboxes and expertise annihilated by a major
rollout at some big content provider.

Re: the ossification concern.  I'm am unable to assess the advantage PN
encryption would bring compared to Brian's approach of changing the wire
image frequently and have co-existing incompatible versions - mainly
because it's pretty difficult to tell now at what rate new QUIC versions
will be rolled out.

Cheers!

---

From: Stephen Farrell <[email-masked]>
Date: Sat, 3 Feb 2018 15:33:42 +0000

> low-power, battery operated devices.

Sorry - isn't the duration for which the radio is turned on
a far more important factor for power consumption in such
cases? I don't recall a real case being described where the
kind of crypto being described here would make a significant
difference but maybe you can describe one?

Ta,
S.

-- 
PGP key change time for me.
New-ID 7B172BEA; old-ID 805F8DA2 expires Jan 24 2018.
NewWithOld sigs in keyservers.
Sorry if that mucks something up;-)

---

From: Roberto Peon <[email-masked]>
Date: Sat, 3 Feb 2018 18:37:56 +0000

Ossification:
In the past tcp middleboxes expected in-order, monotonically increasing offsets and tried to fix reordering before forwarding. This middlebox behavior, predicated on observable flows, prevented deployment of things which would have made tcp more efficient.

 This is the kind of non-theoretical ossification that drove the creation of QUIC in the first place.

Network monitoring:
Networks may always wrap the packet(s) they have received with whatever data they wish, and unwrap before forwarding. This can allow any network to understand ordering,  loss, etc within its boundaries.

Debugging:
Debugging of flows can still happen for flows that wish to be debugged-- the QUIC connection's key material(s) can be shared manually and used by the debugging tool. This is how one debugs http2, https, etc. already today.

-=R

-------- Original message --------
From: Piotr Galecki <[email-masked]>
Date: 2/2/18 7:00 PM (GMT-08:00)
To: Jana Iyengar <[email-masked]>, "Fossati, Thomas (Nokia - GB/Cambridge, UK)" <[email-masked]>
Cc: Gorry Fairhust <[email-masked]>, Eric Rescorla <[email-masked]>, Christian Huitema <[email-masked]>, "Eggert, Lars" <[email-masked]>, Brian Trammell <[email-masked]>, Mirja Kühlewind <[email-masked]>, QUIC WG <[email-masked]>, Martin Thomson <[email-masked]>
Subject: RE: Packet number encryption

Third, yes, ossification is a real concern. A simple example: using increasing packet numbers as a signature for detecting QUIC. Even if you argue against this example, there are innovative ways in which ossification will happen. This is based on a true story: We had no idea how GQUIC's flags field could get ossified, the value that was being used commonly became used as a signature for QUIC traffic (see Section 7.5 in the SIGCOMM QUIC paper<https://urldefense.proofpoint.com/v2/url?u=https-3A__static.googleusercontent.com_media_research.google.com_en_pubs_archive_46403.pdf&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=SY87I4v_IqTCGOds-j5NCOwkQ0cJcYHGyKdXmMIVn6I&s=U9o6dI-dwzKwvXHvvmv7WtVm84puvxhaP0cN2of3VvA&e=>).

(*) https://www.symmetrymagazine.org/article/the-deconstructed-standard-model-equation<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.symmetrymagazine.org_article_the-2Ddeconstructed-2Dstandard-2Dmodel-2Dequation&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=SY87I4v_IqTCGOds-j5NCOwkQ0cJcYHGyKdXmMIVn6I&s=etIO0AUHiJm8zucsWaTgo-XULFisCMoG2iDvwC69s9Q&e=>

---

From: Piotr Galecki <[email-masked]>
Date: Sat, 3 Feb 2018 19:23:33 +0000

It is very unlikely that content provider would share keying material with a network operator.
If a subscriber makes a call to a network operator that it is getting poor video quality
the network operator personnel would debug packet stream delivery issues.
With TCP the stream can be captured and tools can be used to measure packet/data loss, level of retransmissions or misordering.
This can be done in different points in the network.
With QUIC it sounds like network troubleshooting will be much more difficult.

From: Patrick McManus [mailto:[email-masked]]
Sent: Saturday, February 03, 2018 9:04 AM
To: Piotr Galecki <[email-masked]>
Cc: Jana Iyengar <[email-masked]>; Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]>; Gorry Fairhust <[email-masked]>; Eric Rescorla <[email-masked]>; Christian Huitema <[email-masked]>; Eggert, Lars <[email-masked]>; Brian Trammell <[email-masked]>; Mirja Kühlewind <[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: Re: Packet number encryption

On Fri, Feb 2, 2018 at 9:59 PM, Piotr Galecki <[email-masked]<mailto:[email-masked]>> wrote:
What is the group’s proposal for allowing network monitoring tools to detect packet retransmission or packet reordering ?

quic retransmits data, it does not retransmit packets. I'm not trying to be pedantic - that's the answer to your question because obviously data needs keys. Cooperating parties need to cooperate to share keying material... this is something wireshark and tls have worked out in the past.

From: QUIC [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Jana Iyengar
Sent: Friday, February 02, 2018 9:31 PM
To: Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]<mailto:[email-masked]>>
Cc: Gorry Fairhust <[email-masked]<mailto:[email-masked]>>; Eric Rescorla <[email-masked]<mailto:[email-masked]>>; Mirja Kühlewind <[email-masked]<mailto:[email-masked]>>; Christian Huitema <[email-masked]<mailto:[email-masked]>>; Brian Trammell <[email-masked]<mailto:[email-masked]>>; Eggert, Lars <[email-masked]<mailto:[email-masked]>>; QUIC WG <[email-masked]<mailto:[email-masked]>>; Martin Thomson <[email-masked]<mailto:[email-masked]>>
Subject: Re: Packet number encryption

---

From: "Roni Even (A)" <[email-masked]>
Date: Sun, 4 Feb 2018 07:47:06 +0000

Hi,
Network monitoring:

RE- so you are saying that we can for example  leverage  RTP and define RTP payload for QUIC that will carry QUIC packets, we will just need to resolve the MTU size.
Roni

From: QUIC [mailto:[email-masked]] On Behalf Of Roberto Peon
Sent: Saturday, February 03, 2018 8:38 PM
To: Piotr Galecki; Jana Iyengar; Fossati, Thomas (Nokia - GB/Cambridge, UK)
Cc: Gorry Fairhust; Eric Rescorla; Mirja Kühlewind; Eggert, Lars; Brian Trammell; Christian Huitema; QUIC WG; Martin Thomson
Subject: Re: Packet number encryption

Ossification:

Network monitoring:

Debugging:

-=R

-------- Original message --------
From: Piotr Galecki <[email-masked]<mailto:[email-masked]>>
Date: 2/2/18 7:00 PM (GMT-08:00)
To: Jana Iyengar <[email-masked]<mailto:[email-masked]>>, "Fossati, Thomas (Nokia - GB/Cambridge, UK)" <[email-masked]<mailto:[email-masked]>>
Cc: Gorry Fairhust <[email-masked]<mailto:[email-masked]>>, Eric Rescorla <[email-masked]<mailto:[email-masked]>>, Christian Huitema <[email-masked]<mailto:[email-masked]>>, "Eggert, Lars" <[email-masked]<mailto:[email-masked]>>, Brian Trammell <[email-masked]<mailto:[email-masked]>>, Mirja Kühlewind <[email-masked]<mailto:[email-masked]>>, QUIC WG <[email-masked]<mailto:[email-masked]>>, Martin Thomson <[email-masked]<mailto:[email-masked]>>
Subject: RE: Packet number encryption

What is the group's proposal for allowing network monitoring tools to detect packet retransmission or packet reordering ?
These metrics are pretty straightforward to measure for TCP flows.
Tools like Wireshark can analyze packets and spot issues.
How can the network tools perform this kind of measurements for QUIC flows if PN field is encrypted?

---

From: "Lubashev, Igor" <[email-masked]>
Date: Sun, 4 Feb 2018 17:18:13 +0000

The need to troubleshoot networks (at a minimum, to detect packet loss and reordering) is real.  I would assume that every network would need to have an ability to do so.  Ability to quickly troubleshoot and fix networks is not only "nice" for the networks themselves, but it is also very beneficial to QUIC users.

Having networks wrap and unwrap packets at their boundaries is a possibility, though it has real costs and downsides.  First, while networks could setup such wrapping for UDP port 443 traffic, what about QUIC traffic using other ports?  Would networks need to wrap all UDP traffic?  Second, this does reduce MTU for end users, reducing available bandwidth.  Third, this would allow networks to only answer questions like: "Is my network the culprit of the problem?" instead of "Is the problem in my network, upstream from me, or downstream from me?".

Would it be considered outrageous to suggest that packet number encryption be an option negotiated during connection handshake?  An end user client in "Privacy-Enhanced" mode would negotiate this, while end users not particularly concerned with connection migration linkability issues would leave this off?

- Igor

From: Roni Even (A) [mailto:[email-masked]]
Sent: Sunday, February 04, 2018 2:47 AM
To: Roberto Peon <[email-masked]>; Piotr Galecki <[email-masked]>; Jana Iyengar <[email-masked]>; Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]>
Cc: Gorry Fairhust <[email-masked]>; Eric Rescorla <[email-masked]>; Christian Huitema <[email-masked]>; Eggert, Lars <[email-masked]>; Brian Trammell <[email-masked]>; Mirja Kühlewind <[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: RE: Packet number encryption

Hi,
Network monitoring:

Ossification:

Network monitoring:

Debugging:

-=R

---

From: "Lubashev, Igor" <[email-masked]>
Date: Sun, 4 Feb 2018 17:33:23 +0000

*   Would it be considered outrageous to suggest that packet number encryption be an option negotiated during connection handshake?

Answering my own question, the trade-off is, of course, is that this opens up the possibility of network operators choosing to prohibit "enhanced privacy" connections by breaking those connections.

The dilemma may still be there - QUIC's primary goal is to ensure better end user performance, but would that goal be satisfied, if networks find it much harder to troubleshoot QUIC traffic?  Or is there no dilemma, and networks will find ways to effectively troubleshoot QUIC traffic on par with what they can do with TCP?

From: Lubashev, Igor
Sent: Sunday, February 04, 2018 12:18 PM
To: 'Roni Even (A)' <[email-masked]>; Roberto Peon <[email-masked]>; Piotr Galecki <[email-masked]>; Jana Iyengar <[email-masked]>; Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]>
Cc: Gorry Fairhust <[email-masked]>; Eric Rescorla <[email-masked]>; Christian Huitema <[email-masked]>; Eggert, Lars <[email-masked]>; Brian Trammell <[email-masked]>; Mirja Kühlewind <[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: RE: Packet number encryption

- Igor

From: Roni Even (A) [mailto:[email-masked]]
Sent: Sunday, February 04, 2018 2:47 AM
To: Roberto Peon <[email-masked]<mailto:[email-masked]>>; Piotr Galecki <[email-masked]<mailto:[email-masked]>>; Jana Iyengar <[email-masked]<mailto:[email-masked]>>; Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]<mailto:[email-masked]>>
Cc: Gorry Fairhust <[email-masked]<mailto:[email-masked]>>; Eric Rescorla <[email-masked]<mailto:[email-masked]>>; Christian Huitema <[email-masked]<mailto:[email-masked]>>; Eggert, Lars <[email-masked]<mailto:[email-masked]>>; Brian Trammell <[email-masked]<mailto:[email-masked]>>; Mirja Kühlewind <[email-masked]<mailto:[email-masked]>>; QUIC WG <[email-masked]<mailto:[email-masked]>>; Martin Thomson <[email-masked]<mailto:[email-masked]>>
Subject: RE: Packet number encryption

Hi,
Network monitoring:

Ossification:

Network monitoring:

Debugging:

-=R

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sun, 4 Feb 2018 09:35:48 -0800

If you need to troubleshoot a network, what prevents you from injecting
probe packets with random content on the same route? The more opaque QUIC
is, the more likely it is that probe packets would behave the same.

For that matter, QUIC could define stateless probe packets that must route
the same as real connections.

Mikkel

On 4 February 2018 at 18.19.16, Lubashev, Igor ([email-masked]) wrote:

The need to troubleshoot networks (at a minimum, to detect packet loss and
reordering) is real.  I would assume that every network would need to have
an ability to do so.  Ability to quickly troubleshoot and fix networks is
not only “nice” for the networks themselves, but it is also very beneficial
to QUIC users.

Having networks wrap and unwrap packets at their boundaries is a
possibility, though it has real costs and downsides.  First, while networks
could setup such wrapping for UDP port 443 traffic, what about QUIC traffic
using other ports?  Would networks need to wrap all UDP traffic?  Second,
this does reduce MTU for end users, reducing available bandwidth.  Third,
this would allow networks to only answer questions like: “Is my network the
culprit of the problem?” instead of “Is the problem in my network, upstream
from me, or downstream from me?”.

Would it be considered outrageous to suggest that packet number encryption
be an option negotiated during connection handshake?  An end user client in
“Privacy-Enhanced” mode would negotiate this, while end users not
particularly concerned with connection migration linkability issues would
leave this off?

- Igor

*From:* Roni Even (A) [mailto:[email-masked]]
*Sent:* Sunday, February 04, 2018 2:47 AM
*To:* Roberto Peon <[email-masked]>; Piotr Galecki <
[email-masked]>; Jana Iyengar <[email-masked]>;
Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]>
*Cc:* Gorry Fairhust <[email-masked]>; Eric Rescorla <[email-masked]>;
Christian Huitema <[email-masked]>; Eggert, Lars <[email-masked]>;
Brian Trammell <[email-masked]>; Mirja Kühlewind <
[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <
[email-masked]>
*Subject:* RE: Packet number encryption

Hi,

Network monitoring:

Networks may always wrap the packet(s) they have received with whatever
data they wish, and unwrap before forwarding. This can allow any network to
understand ordering,  loss, etc within its boundaries.

RE- so you are saying that we can for example  leverage  RTP and define RTP
payload for QUIC that will carry QUIC packets, we will just need to resolve
the MTU size.

Roni

*From:* QUIC [mailto:[email-masked] <[email-masked]>] *On
Behalf Of *Roberto Peon
*Sent:* Saturday, February 03, 2018 8:38 PM
*To:* Piotr Galecki; Jana Iyengar; Fossati, Thomas (Nokia - GB/Cambridge,
UK)
*Cc:* Gorry Fairhust; Eric Rescorla; Mirja Kühlewind; Eggert, Lars; Brian
Trammell; Christian Huitema; QUIC WG; Martin Thomson
*Subject:* Re: Packet number encryption

Ossification:

In the past tcp middleboxes expected in-order, monotonically increasing
offsets and tried to fix reordering before forwarding. This middlebox
behavior, predicated on observable flows, prevented deployment of things
which would have made tcp more efficient.

 This is the kind of non-theoretical ossification that drove the creation
of QUIC in the first place.

Network monitoring:

Networks may always wrap the packet(s) they have received with whatever
data they wish, and unwrap before forwarding. This can allow any network to
understand ordering,  loss, etc within its boundaries.

Debugging:

Debugging of flows can still happen for flows that wish to be debugged--
the QUIC connection's key material(s) can be shared manually and used by
the debugging tool. This is how one debugs http2, https, etc. already today.

-=R

-------- Original message --------

To: Jana Iyengar <[email-masked]>, "Fossati, Thomas (Nokia - GB/Cambridge,
UK)" <[email-masked]>

Cc: Gorry Fairhust <[email-masked]>, Eric Rescorla <[email-masked]>,
Christian Huitema <[email-masked]>, "Eggert, Lars" <[email-masked]>,
Brian Trammell <[email-masked]>, Mirja Kühlewind <
[email-masked]>, QUIC WG <[email-masked]>, Martin Thomson <
[email-masked]>

How can the network tools perform this kind of measurements for QUIC flows
if PN field is encrypted?

*From:* QUIC [mailto:[email-masked] <[email-masked]>] *On
Behalf Of *Jana Iyengar
*Sent:* Friday, February 02, 2018 9:31 PM
*To:* Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]>
*Cc:* Gorry Fairhust <[email-masked]>; Eric Rescorla <[email-masked]>;
Mirja Kühlewind <[email-masked]>; Christian Huitema <
[email-masked]>; Brian Trammell <[email-masked]>; Eggert, Lars <
[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <
[email-masked]>
*Subject:* Re: Packet number encryption

<https://urldefense.proofpoint.com/v2/url?u=https-3A__static.googleusercontent.com_media_research.google.com_en_pubs_archive_46403.pdf&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=SY87I4v_IqTCGOds-j5NCOwkQ0cJcYHGyKdXmMIVn6I&s=U9o6dI-dwzKwvXHvvmv7WtVm84puvxhaP0cN2of3VvA&e=>
).

(*)
https://www.symmetrymagazine.org/article/the-deconstructed-standard-model-equation
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.symmetrymagazine.org_article_the-2Ddeconstructed-2Dstandard-2Dmodel-2Dequation&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=SY87I4v_IqTCGOds-j5NCOwkQ0cJcYHGyKdXmMIVn6I&s=etIO0AUHiJm8zucsWaTgo-XULFisCMoG2iDvwC69s9Q&e=>

---

From: Christian Huitema <[email-masked]>
Date: Sun, 4 Feb 2018 07:40:41 -1000

> On Feb 4, 2018, at 7:18 AM, Lubashev, Igor <[email-masked]> wrote:
> ...
>  
> Having networks wrap and unwrap packets at their boundaries is a possibility, though it has real costs and downsides.  First, while networks could setup such wrapping for UDP port 443 traffic, what about QUIC traffic using other ports?  Would networks need to wrap all UDP traffic?  Second, this does reduce MTU for end users, reducing available bandwidth.  Third, this would allow networks to only answer questions like: “Is my network the culprit of the problem?” instead of “Is the problem in my network, upstream from me, or downstream from me?”.

This is exactly why I believe the proper place for things like spin bits or short sequence numbers is in the IP header. That would be a really neat use of the IPv6 flow ID.
>  
> Would it be considered outrageous to suggest that packet number encryption be an option negotiated during connection handshake?  An end user client in “Privacy-Enhanced” mode would negotiate this, while end users not particularly concerned with connection migration linkability issues would leave this off?

Negotiate privacy off in the name of network performance? Very slippery slope! For example, the connections that do require privacy would stick out.

-- Christian Huitema

---

From: "Lubashev, Igor" <[email-masked]>
Date: Sun, 4 Feb 2018 17:48:58 +0000

If an end user is complaining of a problem watching a video stream, extraneous probe packets would not help to identify what exactly happened to user’s packets.  In fact, just capturing QUIC’s encrypted packets numbers on two devices within your network would probably be enough to identify any issues between those devices.

So it should be possible to rule in or rule out your own network, but it still would be impossible to answer the question: “If user claims a problem, and my networks seems ok, is the a problem upstream or downstream from me?”

  *   Igor

From: Mikkel Fahnøe Jørgensen [mailto:[email-masked]]
Sent: Sunday, February 04, 2018 12:36 PM
To: Roni Even (A) <[email-masked]>; Jana Iyengar <[email-masked]>; Roberto Peon <[email-masked]>; Lubashev, Igor <[email-masked]>; Piotr Galecki <[email-masked]>; Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]>
Cc: QUIC WG <[email-masked]>; Brian Trammell <[email-masked]>; Martin Thomson <[email-masked]>; Eric Rescorla <[email-masked]>; Mirja Kühlewind <[email-masked]>; Christian Huitema <[email-masked]>; Gorry Fairhust <[email-masked]>; Eggert, Lars <[email-masked]>
Subject: RE: Packet number encryption

If you need to troubleshoot a network, what prevents you from injecting probe packets with random content on the same route? The more opaque QUIC is, the more likely it is that probe packets would behave the same.

For that matter, QUIC could define stateless probe packets that must route the same as real connections.

Mikkel

On 4 February 2018 at 18.19.16, Lubashev, Igor ([email-masked]<mailto:[email-masked]>) wrote:
The need to troubleshoot networks (at a minimum, to detect packet loss and reordering) is real.  I would assume that every network would need to have an ability to do so.  Ability to quickly troubleshoot and fix networks is not only “nice” for the networks themselves, but it is also very beneficial to QUIC users.

- Igor

From: Roni Even (A) [mailto:[email-masked]<mailto:[email-masked]>]

Hi,
Network monitoring:

Ossification:

Network monitoring:

Debugging:

-=R

---

From: Stephen Farrell <[email-masked]>
Date: Sun, 4 Feb 2018 18:51:29 +0000

Hiya,

On 04/02/18 17:18, Lubashev, Igor wrote:
> Would it be considered outrageous to suggest that packet number
> encryption be an option negotiated during connection handshake?  An
> end user client in "Privacy-Enhanced" mode would negotiate this,
> while end users not particularly concerned with connection migration
> linkability issues would leave this off?

I agree with Christian's points.

In addition, I'd like to point out a flaw in the above - the question
is posed in terms of an "end user client" making decisions for "end
users not particularly concerned" with a privacy enhancing feature. I
don't believe that programmers can really determine such things (no
matter how often someone says they "care deeply";-) so would hope the
WG adopts an approach of not making such features optional for this
reason, as well as for the usual "more options == bad plan" reasons.

Cheers,
S.

---

From: "Fossati, Thomas (Nokia - GB/Cambridge)" <[email-masked]>
Date: Sun, 4 Feb 2018 19:11:06 +0000

On 03/02/2018, 15:33, "Stephen Farrell" <[email-masked]>
wrote:
> Sorry - isn't the duration for which the radio is turned on a far more
> important factor for power consumption in such cases?

Hi Stephen, that's a very good point; apparently, I could not see the
wood for the trees!

You are right, we are talking about a symmetric cipher here, so the work
for even doubling the amount of CPU per packet will probably be dwarfed
by the energy it takes to send the frame out in the air - activate the
radio transceiver, assess the channel, actually send the data, switch
back to receive mode to read the ACK, etc.

Cheers

---

From: "Fossati, Thomas (Nokia - GB/Cambridge)" <[email-masked]>
Date: Sun, 4 Feb 2018 19:14:23 +0000

On 04/02/2018, 18:51, "Stephen Farrell" <[email-masked]> wrote:
> matter how often someone says they "care deeply";-)

In user agents I guess this would be handled as a configurable user's
preference, not a as hard-coded value.  We could have it MTI and on by
default.  Software vendors who produce user facing technology have a
strong incentive to do this kind of things right.

Cheers!

---

From: Stephen Farrell <[email-masked]>
Date: Sun, 4 Feb 2018 19:23:21 +0000

Hiya,

On 04/02/18 19:14, Fossati, Thomas (Nokia - GB/Cambridge) wrote:
>> In addition, I'd like to point out a flaw in the above - the
>> question is posed in terms of an "end user client" making decisions
>> for "end users not particularly concerned" with a privacy enhancing
>> feature. I don't believe that programmers can really determine such
>> things (no matter how often someone says they "care deeply";-)
> 
> In user agents I guess this would be handled as a configurable
> user's preference, not a as hard-coded value.  We could have it MTI
> and on by default.  Software vendors who produce user facing
> technology have a strong incentive to do this kind of things right.

Hmm, it's a bit of an aside, and likely not decidable, but I'm not
sure you're right that implementers (whether vendors or open-source
implementers or whatever) are always driven by end-user's concerns.

And in this case, I'm not sure that an application-by-application
approach even works - even if N-1 of the N applications I use do a
great job wrt linkability, when the N-th application is careless I
can lose all benefits. (And yes, combinations of TCP and DHCP can
result in the same bad outcome, but that's not a reason to do less
well than we can IMO.)

So I still hope the WG study the trade-offs and make doing the best
we can for privacy the only option.

S.

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 4 Feb 2018 19:25:50 +0100

On Sun, Feb 04, 2018 at 07:40:41AM -1000, Christian Huitema wrote:
> Negotiate privacy off in the name of network performance? Very slippery
> slope! For example, the connections that do require privacy would stick out.

Well, I personally have stopped fighting on this subject. I've long been
bored by the obsession of this so-called "privacy" which justifies any
possible protocol degradation and loss of efficiency to hide I-don't-
know-what, even sometimes destroying basic security at the same time.

Today, people who want to hide their activities use VPNs. Those of us
having had to help diagnose network issues on IPSEC connections know
that you can't debug anything and that the only fix is to try to randomly
replace elements and settings on the path, and to ultimately get rid of
this protocol in favor of other solutions with less moving parts.

Yes I too am a bit worried about what QUIC will become. The lack of
observability may even result in the impossibility at all to build a
working firewall to protect the users. Thus instead of nasty people
just counting packets on the wire, they'll have an open path to remotely
install malware into their victim's devices without any possibility
for them to be protected, leaking all their data and history even
more easily than today.

The Internet infrastructure is robust and fragile at the same time,
it's robust against attacks and hardware failures, but it's fragile
in front of design mistakes, especially if they're deployed on each
and every browser and/or web server. I think it's important to take
care of it and to ensure it always remains possible to heal it and
to enhance it.

I'd rather see some choices offered to the end user in fact. Just like
right now some decide to use a VPN, some may decide to enable some
super-privacy features that possibly make their connections a bit less
reliable, less performant, and prevent them from using an efficient
firewall, but it will be their choice at least. And it would allow
us to go even further in the privacy domain, possibly contemplating
options making it even harder to fix protocol issues but protect users
better (eg: the spin bit was a tradeoff to satisfy everyone, with an
opt-in solution it's not even needed anymore).

Having a bit indicate if the sequence number is encrypted or not could
be enough :
  - the sender decides to use clear sequence numbers and clears the bit
  - the sender decides to use encrypted sequence numbers and sets the bit.
  - the receiver knows how to decode the sequence number based on this bit.
  - a middle box could not achieve anything by clearing this bit since it
    would mean that the resulting values would appear as almost random and
    would not be usable by the other end.

Just my two cents,
Willy

---

From: Praveen Balasubramanian <[email-masked]>
Date: Sun, 4 Feb 2018 19:31:14 +0000

>From the charter "This work will ensure that QUIC has security and privacy properties that are at least as good as a stack composed of TLS 1.3  using TCP (or MPTCP when using multipath)."

Is the discussion mainly focused on connection migration which is not present in TCP? There seem to be bigger privacy problems due to DNS and TLS SNI extension so in real deployments will packet number encryption add value other than preventing ossification? 

Is a cryptographically secure random initial packet number and random (forward) jumps in packet number not good enough to prevent ossification? The missing packet number range due to the forward jump could be communicated in the encrypted payload so the receiver knows they are not missing.

Thanks

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Willy Tarreau
Sent: Sunday, February 4, 2018 10:26 AM
To: huitema <[email-masked]>
Cc: Gorry Fairhust <[email-masked]>; Eric Rescorla <[email-masked]>; Brian Trammell <[email-masked]>; Lubashev, Igor <[email-masked]>; Roberto Peon <[email-masked]>; Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]>; Roni Even (A) <[email-masked]>; Mirja Kühlewind <[email-masked]>; Jana Iyengar <[email-masked]>; Eggert, Lars <[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>; Piotr Galecki <[email-masked]>
Subject: Re: Packet number encryption

On Sun, Feb 04, 2018 at 07:40:41AM -1000, Christian Huitema wrote:
> Negotiate privacy off in the name of network performance? Very 
> slippery slope! For example, the connections that do require privacy would stick out.

Well, I personally have stopped fighting on this subject. I've long been bored by the obsession of this so-called "privacy" which justifies any possible protocol degradation and loss of efficiency to hide I-don't- know-what, even sometimes destroying basic security at the same time.

Today, people who want to hide their activities use VPNs. Those of us having had to help diagnose network issues on IPSEC connections know that you can't debug anything and that the only fix is to try to randomly replace elements and settings on the path, and to ultimately get rid of this protocol in favor of other solutions with less moving parts.

Yes I too am a bit worried about what QUIC will become. The lack of observability may even result in the impossibility at all to build a working firewall to protect the users. Thus instead of nasty people just counting packets on the wire, they'll have an open path to remotely install malware into their victim's devices without any possibility for them to be protected, leaking all their data and history even more easily than today.

The Internet infrastructure is robust and fragile at the same time, it's robust against attacks and hardware failures, but it's fragile in front of design mistakes, especially if they're deployed on each and every browser and/or web server. I think it's important to take care of it and to ensure it always remains possible to heal it and to enhance it.

I'd rather see some choices offered to the end user in fact. Just like right now some decide to use a VPN, some may decide to enable some super-privacy features that possibly make their connections a bit less reliable, less performant, and prevent them from using an efficient firewall, but it will be their choice at least. And it would allow us to go even further in the privacy domain, possibly contemplating options making it even harder to fix protocol issues but protect users better (eg: the spin bit was a tradeoff to satisfy everyone, with an opt-in solution it's not even needed anymore).

Having a bit indicate if the sequence number is encrypted or not could be enough :

Just my two cents,
Willy

---

From: Christian Huitema <[email-masked]>
Date: Sun, 4 Feb 2018 09:46:11 -1000

> On Feb 4, 2018, at 9:31 AM, Praveen Balasubramanian <[email-masked]> wrote:
> 

Yes, we need to fix the SNI issue. Clear text ALPN too. But should not prevent us from fixing everything else.

> Is a cryptographically secure random initial packet number and random (forward) jumps in packet number not good enough to prevent ossification? The missing packet number range due to the forward ju…

No, random jumps are not enough. It turns out that during migration packets are often sent simultaneously on both paths. The pattern of increases and holes is enough to correlate the two paths.

In practice there are two possible designs: packet number encryption as currently documented; or, independent sequence number space and encryption context for each path. Both work. The one we have is significantly easier to implement. The other exposes the sequence number, which may ease network monitoring but also could drive ossification.

Now is a good time to have this discussion.

-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Sun, 4 Feb 2018 19:55:52 +0000

Ok this answers my question that this is targeted at the migration scenario. 

If jumps are random and cryptographically seeded I don't understand why there would be a pattern. We already use this type of mechanism for TCP ISN and IP header Identification fields. The adversary could also use the side channel timing attack of just observing the sending and receiving application rate for correlation between the two paths (after factoring in the RTT). 

If independent sequence space (with randomization) can work just as well I'd rather pick that over encryption for efficiency reasons. It should solve ossification but I am not a privacy expert.

-----Original Message-----
From: Christian Huitema [mailto:[email-masked]] 
Sent: Sunday, February 4, 2018 11:46 AM
To: Praveen Balasubramanian <[email-masked]>
Cc: [email-masked]; Gorry Fairhust <[email-masked]>; Eric Rescorla <[email-masked]>; Brian Trammell <[email-masked]>; Lubashev, Igor <[email-masked]>; Roberto Peon <[email-masked]>; Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]>; Roni Even (A) <[email-masked]>; Mirja Kühlewind <[email-masked]>; Jana Iyengar <[email-masked]>; Eggert, Lars <[email-masked]>; Martin Thomson <[email-masked]>; Piotr Galecki <[email-masked]>
Subject: Re: Packet number encryption

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sun, 4 Feb 2018 12:31:55 -0800

One aspect of randomised gaps that might be overlooked is that packet
numbers are only sent with the lower 4 bytes at most. This means that a
randomised gap on a new connection ID is not different from a new packet
number that starts at a random offset because you cannot tell if it rolled
over without knowing more about the connection.

However, if migration does not look like a new connection, at least a
0-RTT, it is all a mood point because the sudden appearance of data on a
new connection ID combined with the silence of another ID, and probably
also characteristic changes in packet delays and sizes during migration,
along with GeoIP, are all fingerprinting the connection.

On monitoring packets inside a network, there are probably many new
unexplored possibilities to track reordering of encrypted packet numbers.
For example, one could create a bloom filter to record the packet numbers
seen in a block of N packets, for example syncing wherever the lower packet
number bits are all 0, or something. Packet loss would show up as fewer
matches when comparing filters at different nodes. Or the lower 8 bits of
each packet number in a block could be concatenated to a string and
compared with levenshtein edit distance - even just the lower two bits of
each packet might suffice. The fact the packet numbers are random could be
used pro-actively to simplify hashing.

While the above has some statistical uncertainties, and they do require a
bit more data and time, they also possibly look at longer sequences and
thereby give an overall better quality of understanding than just looking
for out of sequence numbers. After developing good hashes that somehow
encode ordering relations, a neural network might be trained to identify
abnormal traffic patterns efficiently.

Mikkel

On 4 February 2018 at 20.55.59, Praveen Balasubramanian ([email-masked])
wrote:

Ok this answers my question that this is targeted at the migration
scenario.

If jumps are random and cryptographically seeded I don't understand why
there would be a pattern. We already use this type of mechanism for TCP ISN
and IP header Identification fields. The adversary could also use the side
channel timing attack of just observing the sending and receiving
application rate for correlation between the two paths (after factoring in
the RTT).

If independent sequence space (with randomization) can work just as well
I'd rather pick that over encryption for efficiency reasons. It should
solve ossification but I am not a privacy expert.

-----Original Message-----
From: Christian Huitema [mailto:[email-masked]]
Sent: Sunday, February 4, 2018 11:46 AM
To: Praveen Balasubramanian <[email-masked]>
Cc: [email-masked]; Gorry Fairhust <[email-masked]>; Eric Rescorla <
[email-masked]>; Brian Trammell <[email-masked]>; Lubashev, Igor <
[email-masked]>; Roberto Peon <[email-masked]>; Fossati, Thomas (Nokia -
GB/Cambridge, UK) <[email-masked]>; Roni Even (A) <
[email-masked]>; Mirja Kühlewind <[email-masked]>;
Jana Iyengar <[email-masked]>; Eggert, Lars <[email-masked]>; Martin
Thomson <[email-masked]>; Piotr Galecki <
[email-masked]>
Subject: Re: Packet number encryption

> From the charter "This work will ensure that QUIC has security and
privacy properties that are at least as good as a stack composed of TLS 1.3
using TCP (or MPTCP when using multipath)."
>
> Is the discussion mainly focused on connection migration which is not
present in TCP? There seem to be bigger privacy problems due to DNS and TLS
SNI extension so in real deployments will packet number encryption add
value other than preventing ossification?

Yes, we need to fix the SNI issue. Clear text ALPN too. But should not
prevent us from fixing everything else.

>
> Is a cryptographically secure random initial packet number and random
(forward) jumps in packet number not good enough to prevent ossification?
The missing packet number range due to the forward jump could be
communicated in the encrypted payload so the receiver knows they are not
missing.

No, random jumps are not enough. It turns out that during migration packets
are often sent simultaneously on both paths. The pattern of increases and
holes is enough to correlate the two paths.

In practice there are two possible designs: packet number encryption as
currently documented; or, independent sequence number space and encryption
context for each path. Both work. The one we have is significantly easier
to implement. The other exposes the sequence number, which may ease network
monitoring but also could drive ossification.

-- Christian Huitema

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Sun, 4 Feb 2018 21:52:01 +0100

> On 4 Feb 2018, at 18:40, Christian Huitema <[email-masked]> wrote:
> 
> 
> 
> On Feb 4, 2018, at 7:18 AM, Lubashev, Igor <[email-masked]> wrote:...
>> 

Let me say this again, more directly: the IPv6 flow ID will absolutely not work for this. It's used, when it's used, for ECMP entropy and other end-of-the-path routing things. This is equivalent to the problems with ECN on older hardware / configurations using TOS for ECMP entropy: reassign these

Measurement bits aren't flow identifiers. Overloading the latter to become the former because we don't want to deal with the protocol stack we've inherited is not a serious option.

If you've got data to the contrary, please share it. But saying this will work is an exceptional claim, and requires exceptional evidence.

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Sun, 4 Feb 2018 21:53:28 +0100

> On 4 Feb 2018, at 21:52, Brian Trammell (IETF) <[email-masked]> wrote:
> 
>> 

... and things will break in unexpected and hard to debug ways.

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Sun, 4 Feb 2018 22:31:17 +0100

hi Jana,

> On 3 Feb 2018, at 03:30, Jana Iyengar <[email-masked]> wrote:
> 

After having worked my way through the PR again (and essentially rewriting it from the receiver's point of view, which I believe would be useful text to have in the final document, should we end up doing this), I buy this.

> Second, this simplifies and increases robustness of implementation. Avoiding random PN jumps, as Kazuho points out, makes special-casing of code unnecessary. Special code that is exercised occasion…

Agree completely. Packet number jumps are a bad idea, and there seems to be a (not explicitly examined) consensus that attempting to make linkability difficult is worth the effort.

> Third, yes, ossification is a real concern. A simple example: using increasing packet numbers as a signature for detecting QUIC. Even if you argue against this example, there are innovative ways in which ossification will happen. This is based on a true story: We had no idea how GQUIC's flags field could get ossified, the value that was being used commonly became used as a signature for QUIC traffic (see Section 7.5 in the SIGCOMM QUIC paper).

There's an overgeneralization of experience from ossification of single static fields (as in the TLS greasing and GUIC flags cases) to functions over the time series of these fields which has not yet been shown to be valid IMO in the case of integrity-protected wire images -- will expand on this in my next message, replying to Roberto.

> There's a win here in terms of implementation complexity and several implementers have said so.

Compared to packet number jumps, absolutely. Compared to the state two revs of the draft ago, it's a *little* more complex but at least it doesn't add completely new operations at the sender or the receiver.

> There's a win in terms of ossification and our experience says so.

That experience is IMO overgeneralized.

> There's a potential loss of manageability, in being able to detect reordering.

Well, upstream loss and upstream reordering.

As Christian suggested at the start of this thread, if we simultaneously add back a signal that would allow on-path loss and reordering detection, that would make this a usable-information-content-neutral change to the wire image, and everyone could leave the table happy. The only problem with this is that the most obvious place to put this signal -- the four bits we get back from the short header type field (assuming we add the spin bit) aren't really enough to track the loss and reordering patterns we care about with a naive counter.

There are lots of other options in this space (e.g. decide we care about latency more than reordering, and that we don't care about loss at all, and use a two-bit spin to protect the latency signal from reordering, and so on), but that seems like a third thread we should start.

Cheers,

Brian

> This is the trade-off, and I am still in favor of encrypting packet numbers.
> 
> On Fri, Feb 2, 2018 at 7:32 AM, Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]> wrote:

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Sun, 4 Feb 2018 22:31:26 +0100

hi Roberto, all,

(trimming CC, forking thread, since this *really* is separate from the question of packet number encryption.)

> On 3 Feb 2018, at 19:37, Roberto Peon <[email-masked]> wrote:
> 
> Ossification:

This becomes more theoretical when you apply observations about TCP to QUIC:

(1) Sequence numbering in TCP is far more broken than packet numbering (encrypted or not) ever could be, simply because it's not end-to-end integrity protected. The first deployed sequence number meddling boxes, IIRC, were designed to randomize ISNs (i.e., rewrite them), since stacks that did so were slow to deploy. PN rewriting is not only useless in QUIC, it'll break the connection.

(2) Many TCP stacks actually had (and some have) crap performance under certain reordering patterns, such that the deployment of a middlebox that did "fix" reordered packets would have had measurable benefits when tested against the stacks prevalent on the networks on which they were deployed at the time they were deployed. A QUIC PN-reordering middlebox would not show any such benefit, and would be correspondingly difficult to sell.

(3) The time to "path interference decay" for the TCP sequence number was years. Yes, things move more quickly now (indeed, as evidenced by experience with GUIC), but unless we intend not to deploy QUIC version 2 rapidly on the heels of QUIC version 1, I would submit that there's still time to revise everything not in the invariants. (And, indeed, if we do *not* intend to deploy QUIC version 2 rapidly on the heels of QUIC version 1, i.e., not until the "path interference decay" time, then this entire exercise is academic and we can all go home, because the version negotiation mechanism will ossify, greasing or not.)

I think we need to take a step back and take a more systematic view toward ossification prevention.

Greasing, in the TLS sense, refers to not allowing extensibility codepoints to have a single value or set of values that can be used for traffic classification and/or bad traffic rejection by a naive device on the path. This is useful when you're trying to keep parts of the codepoint space available for future use, but the design of QUIC is such that the set of codepoints *in the wire image* is completely determined by the negotiated version; i.e., adding a new packet type to the QUIC long header would require a new version. Therefore, protecting the version negotiation mechanism itself is necessary and sufficient to prevent the wire image from ossifying.

This points the way to an alternate approach which will free us from having to have the ossification discussion about every single bit in the Version 1 header. Early and meaningful exercise of the version negotiation mechanism, with at least two versions whose wire images have *only* our intended invariants in common, is far superior to greasing the version codepoints as a method to protect the version negotiation mechanism. Greasing is a way to "fake" agility when you don't have any real agility yet -- but we're designing a protocol with real agility, with a version negotiation mechanism based at least in its core operation on GUIC's mechanism -- i.e., on one which we know at least one deployer of the protocol has used successfully multiple times.

Greasing is still useful for fields in the invariants that have static values for codepoints we might want to use in the future -- but the greasing space should cover the whole codepoint space in these cases, since post VN we know which values are grease.

(The PN encryption question is separate. The fundamental question here is "should the path see the PN?"... before Melbourne, the answer was "it doesn't matter because we haven't figured out a scalable way to do that". Now the answer is "probably not the whole PN, but the bottom N bits seem quite useful for maintaining TCP-equivalent measurability for the protocol, with a linear linkability risk / measurement utility tradeoff depending on N." Those N bits ossify only if Version 1 is the only QUIC version for the foreseeable future, or if we put them at the same observable offset in the Version 2 header.)

Cheers,

Brian

---

From: Praveen Balasubramanian <[email-masked]>
Date: Sun, 4 Feb 2018 22:52:48 +0000

There seems to be a 1 to 1.25% overhead due to the additional encryption/decryption operation based on a quick microbenchmark experiment. This number will vary for each implementation / system config but seems to be non-negligible. Let's weigh this against the implementation complexities of dealing with packet number jumps. 

Thanks

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Brian Trammell (IETF)
Sent: Sunday, February 4, 2018 1:31 PM
To: Jana Iyengar <[email-masked]>
Cc: Gorry Fairhurst <[email-masked]>; Eric Rescorla <[email-masked]>; Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]>; Mirja Kühlewind <[email-masked]>; Eggert, Lars <[email-masked]>; huitema <[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson <[email-masked]>
Subject: Re: Packet number encryption

hi Jana,

Cheers,

Brian

> This doesn't sound great.  It's a bit like saying that adding the 
> Higgs mechanism to the standard model's lagrangian doesn't make it 
> much more complicate :-)  (*)
> 
> implementation complexity, it's also the energy cost per packet that 
> is quite crucial.  I haven't done the math but ISTM that bringing in 
> ecosystem creates an artificial gap which then will need a middlebox 
> to bridge.  (Sure, we have specified the behaviour of a similar box 
> already, but it'd be really better if the extra translation logic 
> could be avoided in the first place.)
> 
> that can't be negotiated at handshake time reduces our ability to 
> later profile QUIC for the IoT, which would be a bit unlucky.
>

---

From: Roberto Peon <[email-masked]>
Date: Mon, 5 Feb 2018 00:37:53 +0000

A network could carry whatever protocol it wished to, i.e. by wrapping any/all packets on ingress to the network, and then un-wrapping when forwarding to another network. This would allow a network to record and track whatever it wanted to without placing constraints on layers which should be above it.

If the network placed a sequence number on incoming packets, it could detect reordering across all flows. If it do so on a 5-tuple basis, it could detect reordering/duplication/loss across any 5-tuple.

-=R

________________________________
From: Roni Even (A) <[email-masked]>
Sent: Saturday, February 3, 2018 11:47:06 PM
To: Roberto Peon; Piotr Galecki; Jana Iyengar; Fossati, Thomas (Nokia - GB/Cambridge, UK)
Cc: Gorry Fairhust; Eric Rescorla; Mirja Kühlewind; Eggert, Lars; Brian Trammell; Christian Huitema; QUIC WG; Martin Thomson
Subject: RE: Packet number encryption

Hi,

Network monitoring:

Roni

Ossification:

Network monitoring:

Debugging:

-=R

---

From: Roberto Peon <[email-masked]>
Date: Mon, 5 Feb 2018 00:41:11 +0000

Placing/consuming unnecessary bits into the protocol also may reduce bandwidth, even on LANs/intranets.

Optional security tends to devolve to non-secure.

-=R

________________________________
From: Lubashev, Igor <[email-masked]>
Sent: Sunday, February 4, 2018 9:18:13 AM
To: Roni Even (A); Roberto Peon; Piotr Galecki; Jana Iyengar; Fossati, Thomas (Nokia - GB/Cambridge, UK)
Cc: Gorry Fairhust; Eric Rescorla; Christian Huitema; Eggert, Lars; Brian Trammell; Mirja Kühlewind; QUIC WG; Martin Thomson
Subject: RE: Packet number encryption

- Igor

Hi,

Network monitoring:

Roni

Ossification:

Network monitoring:

Debugging:

-=R

---

From: "Salz, Rich" <[email-masked]>
Date: Mon, 5 Feb 2018 03:51:21 +0000

> Optional security tends to devolve to non-secure.

That’s a great aphorism.  And sadly all too true.

---

From: Jana Iyengar <[email-masked]>
Date: Sun, 4 Feb 2018 20:34:37 -0800

Privacy-protection can't be a user choice, for the reasons others have
noted on this thread.

That said, my primary argument is for encryption to avoid ossification. Not
that it matters now, but I'll note that much of GQUIC's original motivation
for encrypting headers was to avoid ossification.

I'll reiterate that fields we expose will get ossified and there are
long-term ecosystem effects to this. Let me illustrate this with precisely
the packet number field. Middleboxes commonly assume that a TCP flow can
only handle packets in-order. This assumption comes from the fact that TCP
implementations get poor performance when packets are reordered. This was
definitely true for implementations of TCP, but that is TCP's problem, not
the network's. However, almost all load-balancers I know of now will pin
all packets within a TCP flow to one path, leading to sub-optimal
performance in the network, and destroying incentives for the endpoints to
deploy reordering-resilient TCP implementations (even though there are
plenty of ways of doing this.)

Exposing QUIC's packet number field (as any field) is likely to have
similar consequences and a similar ecosystem arc.

---

From: "Roni Even" <[email-masked]>
Date: Mon, 5 Feb 2018 06:58:17 +0200

Does not it mean that the network need to identify packets  that are from
the same quic connection to make such wrapping?

Roni

From: QUIC [mailto:[email-masked]] On Behalf Of Roberto Peon
Sent: Monday, February 05, 2018 2:38 AM
To: Roni Even (A); Piotr Galecki; Jana Iyengar; Fossati, Thomas (Nokia -
GB/Cambridge, UK)
Cc: Gorry Fairhust; Eric Rescorla; Christian Huitema; Eggert, Lars; Brian
Trammell; Mirja Kühlewind; QUIC WG; Martin Thomson
Subject: Re: Packet number encryption

A network could carry whatever protocol it wished to, i.e. by wrapping
any/all packets on ingress to the network, and then un-wrapping when
forwarding to another network. This would allow a network to record and
track whatever it wanted to without placing constraints on layers which
should be above it. 

If the network placed a sequence number on incoming packets, it could detect
reordering across all flows. If it do so on a 5-tuple basis, it could detect
reordering/duplication/loss across any 5-tuple.

-=R

  _____  

From: Roni Even (A) <[email-masked]>
Sent: Saturday, February 3, 2018 11:47:06 PM
To: Roberto Peon; Piotr Galecki; Jana Iyengar; Fossati, Thomas (Nokia -
GB/Cambridge, UK)
Cc: Gorry Fairhust; Eric Rescorla; Mirja Kühlewind; Eggert, Lars; Brian
Trammell; Christian Huitema; QUIC WG; Martin Thomson
Subject: RE: Packet number encryption 

Hi,

Network monitoring:

Networks may always wrap the packet(s) they have received with whatever data
they wish, and unwrap before forwarding. This can allow any network to
understand ordering,  loss, etc within its boundaries.

Roni

Cc: Gorry Fairhust; Eric Rescorla; Mirja Kühlewind; Eggert, Lars; Brian

Ossification:

 This is the kind of non-theoretical ossification that drove the creation of
QUIC in the first place.

Network monitoring:

Networks may always wrap the packet(s) they have received with whatever data
they wish, and unwrap before forwarding. This can allow any network to
understand ordering,  loss, etc within its boundaries.

Debugging:

Debugging of flows can still happen for flows that wish to be debugged-- the
QUIC connection's key material(s) can be shared manually and used by the
debugging tool. This is how one debugs http2, https, etc. already today.

-=R

-------- Original message --------

Cc: Gorry Fairhust <[email-masked]>, Eric Rescorla <[email-masked]>,
Christian Huitema <[email-masked]>, "Eggert, Lars" <[email-masked]>,
Brian Trammell <[email-masked]>, Mirja Kühlewind
<[email-masked]>, QUIC WG <[email-masked]>, Martin Thomson
<[email-masked]> 

What is the group’s proposal for allowing network monitoring tools to detect
packet retransmission or packet reordering ?

 

From: QUIC [mailto:[email-masked]] On Behalf Of Jana Iyengar
Sent: Friday, February 02, 2018 9:31 PM
To: Fossati, Thomas (Nokia - GB/Cambridge, UK) <[email-masked]>
Cc: Gorry Fairhust <[email-masked]>; Eric Rescorla <[email-masked]>;
Mirja Kühlewind <[email-masked]>; Christian Huitema
<[email-masked]>; Brian Trammell <[email-masked]>; Eggert, Lars
<[email-masked]>; QUIC WG <[email-masked]>; Martin Thomson
<[email-masked]>
Subject: Re: Packet number encryption

Second, this simplifies and increases robustness of implementation. Avoiding
random PN jumps, as Kazuho points out, makes special-casing of code
unnecessary. Special code that is exercised occasionally is a strong bug
attractor, and I would strongly argue for as few of those as possible in
implementation.

Third, yes, ossification is a real concern. A simple example: using
increasing packet numbers as a signature for detecting QUIC. Even if you
argue against this example, there are innovative ways in which ossification
will happen. This is based on a true story: We had no idea how GQUIC's flags
field could get ossified, the value that was being used commonly became used
as a signature for QUIC traffic (see Section 7.5 in the SIGCOMM QUIC paper
<https://urldefense.proofpoint.com/v2/url?u=https-3A__static.googleuserconte
nt.com_media_research.google.com_en_pubs_archive_46403.pdf&d=DwMGaQ&c=5VD0RT
tNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=SY87I4v_IqTCGOds-j5NCOwkQ0cJcYHG
yKdXmMIVn6I&s=U9o6dI-dwzKwvXHvvmv7WtVm84puvxhaP0cN2of3VvA&e=> ).

There's a win here in terms of implementation complexity and several
implementers have said so. There's a win in terms of ossification and our
experience says so. There's a potential loss of manageability, in being able
to detect reordering. This is the trade-off, and I am still in favor of
encrypting packet numbers.

(*)
https://www.symmetrymagazine.org/article/the-deconstructed-standard-model-eq
uation
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.symmetrymagazine.o
rg_article_the-2Ddeconstructed-2Dstandard-2Dmodel-2Dequation&d=DwMGaQ&c=5VD0
RTtNlTh3ycd41b3MUw&r=C0sUo-LFNBaYfyoaCsf6TA&m=SY87I4v_IqTCGOds-j5NCOwkQ0cJcY
HGyKdXmMIVn6I&s=etIO0AUHiJm8zucsWaTgo-XULFisCMoG2iDvwC69s9Q&e=>

---

From: Christian Huitema <[email-masked]>
Date: Sun, 4 Feb 2018 20:14:23 -1000

On 2/4/2018 6:58 PM, Roni Even wrote:

> Does not it mean that the network need to identify packets  that are
> from the same quic connection to make such wrapping?
>

Not really. The network can take any slice of the input packets,
assigned them a sequence number, and verify that the slice comes out the
same way it went in. Or count losses, reordering, etc. Slicing by five
tuple should work just fine.

-- Christian Huitema

---

From: "Roni Even (A)" <[email-masked]>
Date: Mon, 5 Feb 2018 06:24:03 +0000

BTW:
This will only provide information about reorder in this network segment and not end to end which is what the requirement is for.
Roni

From: QUIC [mailto:[email-masked]] On Behalf Of Christian Huitema
Sent: Monday, February 05, 2018 8:14 AM
To: [email-masked]
Subject: Re: Packet number encryption

On 2/4/2018 6:58 PM, Roni Even wrote:
Does not it mean that the network need to identify packets  that are from the same quic connection to make such wrapping?

Not really. The network can take any slice of the input packets, assigned them a sequence number, and verify that the slice comes out the same way it went in. Or count losses, reordering, etc. Slicing by five tuple should work just fine.

-- Christian Huitema

---

From: Roberto Peon <[email-masked]>
Date: Mon, 5 Feb 2018 07:16:57 +0000

That depends.
Arguably, all the network can know (or should know) is on the basis of a 5-tuple, thus the network could assign sequence numbers based on order of arrival of a 5-tuple.

However, one could do more useful and/or general things: If the intended egress node is known at the ingress point for a packet then, when applying the appropriate route, stamp the packet with a sequence number on/for that route. At that point you’re able to look at reordering across any/all flows within a route. This is a stronger ordering guarantee than is necessary, and thus is sufficient for any individual flow on that route. In my belief, this would be a superior metric for network health detection overall.

-=R

From: Roni Even <[email-masked]>
Date: Sunday, February 4, 2018 at 8:58 PM
To: Roberto Peon <[email-masked]>, "'Roni Even (A)'" <[email-masked]>, 'Piotr Galecki' <[email-masked]>, 'Jana Iyengar' <[email-masked]>, "'Fossati, Thomas (Nokia - GB/Cambridge, UK)'" <[email-masked]>
Cc: 'Gorry Fairhust' <[email-masked]>, 'Eric Rescorla' <[email-masked]>, 'Christian Huitema' <[email-masked]>, "'Eggert, Lars'" <[email-masked]>, 'Brian Trammell' <[email-masked]>, 'Mirja Kühlewind' <[email-masked]>, 'QUIC WG' <[email-masked]>, 'Martin Thomson' <[email-masked]>
Subject: RE: Packet number encryption

To: Roni Even (A); Piotr Galecki; Jana Iyengar; Fossati, Thomas (Nokia - GB/Cambridge, UK)

-=R

Hi,

Network monitoring:

Roni

Ossification:

Network monitoring:

Debugging:

-=R

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 05 Feb 2018 08:36:40 +0000

On 04/02/2018, 21:31, Brian Trammell (IETF) wrote:
>> On 3 Feb 2018, at 19:37, Roberto Peon<[email-masked]>  wrote:
>>
>> Ossification:
+1 Experience with TCP doesn’t; necessarily help people understand what 
will happen with QUIC. Because TCP exposes all its protocol information, 
a middlebox can do a lot more than observe the sequence number, it can 
actually insert itself as a proxy for protocol mechanisms. This is not 
possible with QUIC.

> (2) Many TCP stacks actually had (and some have) crap performance under certain reordering patterns, such that the deployment of a middlebox that did "fix" reordered packets would have had measurab…
It would not be fair to say that all boxes that observe TCP sequence 
numbers also interpret TCP options (or any other field used by the 
protocol). My point is that we shouldn’t now “fear” everything, and then 
suspect that evil may come form this.
> (The PN encryption question is separate. The fundamental question here is "should the path see the PN?"... before Melbourne, the answer was "it doesn't matter because we haven't figured out a scala…
I think it is entirely possible that a small set of fields in the 
transport header may be usefully ossified to increase the deployability, 
manageability and flexibility across a range of use cases.

Gorry

---

From: Brian Trammell (IETF) <[email-masked]>
Date: Mon, 5 Feb 2018 09:38:45 +0100

hi Roberto, all,

(Trimming recipients)

Briefly: this is the approach under development with the IOAM work in the IPPM (and other) working groups; please come by if you‘d like to contribute there.

There are two serious drawbacks of an IOAM-only approach to measurability of encrypted transports.

First, they only allow measurement on the granularity of the tunnels carrying IOAM traffic. As you point out, sometimes that is exactly what you want. Often it isn’t. Requiring an operator to tunnel traffic differently in order to be able to get the measurement aggregates necessary for a given measurement task runs into a Heisenberg problem: you’ve changed the thing you want to measure in trying to measure it.

Second, they only measure the treatment of a traffic flow between tunnel endpoints. This implies that they're good for intranet diagnostics (of aggregates you generally already have) or for diagnostics for tunnel services provided to customers (assuming you ship IOAM-capable CPE for the customer end; this is indeed one of the reasons to standardize this approach). They're bad for diagnosing stuff within an end network (between the tunnel end and the endpoint) or for anything interdomain.

IOAM approaches are an important part of the measurement toolkit, but they're not a replacement for end-to-end signals for all tasks.

Cheers,

Brian

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 5 Feb 2018 22:35:31 +0900

2018-02-05 6:31 GMT+09:00 Brian Trammell (IETF) <[email-masked]>:
> As Christian suggested at the start of this thread, if we simultaneously add back a signal that would allow on-path loss and reordering detection, that would make this a usable-information-content-neutral change to the wire image, and everyone could leave the table happy.

+1

There are two parties with different interests. Endpoint developers
want something more straightforward to implement and also is safe from
ossification. Operators want a tool for diagnosis.

In TCP, both parties have used the same signal for their tasks.

In QUIC, we can split the signal into two, which can be tailored for
each use-case.

The split would help the protocol evolve (as fit from endpoint
developers' perspective). At the same time, it could also help
operators, since the signal they use can become more stable or evolve
in a different pace (since the changes to the encrypted part of the
protocol will not affect what the operators will see, or vice-versa).

Note also that even in the current unencrypted form, packet numbers of
QUIC is not as easy to use compared to TCP (from operators' viewpoint)
since the numbers monotonically increase and since ACKs are encrypted.
So there's a chance to make the signal more useful by splitting the
signals. Spin bit is a good example.

> The only problem with this is that the most obvious place to put this signal -- the four bits we get back from the short header type field (assuming we add the spin bit) aren't really enough to track the loss and reordering patterns we care about with a naive counter.
>

-- 
Kazuho Oku

---

From: "Roni Even (A)" <[email-masked]>
Date: Mon, 5 Feb 2018 14:28:04 +0000

Hi,

I want to comment on 
" There are two parties with different interests. Endpoint developers want something more straightforward to implement and also is safe from ossification. Operators want a tool for diagnosis."

I agree that there are two parties but I hope that we share the same interest which is to have a secure, reliable protocol that also support manageability in actual transport networks.

As for having parallel mechanisms, one of the points about the usability of spin bit was that it not part of the protocol state machine and that makes it less reliable,

Roni

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Kazuho Oku
Sent: Monday, February 05, 2018 3:36 PM
To: Brian Trammell (IETF)
Cc: Gorry Fairhurst; Eric Rescorla; Christian Huitema; Fossati, Thomas (Nokia - GB/Cambridge, UK); Mirja Kühlewind; Eggert, Lars; Jana Iyengar; QUIC WG; Martin Thomson
Subject: Re: Packet number encryption

+1

There are two parties with different interests. Endpoint developers want something more straightforward to implement and also is safe from ossification. Operators want a tool for diagnosis.

In QUIC, we can split the signal into two, which can be tailored for each use-case.

The split would help the protocol evolve (as fit from endpoint developers' perspective). At the same time, it could also help operators, since the signal they use can become more stable or evolve in a different pace (since the changes to the encrypted part of the protocol will not affect what the operators will see, or vice-versa).

Note also that even in the current unencrypted form, packet numbers of QUIC is not as easy to use compared to TCP (from operators' viewpoint) since the numbers monotonically increase and since ACKs are encrypted.
So there's a chance to make the signal more useful by splitting the signals. Spin bit is a good example.

>> More seriously though, I'd like to point out that it is not just 
>> about implementation complexity, it's also the energy cost per packet 
>> that is quite crucial.  I haven't done the math but ISTM that 
>> bringing in another batch of non-optional crypto computation on a per 
>> packet basis is not going to move the needle in the right direction 
>> for stacks that run on low power (IoT).
>>

--
Kazuho Oku

---

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 5 Feb 2018 16:09:22 +0100

> Am 05.02.2018 um 09:36 schrieb Gorry Fairhurst <[email-masked]>:
> 
> I think it is entirely possible that a small set of fields in the transport header may be usefully ossified to increase the deployability, manageability and flexibility across a range of use cases.

+1 if a field ossifies that is fully specified in one version and indented to be used by the network in a certain way that can be helpful for deployment. We really need to make sure however that middleboxes are version-aware and basically deploying two or multiple versions at once could be the right approach to make sure the version mechanism will be usable in future.

---

From: "Salz, Rich" <[email-masked]>
Date: Mon, 5 Feb 2018 16:39:33 +0000

> I think it is entirely possible that a small set of fields in the transport header may be usefully ossified to increase the deployability, manageability and flexibility across a range of use cases.

Then we should use a word besides ossified, which at this point has entirely negative connotations.  Perhaps “frozen”?

---

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 5 Feb 2018 17:42:52 +0100

If that helps I’m all in for that!

> Am 05.02.2018 um 17:39 schrieb Salz, Rich <[email-masked]>:
>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Mon, 5 Feb 2018 17:24:56 +0000

>> This was definitely true for implementations of TCP, but that is TCP's problem, not the network's
Flow classification happens through the network and also on the end host with RSS where a flow is mapped to a core. This allows for building high performance receive processing that could be lock free for the most part. The only downside of ECMP on the network and RSS on the CPU is that a single flow will not take multiple paths or get processed on more than one core. Windows on server machines today can saturate 25-35 Gbps for a single TCP connection before being CPU limited. This is a  reasonable trade off because I don’t know of cases where a single flow is driving more than that amount of traffic. Assumptions about how flows get classified help make for more efficient processing. They also lead to consistent latency. You do not want the traffic to take a bi-modal or N-modal paths (on the network or the host) to being processed because the fluctuations in latency will hurt the workload. I am not arguing for TCP not being resilient to reordering but IMO that should be the uncommon case not the common case. With QUIC if streams were exposed on the network you could take advantage of stream level ECMP and RSS to scale better than TCP but we have chosen to keep the 4-tuple (or the 5-tuple) as the flow classifier on the network which is ok by me since it leads to TCP parity.

Thanks

From: QUIC [mailto:[email-masked]] On Behalf Of Jana Iyengar
Sent: Sunday, February 4, 2018 8:35 PM
To: Salz, Rich <[email-masked]>
Cc: Lubashev, Igor <[email-masked]>; Roberto Peon <[email-masked]>; QUIC WG <[email-masked]>
Subject: Re: Packet number encryption

Privacy-protection can't be a user choice, for the reasons others have noted on this thread.

That said, my primary argument is for encryption to avoid ossification. Not that it matters now, but I'll note that much of GQUIC's original motivation for encrypting headers was to avoid ossification.

I'll reiterate that fields we expose will get ossified and there are long-term ecosystem effects to this. Let me illustrate this with precisely the packet number field. Middleboxes commonly assume that a TCP flow can only handle packets in-order. This assumption comes from the fact that TCP implementations get poor performance when packets are reordered. This was definitely true for implementations of TCP, but that is TCP's problem, not the network's. However, almost all load-balancers I know of now will pin all packets within a TCP flow to one path, leading to sub-optimal performance in the network, and destroying incentives for the endpoints to deploy reordering-resilient TCP implementations (even though there are plenty of ways of doing this.)

Exposing QUIC's packet number field (as any field) is likely to have similar consequences and a similar ecosystem arc.

---

From: Mike Bishop <[email-masked]>
Date: Mon, 5 Feb 2018 18:59:09 +0000

There are overlapping interests, but they are certainly not the same.  When someone on the network side says "reliable," everyone at the endpoints hears "ossified," because once the network starts relying on a behavior, you can't realistically change it.

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Roni Even (A)
Sent: Monday, February 5, 2018 6:28 AM
To: [email-masked]
Cc: Kazuho Oku <[email-masked]>
Subject: RE: Packet number encryption

Hi,

I want to comment on

Roni

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Kazuho Oku
Sent: Monday, February 05, 2018 3:36 PM
To: Brian Trammell (IETF)
Cc: Gorry Fairhurst; Eric Rescorla; Christian Huitema; Fossati, Thomas (Nokia - GB/Cambridge, UK); Mirja Kühlewind; Eggert, Lars; Jana Iyengar; QUIC WG; Martin Thomson
Subject: Re: Packet number encryption

+1

--
Kazuho Oku

---

From: Mike Bishop <[email-masked]>
Date: Mon, 5 Feb 2018 19:02:06 +0000

How about "invariant"?  😊

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 5 Feb 2018 15:32:27 -0800

Praveen,

That was precisely my point about the ecosystem evolving. Now that we have
the ability to classify flows, various things have been built around being
able to tell TCP flows. This obviously has benefits as you point out, but
the downside is that that I can't get good network utilization with a
single flow. I understand the scaling point, but speaking of common cases,
any server in the wild is usually serving a large number of connections at
moderate speeds, not a single one at 25-35 Gbps... which makes that sort of
scaling less exciting than in a microbenchmark.

FWIW, the other downside of ECMP is that multipath transport doesn't work
-- you need to hash on something else besides the 4-tuple. (We had to work
around this in Google's deployment of connection migration.)

I'll disagree with your point about reordering being the uncommon case.
While that's true today, this is again an expectation that the network
works hard to maintain, though the ecosystem and what we expect as "usual"
would be quite different had we not built TCP's requirements into the
network. There's nothing wrong with having n-modal latencies... we can
engineer around that. Any sensible load distribution scheme within the
network will give you increased variance but it should limit the worst-case
latency. You'd perhaps agree that that's a net win.

This was an example of how exposing packet number or not can have long term
ecosystem effects. My point was about the fact that exposing can cause
ossification.

- jana

On Mon, Feb 5, 2018 at 9:24 AM, Praveen Balasubramanian <[email-masked]

> Flow classification happens through the network and also on the end host
> with RSS where a flow is mapped to a core. This allows for building high
> performance receive processing that could be lock free for the most part.
> The only downside of ECMP on the network and RSS on the CPU is that a
> single flow will not take multiple paths or get processed on more than one
> core. Windows on server machines today can saturate 25-35 Gbps for a single
> TCP connection before being CPU limited. This is a  reasonable trade off
> because I don’t know of cases where a single flow is driving more than that
> amount of traffic. Assumptions about how flows get classified help make for
> more efficient processing. They also lead to consistent latency. You do not
> want the traffic to take a bi-modal or N-modal paths (on the network or the
> host) to being processed because the fluctuations in latency will hurt the
> workload. I am not arguing for TCP not being resilient to reordering but
> IMO that should be the uncommon case not the common case. With QUIC if
> streams were exposed on the network you could take advantage of stream
> level ECMP and RSS to scale better than TCP but we have chosen to keep the
> 4-tuple (or the 5-tuple) as the flow classifier on the network which is ok
> by me since it leads to TCP parity.
>
>
>
> Thanks
>
>
>
> *Sent:* Sunday, February 4, 2018 8:35 PM
> *To:* Salz, Rich <[email-masked]>
> *Cc:* Lubashev, Igor <[email-masked]>; Roberto Peon <[email-masked]>;
> QUIC WG <[email-masked]>
> That said, my primary argument is for encryption to avoid ossification.
> Not that it matters now, but I'll note that much of GQUIC's original
> motivation for encrypting headers was to avoid ossification.
>
>
>

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 5 Feb 2018 16:30:44 -0800

I don't think we're converging here, so I'd like to try and encourage us to
move towards it. I have my opinion, but I care more about forward progress
at this point.

There are three designs so far:
1. Packet numbers as is, with random gaps around migration.
2. Packet numbers encrypted, as per Martin's PR.
3. Packet numbers encrypted, plus a small modulo sequence number somewhere.

The concrete ones here are (1) and (2).  I'm listing (3) but am not
inclined to consider it seriously, since it's not concrete enough, (and I
don't want more bits to be spent in the QUIC packet header if that's where
this info goes.)

There are opinions about ossification, privacy, and manageability. I'm not
hearing bloody murder, so for what it's worth, I'd like to stipulate first
that we can all live with the final design, whichever one it is.  If anyone
disagrees, please say so -- I think it'd be useful to hear if there are
folks who think a particular design is fundamentally problematic.

My opinion is still for (2) over (1), but I can live with either.

On Mon, Feb 5, 2018 at 3:32 PM, Jana Iyengar <[email-masked]> wrote:

> single flow. I understand the scaling point, but speaking of common
> cases, any server in the wild is usually serving a large number of
> connections at moderate speeds, not a single one at 25-35 Gbps... which
> makes that sort of scaling less exciting than in a microbenchmark.
>
> This was an example of how exposing packet number or not can have long
> term ecosystem effects. My point was about the fact that exposing can cause
> ossification.
>
> - jana
>
> On Mon, Feb 5, 2018 at 9:24 AM, Praveen Balasubramanian <
> [email-masked]> wrote:
>

---

From: Martin Thomson <[email-masked]>
Date: Tue, 6 Feb 2018 11:41:11 +1100

It's possible that we could decide to add (3) later if we decided to
take (2) now.

I'd prefer to proceed with (2) and I'm sure that we'll collectively
have more information on what it takes to add bits to the header after
we conclude our discussion on spin bits in London.

There's a fourth option, which involves encryption of just the
high-order bits of the packet number.  It still likely means that we
would need gaps to break linkability, but those gaps could be
relatively small and so could be made unilaterally without all of the
machinery that (1) has.  Of course, that has other implications, such
as the need to expose the size of the packet number (unless we switch
to a little-endian encoding...).

On Tue, Feb 6, 2018 at 11:30 AM, Jana Iyengar <[email-masked]> wrote:
> The concrete ones here are (1) and (2).  I'm listing (3) but am not inclined
> to consider it seriously, since it's not concrete enough, (and I don't want
> more bits to be spent in the QUIC packet header if that's where this info
> goes.)
>
>> the downside is that that I can't get good network utilization with a single
>> flow. I understand the scaling point, but speaking of common cases, any
>> server in the wild is usually serving a large number of connections at
>> While that's true today, this is again an expectation that the network works
>> hard to maintain, though the ecosystem and what we expect as "usual" would
>> be quite different had we not built TCP's requirements into the network.
>> There's nothing wrong with having n-modal latencies... we can engineer
>> around that. Any sensible load distribution scheme within the network will
>> give you increased variance but it should limit the worst-case latency.
>> You'd perhaps agree that that's a net win.
>>
>> On Mon, Feb 5, 2018 at 9:24 AM, Praveen Balasubramanian
>> <[email-masked]> wrote:
>>>
>>> The only downside of ECMP on the network and RSS on the CPU is that a single
>>> flow will not take multiple paths or get processed on more than one core.
>>> Windows on server machines today can saturate 25-35 Gbps for a single TCP
>>> connection before being CPU limited. This is a  reasonable trade off because
>>> I don’t know of cases where a single flow is driving more than that amount
>>> of traffic. Assumptions about how flows get classified help make for more
>>> efficient processing. They also lead to consistent latency. You do not want
>>> the traffic to take a bi-modal or N-modal paths (on the network or the host)
>>> to being processed because the fluctuations in latency will hurt the
>>> workload. I am not arguing for TCP not being resilient to reordering but IMO
>>> that should be the uncommon case not the common case. With QUIC if streams
>>> were exposed on the network you could take advantage of stream level ECMP
>>> and RSS to scale better than TCP but we have chosen to keep the 4-tuple (or
>>> the 5-tuple) as the flow classifier on the network which is ok by me since
>>> it leads to TCP parity.
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> the network's. However, almost all load-balancers I know of now will pin all
>>> packets within a TCP flow to one path, leading to sub-optimal performance in
>>> the network, and destroying incentives for the endpoints to deploy
>>> reordering-resilient TCP implementations (even though there are plenty of
>>> ways of doing this.)
>>>
>>>
>>>

---

From: Ted Hardie <[email-masked]>
Date: Mon, 5 Feb 2018 16:49:47 -0800

On Mon, Feb 5, 2018 at 4:30 PM, Jana Iyengar <[email-masked]> wrote:

> I don't think we're converging here, so I'd like to try and encourage us
> to move towards it. I have my opinion, but I care more about forward
> progress at this point.
>
If you'll forgive my saying so, evidence suggests that so many of the
concrete aspects of such a proposal (where the bits would live, how many
bits, etc.) would change over the lifetime of a proposal in this WG that
any completely nailed down suggestion might as well be a blank sheet for
all the bearing it would have on the final placement and length.

The question is:  do we wish to expose a small sequence number to assist in
network analysis and management, yes or no?

We can argue over the bike shed color after making the decision to build a
shed; requiring the shed have a color before considering the question seems
less productive to me than just answering the question.

Just my personal opinion, of course.

Ted

>> That was precisely my point about the ecosystem evolving. Now that we
>> have the ability to classify flows, various things have been built around
>> being able to tell TCP flows. This obviously has benefits as you point out,
>> but the downside is that that I can't get good network utilization with a

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 6 Feb 2018 03:06:51 +0000

Agree on the point about ossification and long term effects on ecosystem. I was just pointing out that the current flow classification scheme has some benefits too so there are trade-offs.

>> you need to hash on something else besides the 4-tuple. (We had to work around this in Google's deployment of connection migration
I would like to understand the workaround better because I have been struggling with how one can incrementally deploy something like connection migration on current load balancers. But that’s for a separate thread.

From: Jana Iyengar [mailto:[email-masked]]
Sent: Monday, February 5, 2018 3:32 PM
To: Praveen Balasubramanian <[email-masked]>
Cc: Salz, Rich <[email-masked]>; Lubashev, Igor <[email-masked]>; Roberto Peon <[email-masked]>; QUIC WG <[email-masked]>
Subject: Re: Packet number encryption

Praveen,

That was precisely my point about the ecosystem evolving. Now that we have the ability to classify flows, various things have been built around being able to tell TCP flows. This obviously has benefits as you point out, but the downside is that that I can't get good network utilization with a single flow. I understand the scaling point, but speaking of common cases, any server in the wild is usually serving a large number of connections at moderate speeds, not a single one at 25-35 Gbps... which makes that sort of scaling less exciting than in a microbenchmark.

FWIW, the other downside of ECMP is that multipath transport doesn't work -- you need to hash on something else besides the 4-tuple. (We had to work around this in Google's deployment of connection migration.)

I'll disagree with your point about reordering being the uncommon case. While that's true today, this is again an expectation that the network works hard to maintain, though the ecosystem and what we expect as "usual" would be quite different had we not built TCP's requirements into the network. There's nothing wrong with having n-modal latencies... we can engineer around that. Any sensible load distribution scheme within the network will give you increased variance but it should limit the worst-case latency. You'd perhaps agree that that's a net win.

This was an example of how exposing packet number or not can have long term ecosystem effects. My point was about the fact that exposing can cause ossification.

- jana

Thanks

From: QUIC [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Jana Iyengar
Sent: Sunday, February 4, 2018 8:35 PM
To: Salz, Rich <[email-masked]<mailto:[email-masked]>>
Cc: Lubashev, Igor <[email-masked]<mailto:[email-masked]>>; Roberto Peon <[email-masked]<mailto:[email-masked]>>; QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Packet number encryption

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 6 Feb 2018 03:13:16 +0000

My opinion is for 1 over 2 but I can live with either. The main concern is cost of another encryption / decryption step, so the benefit of 2 over 1 should be demonstrably better on one of ossification, privacy, or manageability.

Sent: Monday, February 5, 2018 4:31 PM

I don't think we're converging here, so I'd like to try and encourage us to move towards it. I have my opinion, but I care more about forward progress at this point.

The concrete ones here are (1) and (2).  I'm listing (3) but am not inclined to consider it seriously, since it's not concrete enough, (and I don't want more bits to be spent in the QUIC packet header if that's where this info goes.)

There are opinions about ossification, privacy, and manageability. I'm not hearing bloody murder, so for what it's worth, I'd like to stipulate first that we can all live with the final design, whichever one it is.  If anyone disagrees, please say so -- I think it'd be useful to hear if there are folks who think a particular design is fundamentally problematic.

On Mon, Feb 5, 2018 at 3:32 PM, Jana Iyengar <[email-masked]<mailto:[email-masked]>> wrote:
Praveen,

- jana

Thanks

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 6 Feb 2018 07:10:05 +0000

Oops sorry I think I misread that 1 requires random jumps only around migration and not always, which means its decidedly inferior to 2 w.r.t. ossification. Can we include a 1.1 which incorporates random jumps even in steady state? Or have we ruled that option out?

From: QUIC [mailto:[email-masked]] On Behalf Of Praveen Balasubramanian
Sent: Monday, February 5, 2018 7:13 PM
To: Jana Iyengar <[email-masked]>
Cc: Salz, Rich <[email-masked]>; Lubashev, Igor <[email-masked]>; Roberto Peon <[email-masked]>; QUIC WG <[email-masked]>
Subject: RE: Packet number encryption

From: Jana Iyengar [mailto:[email-masked]]
Sent: Monday, February 5, 2018 4:31 PM
To: Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>
Cc: Salz, Rich <[email-masked]<mailto:[email-masked]>>; Lubashev, Igor <[email-masked]<mailto:[email-masked]>>; Roberto Peon <[email-masked]<mailto:[email-masked]>>; QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Packet number encryption

- jana

Thanks

---

From: Marten Seemann <[email-masked]>
Date: Tue, 06 Feb 2018 07:30:40 +0000

Option 2 is the only option that allows us to correct our decision in the
future. If we were to discover that it doesn't work out as we planned (wrt
encryption performance, network manageability or literally any other
unforeseen problem), going back to option 1 (or coming up with another
clever solution) in a future QUIC version would still be a viable option.
Option 1 on the other hand, once deployed, immediately puts us at risk of
ossification by middleboxes, which will prohibit us from ever switching to
option 2 for a future QUIC version.

On Tue, Feb 6, 2018 at 3:10 PM Praveen Balasubramanian <[email-masked]>
wrote:

> Oops sorry I think I misread that 1 requires random jumps only around
> migration and not always, which means its decidedly inferior to 2 w.r.t.
> ossification. Can we include a 1.1 which incorporates random jumps even in
> steady state? Or have we ruled that option out?
>
>
>
> *From:* QUIC [mailto:[email-masked]] *On Behalf Of *Praveen
> Balasubramanian
> *Sent:* Monday, February 5, 2018 7:13 PM
> *To:* Jana Iyengar <[email-masked]>
>
>
> *Cc:* Salz, Rich <[email-masked]>; Lubashev, Igor <[email-masked]>;
> Roberto Peon <[email-masked]>; QUIC WG <[email-masked]>
>
> My opinion is for 1 over 2 but I can live with either. The main concern is
> cost of another encryption / decryption step, so the benefit of 2 over 1
> should be demonstrably better on one of ossification, privacy, or
> manageability.
>
>
>
> *From:* Jana Iyengar [mailto:[email-masked] <[email-masked]>]
> *Sent:* Monday, February 5, 2018 4:31 PM
> *To:* Praveen Balasubramanian <[email-masked]>
> *Cc:* Salz, Rich <[email-masked]>; Lubashev, Igor <[email-masked]>;
> Roberto Peon <[email-masked]>; QUIC WG <[email-masked]>

---

From: "Roni Even (A)" <[email-masked]>
Date: Tue, 6 Feb 2018 07:34:50 +0000

Hi,
This is what version negotiation is for, you can change it in future version based on version number.
Roni

From: QUIC [mailto:[email-masked]] On Behalf Of Marten Seemann
Sent: Tuesday, February 06, 2018 9:31 AM
To: Praveen Balasubramanian
Cc: Jana Iyengar; Lubashev, Igor; Roberto Peon; QUIC WG; Salz, Rich
Subject: Re: Packet number encryption

Option 2 is the only option that allows us to correct our decision in the future. If we were to discover that it doesn't work out as we planned (wrt encryption performance, network manageability or literally any other unforeseen problem), going back to option 1 (or coming up with another clever solution) in a future QUIC version would still be a viable option.
Option 1 on the other hand, once deployed, immediately puts us at risk of ossification by middleboxes, which will prohibit us from ever switching to option 2 for a future QUIC version.

On Tue, Feb 6, 2018 at 3:10 PM Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>> wrote:
Oops sorry I think I misread that 1 requires random jumps only around migration and not always, which means its decidedly inferior to 2 w.r.t. ossification. Can we include a 1.1 which incorporates random jumps even in steady state? Or have we ruled that option out?

From: QUIC [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Praveen Balasubramanian
Sent: Monday, February 5, 2018 7:13 PM
To: Jana Iyengar <[email-masked]<mailto:[email-masked]>>

- jana

Thanks

---

From: Marten Seemann <[email-masked]>
Date: Tue, 06 Feb 2018 07:57:41 +0000

That's only true if you assume that the middlebox reads the version number
from the Long Header packets, and then only takes action if it understands
that version. Additionally, in case of a migrated connection, the middlebox
has no way to learn about the QUIC version at all, and would have to
refrain from taking any action at all. Furthermore, if we rotate connection
IDs more frequently, as Martin suggested, the middlebox would also become
ignorant of the QUIC version for that new connection ID, and would have to
stop reading the packet number for that flow.

These are a lot of ifs, and I'm not sure if it's a wise decision to put all
our trust in the manufacturers of middleboxes, that all of them will adhere
to all the assumptions outlined above.
Note that there's no other way than deploying a new QUIC version to
distinguish middleboxes implementing this correctly from middleboxes that
don't exactly play by these rules. Since in the beginning QUIC v1 will be
the only deployed QUIC version, it might be very tempting for middleboxes
to not implement these rules properly.

On Tue, Feb 6, 2018 at 3:34 PM Roni Even (A) <[email-masked]> wrote:

> *From:* QUIC [mailto:[email-masked]] *On Behalf Of *Marten Seemann
> *Sent:* Tuesday, February 06, 2018 9:31 AM
> *To:* Praveen Balasubramanian
> *Cc:* Jana Iyengar; Lubashev, Igor; Roberto Peon; QUIC WG; Salz, Rich
>
>
> On Tue, Feb 6, 2018 at 3:10 PM Praveen Balasubramanian <

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Tue, 6 Feb 2018 11:13:24 +0100

> On 6 Feb 2018, at 01:30, Jana Iyengar <[email-masked]> wrote:
> 

Unsurprisingly, I'm still for (3). :)

If a concrete proposal would incline you to consider it seriously, I'm happy to write one or two points in this space up, probably for the list at first. (Indeed, I'll probably go ahead and do this anyway today).

> My opinion is still for (2) over (1), but I can live with either.

FWIW, I prefer (2) over (1) as well, modulo a little discomfort about the crypto: the proposed encryption scheme uses a secret derived from the end-to-end secret to ECB-encrypt a plaintext which is completely known to an on-path observer in a place to see all packets in the common case that packets are neither lost nor reordered on the path -- namely the sequence 0x00 - 0x3f, 0x4000 - 0x7fff and so on. I Am Not A Cryptographer, but that property of this algorithm used in this way screams "Call A Cryptographer Before Deploying" to me.)

Cheers,

Brian

> On Mon, Feb 5, 2018 at 9:24 AM, Praveen Balasubramanian <[email-masked]> wrote:

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 6 Feb 2018 21:49:05 +0900

> 3. Packet numbers encrypted, plus a small modulo sequence number somewhere.

I wonder if we can provide opportunity to observe upstream loss /
reorder rate by just reserving one bit per packet.

The strawman is as follows.

Sender exposes one bit per packet that can be observed by on-path
devices. The bit is flipped for every 256 packets that the sender
sends. Observer records the bit of each packet as they pass by.

If there is no loss or reorder, the pattern an on-path device would
observe is repetition of 256 zeros and 256 ones. In other words, the
wavelength is 512 samples.

When there is a loss, the wavelength becomes shorter. The wavelength
becomes 512 * (1-loss_rate) samples. For example, if the wavelength
being observed is 384, loss rate is 25%.

If there is reorder, the edge (i.e. shift from zereos to ones, or vice
versa) becomes unclear. Duration of the transition will reflect the
extent of reordering.

Theoretically, it might be possible to explain the approach as a way
to let the sender emit a wave, then FFT the wave in on-path devices to
observe loss / reorder. Or it could be explained as just sending one
bit of the sequence number, somewhere in the middle (i.e., not LSB nor
MSB).

The pros of the approach is obvious; we will only spend 1 bit per
packet. Privacy impact is minimum. Implementing it is easy. Another
benefit is that it would be difficult (if not impossible) for on-path
devices to use the bit for purposes other than observation (i.e.,
change the ordering of the packets "for performance").

OTOH I am not sure how hard it would be to build tools for observing
upstream behavior using the proposed approach. Or I could well be
missing some requirements that operators have, as I am an endpoint
developer.

-- 
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 6 Feb 2018 08:35:12 -0500

Some overhead considerations of packet encryption:

Sensor and control devices:

The power overhead on sensor networks is small compared to just listening
on the radio and if not using radio, the latency is probably not critical
compared to the speed of encryption. To be a bit sci-fi - low-latency
encrypted control signals in robotics might be relevant. Think exo-skeleton
hacked by enemy through targeted signal injection. Still, packet number
encryption is likely fast enough in such cases.

For servers:

The overhead of packet encryption (single block AES-CTR) is probably around
100 cycles with AES-NI and possibly 160 cycles table based for one block
for Intel arch. This is because parallel multi-block optimisations are not
possible. AES-CTR of one block is about the same as AES-CTR of 4 blocks,
and GCM auth tag adds perhaps 50% to that. Thus packet number encryption
might be twice as fast as AES-GCM of a 64 byte message which is still a 50%
overhead.

High speed low latency applications might receive a packet concurrently on
two cores, one with a fast path reacting specifically to packets containing
only small stream messages, and one core handling everything else. The
message handler could experience a delay of 50% compared to not encrypting
packet numbers.

It is not possible to significantly improve on this in hardware because the
packet number must be decrypted before AEAD can be applied. That is, unless
the encrypted packet number is used as the IV of AEAD in which case
encryption gets a new bottleneck but decryption can be processed faster.

My take is that I prefer to avoid packet number gaps because they are messy
and I don’t believe they provide any significant privacy, but I also don’t
like the crypto cost in low-latency scenarios.

If the argument is for ossification and not privacy, I think the encryption
is the only option as Marten has pointed out. For this reason it makes
sense to apply crypto. Low-latency systems could create a QUIC variant that
drops the packet encryption.

Mikkel

If a concrete proposal would incline you to consider it seriously, I'm
happy to write one or two points in this space up, probably for the list at
first. (Indeed, I'll probably go ahead and do this anyway today).

> There are opinions about ossification, privacy, and manageability. I'm
not hearing bloody murder, so for what it's worth, I'd like to stipulate
first that we can all live with the final design, whichever one it is. If
anyone disagrees, please say so -- I think it'd be useful to hear if there
are folks who think a particular design is fundamentally problematic.
> My opinion is still for (2) over (1), but I can live with either.

FWIW, I prefer (2) over (1) as well, modulo a little discomfort about the
crypto: the proposed encryption scheme uses a secret derived from the
end-to-end secret to ECB-encrypt a plaintext which is completely known to
an on-path observer in a place to see all packets in the common case that
packets are neither lost nor reordered on the path -- namely the sequence
0x00 - 0x3f, 0x4000 - 0x7fff and so on. I Am Not A Cryptographer, but that
property of this algorithm used in this way screams "Call A Cryptographer
Before Deploying" to me.)

Cheers,

Brian

noted on this thread.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 6 Feb 2018 05:43:19 -0800

On 6 February 2018 at 14.35.12, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Low-latency systems could create a QUIC variant that drops the packet
encryption.

I meant drop the packet number encryption, not the full AEAD.

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 6 Feb 2018 15:20:34 +0100

I'd like to agree with Ted on this one.

Specifically I think we are discussing exposing some specific information in a header to assist in measurement and operations.

To me, this includes considering  the possibility that these specific fields could become ossified - but looking back at the thread, I think it is important also to note that such ossification is to the exposed info. It would NOT ossify the QUIC recovery, CC, flow control, options negotiation, etc - because these protocol mechanisms would not be exposed. This is in contrast to what happened to TCP! 

Gorry
 
----

> On 6 Feb 2018, at 01:49, Ted Hardie <[email-masked]> wrote:
> 
> If you'll forgive my saying so, evidence suggests that so many of the concrete aspects of such a proposal (where the bits would live, how many bits, etc.) would change over the lifetime of a proposal in this WG that any completely nailed down suggestion might as well be a blank sheet for all the bearing it would have on the final placement and length.
> 
> The question is:  do we wish to expose a small sequence number to assist in network analysis and management, yes or no?  
> 
> We can argue over the bike shed color after making the decision to build a shed; requiring the shed have a color before considering the question seems less productive to me than just answering the question.
>

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Tue, 6 Feb 2018 15:30:53 +0100

hi Kazuho, all,

(splitting the thread, trimming recipients)

We've been kicking around a few ideas like this one around the office today, more or less as you were writing this...

> On 6 Feb 2018, at 13:49, Kazuho Oku <[email-masked]> wrote:
> 

This is a pretty cool idea, but it doesn't provide enough information to protect the spin bit from reordering. This could be easily fixed by moving to a two-bit spin, as Gorry (I think) suggested. Among other things in this space, Piet's also looking into how much reordering information could be extracted from a two-bit spin itself.

There's a little complexity here -- you'd ideally like the starting point to depend on the window size, which you actually don't know at the starting point, and so on and so forth, but those are details.

> missing some requirements that operators have, as I am an endpoint

It's generally not the computational complexity of the task at a passive measurement point that is limiting to passive observation, it's the I/O (i.e. the amount of data that must be analyzed to generate a useful observation, and how far that data needs to be transmitted). Approaches like this one that trade off additional analysis-time complexity for ease of implementation and endpoint runtime simplicity are where we want to look, IMO.

Cheers,

Brian

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Tue, 6 Feb 2018 15:46:36 +0100

hi Ted,

(trimming, forking, #include apologies for google/apple mail shenanigans)

> If you'll forgive my saying so, evidence suggests that so many of the concrete aspects of such a proposal (where the bits would live, how many bits, etc.) would change over the lifetime of a propos…

Heh. Indeed.

> The question is:  do we wish to expose a small sequence number to assist in network analysis and management, yes or no?

A friendly amendment to refine this question, since I think it presupposes too much about the shape and color of said blank sheet of paper: do we wish to expose a signal that is designed to meet the following requirements:

- assist in on-path detection and measurement of upstream packet loss
- assist in on-path detection and measurement or upstream packet reordering
- improve the passive measurement of RTT in the presence of reordering (i.e., fix the spin bit to work without the packet number)

We (without the encryption PR) already have a single (implicit) signal for these three requirements, but we can without too much thought already see a few ways to add an implicit signal back in, probably even using only bits vacated in the short header by making encrypted packet numbers use varint encoding.

+1 to Ted: if we could at least come to consensus as a WG that these requirements are important, then we can separate these two discussions. Let's reserve some bits in the header, let those of us who care about the shape of the measurement nybble go off and work on a concrete proposal, and move on.

Cheers,

Brian

---

From: "Roni Even (A)" <[email-masked]>
Date: Tue, 6 Feb 2018 14:52:18 +0000

Brian,
I like your proposal as for the requirements I think that we are also talking about one point measurement and not two points.
Roni

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Brian Trammell (IETF)
Sent: Tuesday, February 06, 2018 4:47 PM
To: Ted Hardie
Cc: QUIC WG
Subject: Explicit measurability in the QUIC wire image (was Re: Packet number encryption)

hi Ted,

Heh. Indeed.

Cheers,

Brian

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 6 Feb 2018 23:52:33 +0900

Hi Brian,

Thank you for the response. Responding in-line.

> There's a little complexity here -- you'd ideally like the starting point to depend on the window size, which you actually don't know at the starting point, and so on and so forth, but those are de…

Actually I was thinking of having a bit that is independent from the
"spin bit", which is proposed to measure the latency.

We can have one bit that measures the latency, and one bit that
measures packet loss / reorder.

The former will be flipped every RTT. The latter will be flipped for
for every N packets being sent (I used the N=256 example).

My understanding is that we want to observe three properties: latency,
loss, reordering. To me it seems that we might be able to achieve that
using 2 bits. OTOH there could be other ways to achieve the three
indicators using 2 bits. I think that we would have an interesting
discussion.

> It's generally not the computational complexity of the task at a passive measurement point that is limiting to passive observation, it's the I/O (i.e. the amount of data that must be analyzed to ge…

Thank you very much for explaining that. It helps a lot. I would keep
that in mind while I follow the discussion.

-- 
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 6 Feb 2018 06:54:02 -0800

> sends. Observer records the bit of each packet as they pass by.

This is a pretty cool idea, but it doesn't provide enough information to
protect the spin bit from reordering. This could be easily fixed by moving
to a two-bit spin, as Gorry (I think) suggested. Among other things in this
space, Piet's also looking into how much reordering information could be
extracted from a two-bit spin itself.

This idea can be taken further: You can modulate the signal so you use
multiple packets to encode a zero or a one for half the wave duration. For
example 1 could be encoded as 0101 over four packets and zero could be
encoded as 0011 over four packets.
If there is any reordering, it would show up as an unexpected value in the
pulse.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 6 Feb 2018 07:09:17 -0800

You can take it even further and add error correcting codes into the
sequence.

Or likewise, you could generate a known random sequence such as PI, or
perhaps simpler, an AES generated sequence.

It is easy to sync the signal by hashing bits from multiple packets and
therefore easy to detect any missing bits. If there is massive loss, a
binary search can be conducted to find the location of the surviving
packets. It sort of breaks down if every other packet is reordered but you
could still handle that with some consideration.

The signal could repeat with a period significantly larger that practical
RTT rather than running an infinite sequence.

Technically it would also make it possible to remove the packet number
entirely because the number can be estimated by end-points and trial
decrypted, but that would likely not be performance optimal.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 6 Feb 2018 07:15:08 -0800

Such a signal would work against privacy unless it restarts with each new
connection ID or it is keyed by connection ID.

---

From: Mirja Kühlewind <[email-masked]>
Date: Tue, 6 Feb 2018 16:23:35 +0100

Hi all,
On 06.02.2018 14:35, Mikkel Fahnøe Jørgensen wrote:
> 
> If the argument is for ossification and not privacy, I think the 
> encryption is the only option as Marten has pointed out. For this reason 
> it makes sense to apply crypto. Low-latency systems could create a QUIC 
> variant that drops the packet encryption.

It's also my understanding that we do agree that packet number 
encryption does not help likability (and thereby privacy) a lot, or at 
least that the benefits we might get (or not) do not justify the 
additional complexity.

So let's talk about ossification. I'm also not certain yet that the 
additional complexity (even thought it is low; and I'm not only talking 
about implementation complexity but also the effect for someone new to 
understand the protocol and the spec) is justified by the fear people 
have about the big evil called ossification. And my point really is here 
that this is driven by a fear about things that theoretical could happen 
in the future and less about that the experience we have. Yes, we should 
be really careful here; that's definitely a lesson learned. And yes, I 
do understand well how painful and extremely said it was to see how 
quickly guic ossified. But we really need to look a bit deeper into what 
happened and why, e.g. what does ossification really mean and what do we 
know about it to handle it best.

In case of guic, there was a presumably fixed bit pattern at the 
beginning of each packet that was used to classify traffic as being 
guic. So I think one thing we can learn is that for a field that is not 
supposed to be static we should try make it not to look static (I guess 
the solution here are not exposing it or grease it; in general I do 
prefer the first option). However, the other apart of the question is 
why did this happen. The answer is because the network needed a way to 
classify guic traffic as guic. If we don't provide an easy way to 
identify guic traffic (where currently the only way is observing the 
handshake but that might be fine; we'll see), the chance that other 
information is misused for this purposed is high (and this information 
could also be a certain traffic pattern that is usually observed with 
quic, compared to other UDP based traffic; or a certain port number...). 
Alternatively, if there is no way to distinguish quic from random spam 
that is also based on UDP, I can actually imagine a future where the 
easiest way out is to block all UDP because at least for quic with its 
current use case it will likely fall back to TCP anyway, so no harm 
here, right... expect for all the other protocols that want to use UDP 
encapsulation eventually, but the we can still use TCP an 
encapsulation... or something like this... maybe. So this might be only 
my imagination of an alternative reality but this is a different kind of 
ossification that can happen in the Internet if we do the wrong thing, 
and I think that's the alternate reality I'd really like to avoid!

So let's see what we really know about ossification so far. I think the 
big lesson we can learn from TCP is that we need to be selective about 
what we expose to the network (also for privacy reason) but the other 
part of the lesson is also that using TCP header information for network 
monitoring (that have been exposed accidentally) has been a great win. 
Without the tools we have today, we would know less about the Internet 
than we already do and it would be much more effort and potentially 
economically not viable anymore to manage the networks we have have 
build with it current complexity. I'm all in for privacy and that's also 
a lesson learned, but I would really be disappointed if end up designing 
a protocol that would not be based on the things that we know have been 
good about TCP and just ignore that part.

Coming back to the packet number (and closer to this reality). Guic has 
been exposing the packet number for a long time now and this didn't 
cause an ossification problem. Yes, they presumingly didn't try to 
change it (actually I don't know that) but it is used (if at all) also 
in a different way than the flags field and for a different purpose. The 
packet number is a field that is increased on each packet. So if that is 
what is observed in the network and assumed to be mechanism then that's 
correct (different than the flags field with was assumed to be static 
but was not). Further network monitoring devices will likely use the 
packet number field for measurement but will not block the traffic if 
they don't find what they'll expect.

So to sum up, yes we need to consider possible ossification that can 
happen for all information we are going to expose (and I do mean all! 
including traffic pattern, packet sizes and timing). However, the 
conclusion from the past for me is not to not expose any information. It 
has proven of high value if information for network monitoring are 
available to the network, and the spin bit has show that it is possible 
to expose these information without a privacy impact. I do not see any 
evidence that the packet number is likely to ossify in a way that would 
hinder future protocol deployment and it will not increase likability, 
therefore I do think not changing the protocol and keeping the packet 
number as it is, is the best and easiest solution. However, if people 
really think it's better to encrypt the packet number, we must provide 
other measures for manageability.

Sorry for this long mail but I am very worried that the fear we have 
about this big evil word ossification is driving us into a wrong 
direction and we really should consider the different cases more 
separately and in detail.

Mirja

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 6 Feb 2018 10:25:05 -0500

The overhead could be reduced by generating an AES derived random byte
sequence and store 8 bits from the sequence in each packet. The sequence
can be efficiently precomputed for 64 packets at a time. A simple lookup
table will tell the packet number with high probability, and if it fails,
it is possible to extend the search. Details could be discussed, such as
using 12 bits instead of 8 etc.

This requires fewer header bits on the wire, is private, ossification
resistant, and fast.

---

From: Stephen Farrell <[email-masked]>
Date: Tue, 6 Feb 2018 15:35:27 +0000

> least that the benefits we might get (or not) do not justify the

FWIW, which isn't much, I don't (yet) agree to the above.
I reckon it'll be a while before we know how linkability
pans out. I do agree that packet number encryption is not
a panacea for unlinkability, but it may help, or may even
end up being required, to do a good job wrt linkability.

I also heard people say it was less complex for them so
"additional complexity" may also not be quite right. (I
guess "differently complex" is correct though.)

S.

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 7 Feb 2018 01:22:35 +0900

> If there is any reordering, it would show up as an unexpected value in the

That's definitely an option.

Actually, I considered using sin wave (that distributes the error
among 1-bit samples) as an example instead of the square wave.

The primary reason behind my preference to using a square wave is that
it is easier to implement; in fact, emitting a square wave with the
length of 512 packets is as easy as just taking the 9th bit (counting
from LSB) of the packet number (if we forget about the gaps for a
moment).

-- 
Kazuho Oku

---

From: Mike Bishop <[email-masked]>
Date: Tue, 6 Feb 2018 17:56:00 +0000

Packet number encryption *does* improve linkability equivalently to random jumps, but without the fragmentation of the ACK packet which those jumps otherwise cause.  So I'm with Stephen, we don't yet have agreement on that assertion.

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Stephen Farrell
Sent: Tuesday, February 6, 2018 7:35 AM
To: Mirja Kühlewind <[email-masked]>; Mikkel Fahnøe Jørgensen <[email-masked]>; Brian Trammell (IETF) <[email-masked]>; Jana Iyengar <[email-masked]>
Cc: Praveen Balasubramanian <[email-masked]>; QUIC WG <[email-masked]>; Roberto Peon <[email-masked]>; Lubashev, Igor <[email-masked]>; Salz, Rich <[email-masked]>
Subject: Re: Packet number encryption

FWIW, which isn't much, I don't (yet) agree to the above.
I reckon it'll be a while before we know how linkability pans out. I do agree that packet number encryption is not a panacea for unlinkability, but it may help, or may even end up being required, to do a good job wrt linkability.

I also heard people say it was less complex for them so "additional complexity" may also not be quite right. (I guess "differently complex" is correct though.)

S.

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 6 Feb 2018 15:00:32 -0800

I'm going to try a different attempt at moving towards
convergence. Thinking about this some more, and picking up something that
Mirja said way earlier in this thread, I think the following decomposition
is useful.

1. In the transport, all packet numbers start at 0, increasing
monotonically. ACK frames carry these packet numbers, and therefore do not
have to span large gaps and do not have to deal with increases in the size
of the largest acked. Basically, no gaps are visible within the transport
processing engine.
2. Before sending a packet on the wire, and before processing the packet,
QUIC transforms the packet number with some function, replacing the visible
packet number with the transformed value.

I think we can all agree that (1) is goodness. A sender is also free to use
packet numbers from non-contiguous spaces here FWIW, but that's entirely
independent of what's visible on the wire. This helps me separate transport
processing complexity from the rest of it, which I think is helpful.

I think we are disagreeing on the precise properties we want from the
transformation in (2). If we can agree on these properties, we can then
figure out whether it's possible and how to construct such a transform.
Here's a union of what folks are concerned about:
1. Packet number must be unlinkable across connection ID change (for
migration.)
2. Packet number must start at an arbitrary value, to avoid ossification of
the first packet number.
3. Some sequencing information -- a few bits of the packet number perhaps
-- should be revealed (for monitoring. Number of bits TBD.)
4. Any packet number transformation should not be compute intensive.

I'm not looking for a construction, but I'd like to agree on the problem
first. Does this sound like the set of properties we want? Or is there a
contradiction among these properties that I'm not seeing?

On Tue, Feb 6, 2018 at 9:56 AM, Mike Bishop <[email-masked]> wrote:

> Packet number encryption *does* improve linkability equivalently to random
> jumps, but without the fragmentation of the ACK packet which those jumps
> otherwise cause.  So I'm with Stephen, we don't yet have agreement on that
> assertion.
>
> To: Mirja Kühlewind <[email-masked]>; Mikkel Fahnøe
> Jørgensen <[email-masked]>; Brian Trammell (IETF) <[email-masked]>;
> Jana Iyengar <[email-masked]>
> Cc: Praveen Balasubramanian <[email-masked]>; QUIC WG <[email-masked]>;
> Roberto Peon <[email-masked]>; Lubashev, Igor <[email-masked]>; Salz,
> Rich <[email-masked]>
> I reckon it'll be a while before we know how linkability pans out. I do
> agree that packet number encryption is not a panacea for unlinkability, but
> it may help, or may even end up being required, to do a good job wrt
> linkability.
>
> I also heard people say it was less complex for them so "additional
> complexity" may also not be quite right. (I guess "differently complex" is
> correct though.)
>
> S.
>
> --

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 6 Feb 2018 23:58:33 +0000

I actually think a static 1KB table of Pi could work quite well.

A subset of samples could be indexed in a hashtable and yield loss data early with some probability. Some samples would miss due to reordering or loss. If more exact analysis is wanted the levenshtein distance of a sample sequence against Pi from a synced offset would tell a lot about the data. Or DNA sequencing. This could be hardware optimized.

Yet, transmission is always just the next bit in the Pi table.

Since Pi is effectively random it will ebentually detect mutations if the observed sequence is long enough.

Wrt privacy, the CID may be used as a start offset hash into the Pi table.

There isn't any need to cover a full RTT worth of Pi bits unless it is used for latency detection, but in that case the spin could be a pattern like 10101010 and 00001111 injected into the Pi sequence. A separate spin bit would be simpler.

________________________________
From: Kazuho Oku <[email-masked]>
Sent: Tuesday, February 6, 2018 5:22:35 PM
To: Mikkel Fahnøe Jørgensen
Cc: Brian Trammell (IETF); QUIC WG
Subject: Re: Explicit measurability in the QUIC wire image (was Re: Packet number encryption)

--
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 7 Feb 2018 00:10:42 +0000

Sounds reasonable to me. You could add that the transformation should be space efficient.

________________________________
From: Jana Iyengar <[email-masked]>
Sent: Wednesday, February 7, 2018 12:00:32 AM
To: Mike Bishop
Cc: Stephen Farrell; Mirja Kühlewind; Mikkel Fahnøe Jørgensen; Brian Trammell (IETF); Praveen Balasubramanian; QUIC WG; Roberto Peon; Lubashev, Igor; Salz, Rich
Subject: Re: Packet number encryption

I'm going to try a different attempt at moving towards convergence. Thinking about this some more, and picking up something that Mirja said way earlier in this thread, I think the following decomposition is useful.

1. In the transport, all packet numbers start at 0, increasing monotonically. ACK frames carry these packet numbers, and therefore do not have to span large gaps and do not have to deal with increases in the size of the largest acked. Basically, no gaps are visible within the transport processing engine.
2. Before sending a packet on the wire, and before processing the packet, QUIC transforms the packet number with some function, replacing the visible packet number with the transformed value.

I think we can all agree that (1) is goodness. A sender is also free to use packet numbers from non-contiguous spaces here FWIW, but that's entirely independent of what's visible on the wire. This helps me separate transport processing complexity from the rest of it, which I think is helpful.

I think we are disagreeing on the precise properties we want from the transformation in (2). If we can agree on these properties, we can then figure out whether it's possible and how to construct such a transform. Here's a union of what folks are concerned about:
1. Packet number must be unlinkable across connection ID change (for migration.)
2. Packet number must start at an arbitrary value, to avoid ossification of the first packet number.
3. Some sequencing information -- a few bits of the packet number perhaps -- should be revealed (for monitoring. Number of bits TBD.)
4. Any packet number transformation should not be compute intensive.

I'm not looking for a construction, but I'd like to agree on the problem first. Does this sound like the set of properties we want? Or is there a contradiction among these properties that I'm not seeing?

On Tue, Feb 6, 2018 at 9:56 AM, Mike Bishop <[email-masked]<mailto:[email-masked]>> wrote:
Packet number encryption *does* improve linkability equivalently to random jumps, but without the fragmentation of the ACK packet which those jumps otherwise cause.  So I'm with Stephen, we don't yet have agreement on that assertion.

-----Original Message-----
From: QUIC [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Stephen Farrell
Sent: Tuesday, February 6, 2018 7:35 AM
To: Mirja Kühlewind <[email-masked]<mailto:[email-masked]>>; Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>>; Brian Trammell (IETF) <[email-masked]<mailto:[email-masked]>>; Jana Iyengar <[email-masked]<mailto:[email-masked]>>
Cc: Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>; QUIC WG <[email-masked]<mailto:[email-masked]>>; Roberto Peon <[email-masked]<mailto:[email-masked]>>; Lubashev, Igor <[email-masked]<mailto:[email-masked]>>; Salz, Rich <[email-masked]<mailto:[email-masked]>>
Subject: Re: Packet number encryption

S.

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 6 Feb 2018 16:13:32 -0800

On Tue, Feb 6, 2018 at 3:00 PM, Jana Iyengar <[email-masked]> wrote:

> I think we can all agree that (1) is goodness. A sender is also free to
> use packet numbers from non-contiguous spaces here FWIW, but that's
> entirely independent of what's visible on the wire. This helps me separate
> transport processing complexity from the rest of it, which I think is
> helpful.
>
> -- should be revealed (for monitoring. Number of bits TBD.)

I'm not sure I am persuaded of this point

-Ekr

>> Packet number encryption *does* improve linkability equivalently to
>> random jumps, but without the fragmentation of the ACK packet which those
>> jumps otherwise cause.  So I'm with Stephen, we don't yet have agreement on
>> that assertion.
>>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 7 Feb 2018 00:29:12 +0000

Also note that this approach is insensitive to skipped packet numbers in optimistic ACK attack defence. It leaks zero information when the network is perfect and if the CID hash is random.

________________________________
From: Mikkel Fahnøe Jørgensen <[email-masked]>
Sent: Wednesday, February 7, 2018 12:58:33 AM
To: Kazuho Oku
Cc: Brian Trammell (IETF); QUIC WG
Subject: Re: Explicit measurability in the QUIC wire image (was Re: Packet number encryption)

There isn’t any need to cover a full RTT worth of Pi bits unless it is used for latency detection, but in that case the spin could be a pattern like 10101010 and 00001111 injected into the Pi sequence. A separate spin bit would be simpler.

--
Kazuho Oku

---

From: Ian Swett <[email-masked]>
Date: Tue, 6 Feb 2018 20:05:06 -0500

Thanks Jana, I think this is a nice framing of the problem and discussion.

> I'm not sure I am persuaded of this point

Yes, I think this seems like the largest point of contention, and #4 is the
second largest concern.  Given that, I think it makes sense to try to
resolve #3, if that is possible.

>>> Cc: Praveen Balasubramanian <[email-masked]>; QUIC WG <
>>> [email-masked]>; Roberto Peon <[email-masked]>; Lubashev, Igor <
>>> [email-masked]>; Salz, Rich <[email-masked]>

---

From: Christian Huitema <[email-masked]>
Date: Tue, 6 Feb 2018 15:59:23 -1000

On 2/6/2018 1:00 PM, Jana Iyengar wrote:
> transformation in (2). If we can agree on these properties, we can
> then figure out whether it's possible and how to construct such a
> transform. Here's a union of what folks are concerned about:

> 1. Packet number must be unlinkable across connection ID change (for

Yes. And note the special case when the old and new connection overlap
for some time, so that packet N and N+2 may be sent on the old
connection, packet N+1 and N+3 on the new one. That kind of pattern
should be hidden.

> ossification of the first packet number.

Please explain the reasoning. What do you call packet number there, the
value in the ACK or the value in the header?

> 3. Some sequencing information -- a few bits of the packet number
> perhaps -- should be revealed (for monitoring. Number of bits TBD.)

Careful there. This is per path information, so the network might
monitor the quality of a path. To continue my example of N and N+2 on
the old connection, packet N+1 and N+3 on the new one, I would expose x
and x+1 on the old path, y and y+1 on the new path.

> 4. Any packet number transformation should not be compute intensive.

For some definition of compute intensive. The current PN encryption spec
costs 1 block operation; that compares to 8+8 blocks for a 128 byte
payload (encrypt and checksum), 80+80 on a 1280 byte payload -- 6%
encryption overhead on small packets, 0.6% on large ones. That's less
than the difference between AES and Chacha20.

>
> I'm not looking for a construction, but I'd like to agree on the
> problem first. Does this sound like the set of properties we want? Or
> is there a contradiction among these properties that I'm not seeing?

I am a bit concerned about placing too much emphasis on computation
cost. This can lead to "inventing our own crypto substitute". If the PN
encoding is easy to break, then it will be routinely broken and ossified.

-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 7 Feb 2018 02:04:10 +0000

Great framing of the problem.

Proposal: Transform = Add a secure and fixed random offset to the actual packet number for the lifetime of one path. Transport level packet numbers start at 0. On wire packet number start at the offset. Client and server pick the offsets independently. Each side communicates this offset number in an encrypted frame until the first ACK. Other side subtracts the offset to figure out actual packet number and uses that to maintain receiver state and generate ACKs.

  1.  Packet number must be unlinkable across connection ID change (for migration.)
Each path gets a new secure random increment so maintains unlikability

  1.  Packet number must start at an arbitrary value, to avoid ossification of the first packet number.
True by definition for on wire packet number

  1.  Some sequencing information -- a few bits of the packet number perhaps -- should be revealed (for monitoring. Number of bits TBD.)
All of the bits are revealed on the wire and in sequence

  1.  Any packet number transformation should not be compute intensive.
Not at all compute intensive

From: Ian Swett [mailto:[email-masked]]
Sent: Tuesday, February 6, 2018 5:05 PM
To: Eric Rescorla <[email-masked]>
Cc: Jana Iyengar <[email-masked]>; QUIC WG <[email-masked]>; Praveen Balasubramanian <[email-masked]>; Lubashev, Igor <[email-masked]>; Roberto Peon <[email-masked]>; Mike Bishop <[email-masked]>; Salz, Rich <[email-masked]>; Mirja Kühlewind <[email-masked]>; Mikkel Fahnøe Jørgensen <[email-masked]>; Brian Trammell (IETF) <[email-masked]>; Stephen Farrell <[email-masked]>
Subject: Re: Packet number encryption

On Tue, Feb 6, 2018 at 7:13 PM, Eric Rescorla <[email-masked]<mailto:[email-masked]>> wrote:

On Tue, Feb 6, 2018 at 3:00 PM, Jana Iyengar <[email-masked]<mailto:[email-masked]>> wrote:
I'm going to try a different attempt at moving towards convergence. Thinking about this some more, and picking up something that Mirja said way earlier in this thread, I think the following decomposition is useful.

Yes, I think this seems like the largest point of contention, and #4 is the second largest concern.  Given that, I think it makes sense to try to resolve #3, if that is possible.

-Ekr

-----Original Message-----
From: QUIC [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Stephen Farrell
Sent: Tuesday, February 6, 2018 7:35 AM
To: Mirja Kühlewind <[email-masked]<mailto:[email-masked]>>; Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>>; Brian Trammell (IETF) <[email-masked]<mailto:[email-masked]>>; Jana Iyengar <[email-masked]<mailto:[email-masked]>>
Cc: Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>; QUIC WG <[email-masked]<mailto:[email-masked]>>; Roberto Peon <[email-masked]<mailto:[email-masked]>>; Lubashev, Igor <[email-masked]<mailto:[email-masked]>>; Salz, Rich <[email-masked]<mailto:[email-masked]>>
Subject: Re: Packet number encryption

S.

---

From: Martin Thomson <[email-masked]>
Date: Wed, 7 Feb 2018 17:13:10 +1100

That is exactly the solution I proposed.  As pointed out, this has
several shortcomings, one of which Christian explained in his last
email.  It is also not appreciably less computation.  Considerably
more care is required to get this right.

On Wed, Feb 7, 2018 at 1:04 PM, Praveen Balasubramanian
<[email-masked]> wrote:
> Proposal: Transform = Add a secure and fixed random offset to the actual
> packet number for the lifetime of one path. Transport level packet numbers
> start at 0. On wire packet number start at the offset. Client and server
> pick the offsets independently. Each side communicates this offset number in
> an encrypted frame until the first ACK. Other side subtracts the offset to
> figure out actual packet number and uses that to maintain receiver state and
> generate ACKs.
>
>
>
> Packet number must be unlinkable across connection ID change (for
> migration.)
>
> Packet number must start at an arbitrary value, to avoid ossification of the
> first packet number.
>
> Some sequencing information -- a few bits of the packet number perhaps --
> should be revealed (for monitoring. Number of bits TBD.)
>
> Any packet number transformation should not be compute intensive.
>
> Cc: Jana Iyengar <[email-masked]>; QUIC WG <[email-masked]>; Praveen
> Balasubramanian <[email-masked]>; Lubashev, Igor <[email-masked]>;
> Roberto Peon <[email-masked]>; Mike Bishop <[email-masked]>; Salz, Rich
> <[email-masked]>; Mirja Kühlewind <[email-masked]>;
> Mikkel Fahnøe Jørgensen <[email-masked]>; Brian Trammell (IETF)
> <[email-masked]>; Stephen Farrell <[email-masked]>
>
>
> I'm going to try a different attempt at moving towards convergence. Thinking
> about this some more, and picking up something that Mirja said way earlier
> in this thread, I think the following decomposition is useful.
>
>
>
> 3. Some sequencing information -- a few bits of the packet number perhaps --
> should be revealed (for monitoring. Number of bits TBD.)
>
>
>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 7 Feb 2018 06:31:19 +0000

A subtraction and addition per packet is certainly much less computation compared to encryption/decryption. Our experiment shows 1 to 1.25% overhead due by adding this extra crypto step in the data path. I don’t get the part about more care, its just a transform as Jana pointed out and the core transport code doesn’t need to make any changes, the transform is simply done during framing after or before all processing. I'd like to understand the shortcomings better and see if we can solve them without needing expensive operations in the data path, one option would be to have a one time fixed size packet number jump for the new path so the numbers don’t get interleaved with gaps. 

-----Original Message-----
From: Martin Thomson [mailto:[email-masked]] 
Sent: Tuesday, February 6, 2018 10:13 PM
To: Praveen Balasubramanian <[email-masked]>
Cc: Ian Swett <[email-masked]>; Eric Rescorla <[email-masked]>; Salz, Rich <[email-masked]>; Lubashev, Igor <[email-masked]>; Roberto Peon <[email-masked]>; Mirja Kühlewind <[email-masked]>; Mike Bishop <[email-masked]>; Brian Trammell (IETF) <[email-masked]>; Jana Iyengar <[email-masked]>; Mikkel Fahnøe Jørgensen <[email-masked]>; QUIC WG <[email-masked]>; Stephen Farrell <[email-masked]>
Subject: Re: Packet number encryption

That is exactly the solution I proposed.  As pointed out, this has several shortcomings, one of which Christian explained in his last email.  It is also not appreciably less computation.  Considerably more care is required to get this right.

On Wed, Feb 7, 2018 at 1:04 PM, Praveen Balasubramanian <[email-masked]> wrote:
> Proposal: Transform = Add a secure and fixed random offset to the 
> actual packet number for the lifetime of one path. Transport level 
> packet numbers start at 0. On wire packet number start at the offset. 
> Client and server pick the offsets independently. Each side 
> communicates this offset number in an encrypted frame until the first 
> ACK. Other side subtracts the offset to figure out actual packet 
> number and uses that to maintain receiver state and generate ACKs.
>
>
>
> Packet number must start at an arbitrary value, to avoid ossification 
> of the first packet number.
>
> Some sequencing information -- a few bits of the packet number perhaps 
> Balasubramanian <[email-masked]>; Lubashev, Igor 
> <[email-masked]>; Roberto Peon <[email-masked]>; Mike Bishop 
> <[email-masked]>; Salz, Rich <[email-masked]>; Mirja Kühlewind 
> <[email-masked]>;
> I'm going to try a different attempt at moving towards convergence. 
> Thinking about this some more, and picking up something that Mirja 
> said way earlier in this thread, I think the following decomposition is useful.
>
>
>
> monotonically. ACK frames carry these packet numbers, and therefore do 
> not have to span large gaps and do not have to deal with increases in 
> the size of the largest acked. Basically, no gaps are visible within 
> the transport processing engine.
>
> 2. Before sending a packet on the wire, and before processing the 
> packet, QUIC transforms the packet number with some function, 
> replacing the visible packet number with the transformed value.
>
>
>
> I think we can all agree that (1) is goodness. A sender is also free 
> to use packet numbers from non-contiguous spaces here FWIW, but that's 
> entirely independent of what's visible on the wire. This helps me 
> separate transport processing complexity from the rest of it, which I think is helpful.
>
>
>
> then figure out whether it's possible and how to construct such a transform.
> Yes, I think this seems like the largest point of contention, and #4 
> is the second largest concern.  Given that, I think it makes sense to 
> try to resolve #3, if that is possible.
>
>
>
>
>
> -Ekr
>
>
>
> random jumps, but without the fragmentation of the ACK packet which 
> those jumps otherwise cause.  So I'm with Stephen, we don't yet have 
> agreement on that assertion.
>
> Jørgensen <[email-masked]>; Brian Trammell (IETF) 
> <[email-masked]>; Jana Iyengar <[email-masked]>
> Cc: Praveen Balasubramanian <[email-masked]>; QUIC WG 
> <[email-masked]>; Roberto Peon <[email-masked]>; Lubashev, Igor 
> <[email-masked]>; Salz, Rich <[email-masked]>
>> encryption does not help likability (and thereby privacy) a lot, or 
>> at least that the benefits we might get (or not) do not justify the 
>> additional complexity.
>
> I reckon it'll be a while before we know how linkability pans out. I 
> do agree that packet number encryption is not a panacea for 
> unlinkability, but it may help, or may even end up being required, to 
> do a good job wrt linkability.
>
> complexity" may also not be quite right. (I guess "differently 
> complex" is correct though.)
>
> S.
>
> --

---

From: Christian Huitema <[email-masked]>
Date: Tue, 6 Feb 2018 20:50:14 -1000

> A subtraction and addition per packet is certainly much less computation compared to encryption/decryption. Our experiment shows 1 to 1.25% overhead due by adding this extra crypto step in the data…

There are at least two big shortcomings.

First, the proposal does not obscure the position of the packet number
field in the header. The field contains a number that increments with
every packet, and is thus easy to identify. This means that middle-boxes
will get accustomed to seeing that field and tracking its increments.
Hence, this simple obfuscation provides no protection against
ossification of the packet number field.

Second, the proposal does not protect against linkage, if several paths
are used simultaneously during a migration. Instead of progressing by
increments of 1, the PN will progress by increments that depends on the
number of packets sent on other paths. This allows for easy correlation
of these paths, despite the random increment.

In short, this simple obfuscation does not protect against ossification
and also does not prevent linkability.

-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 7 Feb 2018 06:55:57 +0000

Inline prefixed by >>>

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Christian Huitema
Sent: Tuesday, February 6, 2018 10:50 PM
To: [email-masked]
Subject: Re: Packet number encryption

First, the proposal does not obscure the position of the packet number field in the header. The field contains a number that increments with every packet, and is thus easy to identify. This means that middle-boxes will get accustomed to seeing that field and tracking its increments.
Hence, this simple obfuscation provides no protection against ossification of the packet number field.

>>> Well it seems like it is a requirement (see bullet 3 in Jana's list) to allow middleboxes to see sequencing information. By picking the initial random value middleboxes cannot assume any particular value. If packet number location and size is not part of invariants then it cannot be ossified, middleboxes will need to be resilient. 

Second, the proposal does not protect against linkage, if several paths are used simultaneously during a migration. Instead of progressing by increments of 1, the PN will progress by increments that depends on the number of packets sent on other paths. This allows for easy correlation of these paths, despite the random increment.

>>> You missed this part of my reply: "one option would be to have a one time fixed size packet number jump for the new path so the numbers don’t get interleaved with gaps. " so if packets are sent on both paths for a while they will still progress by increments of 1.

In short, this simple obfuscation does not protect against ossification and also does not prevent linkability.

-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Tue, 6 Feb 2018 21:19:27 -1000

>>>> Well it seems like it is a requirement (see bullet 3 in Jana's list) to allow middleboxes to see sequencing information. By picking the initial random value middleboxes cannot assume any particu…

This is why I would rather see the exposed bits copied at a fixed
location, while the PN itself cannot be located. That way, the exposed
bits can ossify, but the actual PN does not.

-- Christian Huitema

---

From: Marten Seemann <[email-masked]>
Date: Wed, 07 Feb 2018 08:00:45 +0000

On Wed, Feb 7, 2018 at 3:20 PM Christian Huitema <[email-masked]>
wrote:

> location, while the PN itself cannot be located. That way, the exposed bits
> can ossify, but the actual PN does not.
>

We still need to be very careful here. If we decide to expose bits, we must
make sure that a middlebox can only extract coarse-grained information, if
we want to prevent middleboxes from buffering and delaying packets in order
to fix / reduce reordering.
At this point however, I'm not convinced that exposing any reordering
information to the network should be a requirement at all. There's a real
risk of misuse of this information, and of ossification of whatever bits we
expose.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 7 Feb 2018 00:38:39 -0800

I am fairly convinced that you can and should separate exposure to
middleware from packet numbers in whatever form precisely because we want
packet numbers to not rely on path and because packet numbers may expose
false loss if exposed directly in whatever form.

A path specific counter can be done be exposing a few bits of an internal
path specific counter that is entirely independent of packet numbering used
for AEAD IV and is also independent of any artificial packet number
skipping for ACK attack mitigation etc.. It could also be done be emitting
a known 1-bit signal whose pattern can be used to identify deviance from
that pattern, as discussed in separate thread, e.g. a square wave or a
public pseudo-random sequence.

The next, and separate question is then, do we want to expose anything even
if it only leaks information about loss and reordering on a single path? It
could lead to unwanted latency due to middleware packet sorting. It is a
very good point, but the concern might be based on a historical case that
does not apply to current UDP or future QUIC transports. However, if
ordering is exposed as a pseudo-random 1-bit signal such as the Pi
sequence, then middleware would have to work very harder than it would find
worthwhile to sort packets back in order.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 7 Feb 2018 01:10:04 -0800

Instead of a static Pi table, an xorshift+ PRNG could be used. It would
likely be preferable to shorten the cycle such that observers can use hash
lookups and hamming distance computations against selected segments of the
sequence.

https://en.wikipedia.org/wiki/Xorshift#xorshift+

http://xoroshiro.di.unimi.it/xorshift128plus.c

uint64_t s[2];

uint64_t next(void) {
	uint64_t s1 = s[0];
	const uint64_t s0 = s[1];
	const uint64_t result = s0 + s1;
	s[0] = s0;
	s1 ^= s1 << 23; // a
	s[1] = s1 ^ s0 ^ (s1 >> 18) ^ (s0 >> 5); // b, c
	return result;
}

A simpler 32-bit version could also be used since we do not care about
long cycles.

http://excamera.com/sphinx/article-xorshift.html

uint32_t seed = 7;  // 100% random seed value

static uint32_t random()
{
  seed ^= seed << 13;
  seed ^= seed >> 17;
  seed ^= seed << 5;
  return seed;
}

The output would be the bits shifted out of the 32-bit our 64-bit word in
either MSB or LSB first. I’d prefer LSB because it is slightly simpler to
test the LSB bit and divide by 2.

On 7 February 2018 at 01.29.18, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Also note that this approach is insensitive to skipped packet numbers in
optimistic ACK attack defence. It leaks zero information when the network
is perfect and if the CID hash is random.

------------------------------
*From:* Mikkel Fahnøe Jørgensen <[email-masked]>
*Sent:* Wednesday, February 7, 2018 12:58:33 AM
*To:* Kazuho Oku
*Cc:* Brian Trammell (IETF); QUIC WG
*Subject:* Re: Explicit measurability in the QUIC wire image (was Re:
Packet number encryption)

A subset of samples could be indexed in a hashtable and yield loss data
early with some probability. Some samples would miss due to reordering or
loss. If more exact analysis is wanted the levenshtein distance of a sample
sequence against Pi from a synced offset would tell a lot about the data.
Or DNA sequencing. This could be hardware optimized.

Since Pi is effectively random it will ebentually detect mutations if the
observed sequence is long enough.

There isn’t any need to cover a full RTT worth of Pi bits unless it is used
for latency detection, but in that case the spin could be a pattern like
10101010 and 00001111 injected into the Pi sequence. A separate spin bit
would be simpler.

------------------------------
*From:* Kazuho Oku <[email-masked]>
*Sent:* Tuesday, February 6, 2018 5:22:35 PM
*To:* Mikkel Fahnøe Jørgensen
*Cc:* Brian Trammell (IETF); QUIC WG
*Subject:* Re: Explicit measurability in the QUIC wire image (was Re:
Packet number encryption)

--
Kazuho Oku

---

From: Martin Thomson <[email-masked]>
Date: Wed, 7 Feb 2018 21:10:20 +1100

I looked into these when I first considered packet number encryption.
They are not typically bijective on 2^N-1.  That's not a great outcome
here.  Also, random access is a requirement and that isn't
straightforward here, even with the good ones.  Also reversing is
possible, but more complicated.

On Wed, Feb 7, 2018 at 8:10 PM, Mikkel Fahnøe Jørgensen
<[email-masked]> wrote:
> A simpler 32-bit version could also be used since we do not care about long
> cycles.
>
> optimistic ACK attack defence. It leaks zero information when the network is
> perfect and if the CID hash is random.
>
> sequence against Pi from a synced offset would tell a lot about the data. Or
> DNA sequencing. This could be hardware optimized.
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 7 Feb 2018 02:23:56 -0800

On 7 February 2018 at 11.10.21, Martin Thomson ([email-masked])
wrote:

Hi Martin, I think you might be misreading the intention behind this
sequence.

The idea is to generate 1 bit for every packet sent on a specific path, in
the order the packet was sent, not to encode packet numbers directly. The
encoding of packet numbers is a separate thing whether encrypted or
otherwise.

When an observer collects these bits into words from several packets, it is
possible to compare the gathered string with the expected sequence.

The random access is only needed for analytical purposes to know about
packet loss in the network, not to operate the QUIC transport as such. If
the sequence is fixed and repeats with reasonable cycle length, it can be
done efficiently in hardware.

It is true that if you sample, say 32 bits, the mapping is not bijective so
if you match the sequence, it is ambiguous. But since you already know what
to expect next and you can choose the sample length as long as you want,
say 128 bits, you do get very good information about where loss and
reordering happened. If the sequence was not random, you would not have
this property.

As to reverse engineering, packets are already sent in order, so learning
what the original order was, is hardly a significant compromise, except if
you have multiple connections on the same tuple without transmitting an
observable connection ID.

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 7 Feb 2018 20:10:02 +0900

> lookups and hamming distance computations against selected segments of the

The pros of using a sequence that randomly distributes 0 and ones
(i.e. PI or xorshift+) is that you can extract good loss / reordering
information if the error rate is low.

OTOH, it is fragile to high error rate or certain patterns of error.
Let me explain two examples.

Consider the case where the packet loss rate is 50%. I doubt if you
could use pattern matching against those sequences to figure out the
loss rate. This is because the power among the sequence is mostly high
frequencies. If you use a sequence that distributes power among lower
frequencies (like square wave or sin wave that I discussed), you can
calculate the loss rate in such cases as well.

Consider another case where the sender uses multiple queues to emit
packets belonging to a single QUIC connection (BTW, I plan to
implement such one). You would see non-neglible amount of reorders. On
the wire, it will look like if two adjacent packets packets were
swapped; it wouldn't have any practical impact on performance. But
such frequent reordering will destroy the observability of a random
sequence.

To summarize, I think that we need a low frequency signal. It could
either be a square wave or sin wave. Using a sin wave might be better
here, since you the bit pattern (which flips a lot) to observe at a
higher precision when the the error rates of loss and reorder are both
low.

-- 
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 7 Feb 2018 03:26:58 -0800

comment inline

On 7 February 2018 at 12.10.06, Kazuho Oku ([email-masked]) wrote:

This is a good point. However, you could average the random signal over
multiple bits to get a lower frequency signal which will give you signature
markers even when some segments are lost. A sine is the ultimate low
frequency filter.

There might be a middle ground where you overlay a low frequency signal
into the random sequence.

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 7 Feb 2018 12:33:22 +0100

Hi,

see below.

> Am 07.02.2018 um 09:00 schrieb Marten Seemann <[email-masked]>:
> 
> 
> 
> On Wed, Feb 7, 2018 at 3:20 PM Christian Huitema <[email-masked]> wrote:
> 
> 
> This is why I would rather see the exposed bits copied at a fixed location, while the PN itself cannot be located. That way, the exposed bits can ossify, but the actual PN does not.
> 
> We still need to be very careful here. If we decide to expose bits, we must make sure that a middlebox can only extract coarse-grained information, if we want to prevent middleboxes from buffering and delaying packets in order to fix / reduce reordering.

This may be done by the network anyway. We were working on a multipath bandwidth bonding approach for UDP-based traffic and had to add a slim tunnel header with a sequence number for reordering. As we didn’t want look any deeper in the packet and see if we can find any hints if the traffic is reorder-sensitive or not, we had to work on the conservative assumption that we should avoid reordering to avoid unwanted negative impacts as this is the assumption some protocols make today based on what we typically can obverse today.

If we want to say that quic traffic should not be reordered in such situations, we would need an easy way to distinguish quic from other ups traffic, or even provide an explicit signal (but that’s a different topic).

> At this point however, I'm not convinced that exposing any reordering information to the network should be a requirement at all. There's a real risk of misuse of this information, and of ossification of whatever bits we expose.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 7 Feb 2018 07:21:51 -0500

Hmm a periodic truly white noise random signal ought to have an evenly
distributed power spectrum meaning it contains all frequencies up to half
its cycle length. At least there is a lot of solid communications theory on
random signals that could be applied, e.g.

https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-011-introduction-to-communication-control-and-signal-processing-spring-2010/readings/MIT6_011S10_chap09.pdf

http://bme.elektro.dtu.dk/31610/notes/chap_8.pdf

Thus limiting the signal to a square or a sine is effectively wasting a lot
of bandwidth, even if naive hash based signal processing is insufficient to
deal with a proper random signal. Of course there is the argument of
simplicity, but long term I believe more information in the signal is more
valuable.

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 7 Feb 2018 21:52:14 +0900

2018-02-07 19:23 GMT+09:00 Mikkel Fahnøe Jørgensen <[email-masked]>:
> the order the packet was sent, not to encode packet numbers directly.

What Mikkel says.

Or put it another way, below is a short summary of the approaches
being proposed:

* encode couple of least significant bits of the per-path sequence
number ("a small-modulo sequence number") [1]
* encode one bit in the middle of the per-path sequence number [2]

When observing that one bit of [2] spanning across multiple packets
belonging to a particular path, it looks like a square wave. There
came out variations of the approach.

* use sin wave instead of square wave [3]
* use bitsteram of PI instead of square wave [4][5]
* use bitstream of xorshift+ instead of square wave [6]

I would argue that the approaches discussed in [2] to [6] are superior
to [1] due to the fact that they use less bits, expose less
information, while possibly given us reasonable observation of loss
and reorder rate.

[1] https://www.ietf.org/mail-archive/web/quic/current/msg03030.html
[2] https://www.ietf.org/mail-archive/web/quic/current/msg03146.html
[3] https://www.ietf.org/mail-archive/web/quic/current/msg03146.html
[4] https://www.ietf.org/mail-archive/web/quic/current/msg03156.html
[5] https://www.ietf.org/mail-archive/web/quic/current/msg03171.html
[6] https://www.ietf.org/mail-archive/web/quic/current/msg03186.html

> The
> to expect next and you can choose the sample length as long as you want, say
> 128 bits, you do get very good information about where loss and reordering
> happened. If the sequence was not random, you would not have this property.
>

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 7 Feb 2018 22:13:26 +0900

2018-02-07 21:21 GMT+09:00 Mikkel Fahnøe Jørgensen <[email-masked]>:
> deal with a proper random signal.

I mostly disagree.

The reason I am suggesting to use square or sin wave is because it has
a base frequency (i.e. wavelength) that can be used to calculate the
loss rate.

White noise does not have a "base frequency." Therefore, it becomes
harder to observe the metrics when the loss / reorder rate becomes
high.

Going back to the discussion of the approaches, I am not against
mixing in some random signals so that pattern matching can be applied
when the error rate is low. But such "mix" should be mild enough that
it would not disturb the observation of the base frequency under high
error rate.

And in fact, you do not even need to mix randomly flipping bits if you
select a sin wave, since a sin wave quantized to 1-bit will contain
many flipping bits. You can use those bits for pattern matching. OTOH,
if we select square wave, it might make sense to apply some bit
flipping so that we can apply pattern matching. There are other
possibilities as well.

However, with that said, I believe that the design decisions need to
depend on the actual use-case. What are the loss rates that we need to
work on? What are the reorder patterns that we would observe? IMO this
is the tricky part of the 1-bit approach.

> Of course there is the argument of

-- 
Kazuho Oku

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Wed, 7 Feb 2018 14:34:39 +0100

hi Jana,

popping back up from the detailed engineering, thank you for this reframing.

I largely agree that this is the question; a few comments / friendly amendments on each point.

> On 7 Feb 2018, at 00:00, Jana Iyengar <[email-masked]> wrote:
> 
> I think we can all agree that (1) is goodness.

Yep! (modulo the (completely separate and inconsequential for this discussion) question of whether packet numbers start at 0 or 1)

> A sender is also free to use packet numbers from non-contiguous spaces here FWIW, but that's entirely independent of what's visible on the wire. This helps me separate transport processing complexity from the rest of it, which I think is helpful.

+1.

> 1. Packet number must be unlinkable across connection ID change (for migration.)

s/must/should endeavor to be/ -- true unlinkability is unattainable. The goal here is to make sure the packet number doesn't make it a lot easier.

> 2. Packet number must start at an arbitrary value, to avoid ossification of the first packet number.

Presuming that the model of ossification you're using here is "a middlebox vendor grabbing on to any bits it can find to distinguish quic from not-quic UDP traffic", then yes, this seems reasonable.

(There are an entirely different space of designs, which contains TCP, where the packet number itself is an implicit signal to prove to the path that a packet belongs to a flow beyond the five-tuple. In that case the arbitrary start would be necessary to have actual entropy for that signal. But we've abandoned this as a requirement in QUIC, in part due to the fundamental tension between avoiding linkability and proving things to devices on the path about the relationships among packets, i.e., "linkability" in other words.)

> 3. Some sequencing information -- a few bits of the packet number perhaps -- should be revealed (for monitoring. Number of bits TBD.)

This is the crux of the argument. On one side we have the risk of misuse and ossification (well, not ossification -- these bits are *meant* for the path -- rather the risk that we'll figure out later that we specified the wrong thing), on the other side we have the loss of visibility into how QUIC traffic interacts with the network as compared to TCP, with a side question of whether or not this visibility is really the transport layer's problem despite the evolution the practice of diagnostics and troubleshooting using TCP information.

If we can come to agreement on this question, everything else falls into place. I have my arguments here, but as you said, this subthread is not the place for them. :)

> 4. Any packet number transformation should not be compute intensive.

I'd say "should not have an inordinate marginal impact on performance", because otherwise you go down the rabbithole of relative complexity, but yeah.

Cheers,

Brian

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 7 Feb 2018 17:43:56 +0000

Couple of things to add:

1. Connection migration is an optional portion of the transport and not mandatory. It does not seem to be incrementally deployable with today's unmodified load balancers. Many devices may also be single homed and not have multiple active network interfaces. I also think the app and system settings will have to opt-in to it (to prevent aggressive usage of metered cellular data). If this is the case, cross-path linkability will be of limited concern for many QUIC connections. There is also the fact that perfect unlinkability seems to be hard against a motivated adversary.

2. If middlebox ossification is a big concern, we mainly need obfuscation and not encryption. Another option for example would be an XOR transform with some variable field in the encrypted portion of the packet (as an example sequence number of the first stream frame if it is present or some random pad bytes when its not present). I hope we can be creative here to reduce computation cost.

From: Marten Seemann [mailto:[email-masked]]
Sent: Wednesday, February 7, 2018 12:01 AM
To: huitema <[email-masked]>
Cc: Praveen Balasubramanian <[email-masked]>; [email-masked]
Subject: Re: Packet number encryption

On Wed, Feb 7, 2018 at 3:20 PM Christian Huitema <[email-masked]<mailto:[email-masked]>> wrote:

---

From: Christian Huitema <[email-masked]>
Date: Wed, 7 Feb 2018 14:37:17 -1000

> calculate the loss rate in such cases as well.

Actually, it should be possible to build a maximum likelihood decoder
for an xorshift sending pattern. You are mostly concerned with erasure
and permutations, and you roughly know the range of probability, so you
can see likelihood of 1, 2, 3, etc. erasures, or permutations in some
range. Obviously, the range that you can manage depends on the length of
the code. If you can keep the decoder synchronized, you will be able to
measure the erasure rate. If it looses synchronization, you know that
the error rate is truly horrendous, and that's information too. And
xorshift codes are self synchronizing, so you will regain
synchronization quickly.

-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Wed, 7 Feb 2018 14:55:36 -1000

> xorshift codes are self synchronizing, so you will regain

What I had in mind was a polynomial code, such as crc32 or crc64. The
generation operation is:

    unsigned int:1 next_bit_in_pattern()
    {
        static uint32_t seed = 0xc00lcode;
        const uint32_t reminder = 0x1edc6f41;
        overflow = seed>>31;
        seed <<= 1;
        seed^= (overflow*reminder);
        return (seed&1);
    }

Writing the trellis decoder is left as an exercise for the reader...

-- Christian Huitema

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 8 Feb 2018 11:08:25 +0900

2018-02-08 9:37 GMT+09:00 Christian Huitema <[email-masked]>:
> measure the erasure rate.

I agree with you that it is possible to detect loss / reorder if the
decoder can be kept synchronized.

What I am questioning is the possibility of keeping the decoder
synchronized under heavy loss / reorder.

> If it looses synchronization, you know that
> the error rate is truly horrendous, and that's information too.

I disagree.

It might be true that loosing synchronization due to heavy loss can be
a signal. OTOH, loosing synchronization due to massive reorder is not
an issue if the reorder resolves in a certain timeframe (IIRC 1/8
RTT).

My understanding is that the middleboxes' reliance to the sequence
number has led to the ossification of TCP. The ossification has
prevented us from doing some nifty things, such as sending some TCP
packets through a different node while answering others directly. In
QUIC, I am interested in doing such things, and I would expect that
others will be.

Considering that, I would prefer not to exposing a signal that could
give false information to observers when there is a "good" reorder.

> And

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 8 Feb 2018 12:54:45 +0900

> give false information to observers when there is a "good" reorder.

Please let me paraphrase the last sentence to the following:

I would prefer using a signal that is tolerant to "good" reorder (i.e.
happening frequently but resolving in a fraction of RTT so that they
don't trigger a retransmit).

Anyways, I think that we are making progress in sense that we are
discussing what is the "best" way to achieve loss / reorder
observability using 1 bit, rather than using multiple bits.

Use of less bits lead to less chance of leaking information as well as
finding space to store the information.

-- 
Kazuho Oku

---

From: Victor Vasiliev <[email-masked]>
Date: Thu, 8 Feb 2018 14:49:48 -0500

On Wed, Feb 7, 2018 at 12:43 PM, Praveen Balasubramanian <
[email-masked]> wrote:
>
> 2. If middlebox ossification is a big concern, we mainly need obfuscation
> and not encryption. Another option for example would be an XOR transform
> with some variable field in the encrypted portion of the packet (as an
> example sequence number of the first stream frame if it is present or some
> random pad bytes when its not present). I hope we can be creative here to
> reduce computation cost.
>
Obfuscation is just encryption done poorly.

I am concerned that in 2018, we still are talking about encryption being
expensive -- symmetric block encryption, of all things.

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 8 Feb 2018 19:59:51 +0000

*   I am concerned that in 2018, we still are talking about encryption being expensive -- symmetric block encryption, of all things.

+1 !

---

From: Praveen Balasubramanian <[email-masked]>
Date: Thu, 8 Feb 2018 22:32:45 +0000

AFAICT the goal is to perform some transform for preventing middlebox ossification. Encryption is one possible transform but is it the only possible one? The numbers I quoted were from a quick test by modifying the winquic implementation to add an additional encrypt / decrypt step on packet number size worth of data. If anyone else has any data please share. The design choice here impacts per packet processing costs for the server side hence the request to find if we can do something simpler.

From: QUIC [mailto:[email-masked]] On Behalf Of Victor Vasiliev
Sent: Thursday, February 8, 2018 11:50 AM
To: Praveen Balasubramanian <[email-masked]>
Cc: Marten Seemann <[email-masked]>; [email-masked]; huitema <[email-masked]>
Subject: Re: Packet number encryption

On Wed, Feb 7, 2018 at 12:43 PM, Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>> wrote:

I am concerned that in 2018, we still are talking about encryption being expensive -- symmetric block encryption, of all things.

---

From: Mike Bishop <[email-masked]>
Date: Thu, 8 Feb 2018 23:13:37 +0000

#2 won’t work.  The packet number is an input to the AEAD nonce, so you have to get to the packet number before you can decrypt the payload.  It can’t be based on anything internal to the encrypted packet.

In Martin’s proposal, it’s using a combination of the encrypted payload (which requires nothing to access) and some stored state on each endpoint, which requires only the Connection ID to look up.  You want the thing you XOR by to be constantly changing, or it becomes easier to look at sequences of obfuscated packet numbers and start drawing inferences about what the XOR value is.  If overhead is the primary concern, I think there are a couple approaches that might allow you to precompute values and amortize the cost of doing the encryption step.  (At the cost of keeping the table of precomputed values, of course.)

I still think Jana’s pushing in the right direction here:  Let’s agree on goals, and then see if we can craft a design that meets those goals.  I hear you saying that low overhead is a goal, which I think we can all agree on.  I also think I hear you saying that 1-1.5% overhead is too high, but it sounds like there’s less agreement on that.  You also raise an interesting point that some clients will care less about linkability because they will never intentionally migrate.

From: QUIC [mailto:[email-masked]] On Behalf Of Praveen Balasubramanian
Sent: Wednesday, February 7, 2018 9:44 AM
To: Marten Seemann <[email-masked]>; huitema <[email-masked]>
Cc: [email-masked]
Subject: RE: Packet number encryption

Couple of things to add:

From: Marten Seemann [mailto:[email-masked]]
Sent: Wednesday, February 7, 2018 12:01 AM
To: huitema <[email-masked]<mailto:[email-masked]>>
Cc: Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: Packet number encryption

---

From: Christian Huitema <[email-masked]>
Date: Thu, 8 Feb 2018 13:53:46 -1000

On 2/8/2018 1:13 PM, Mike Bishop wrote:

> #2 won’t work.  The packet number is an input to the AEAD nonce, so
> you have to get to the packet number before you can decrypt the
> payload.  It can’t be based on anything internal to the encrypted packet.
>
>  
>
> In Martin’s proposal, it’s using a combination of the encrypted
> payload (which requires nothing to access) and some stored state on
> each endpoint, which requires only the Connection ID to look up.  You
> want the thing you XOR by to be constantly changing, or it becomes
> easier to look at sequences of obfuscated packet numbers and start
> drawing inferences about what the XOR value is.  If overhead is the
> primary concern, I think there are a couple approaches that might
> allow you to precompute values and amortize the cost of doing the
> encryption step.  (At the cost of keeping the table of precomputed
> values, of course.)
>
>  
>
> I still think Jana’s pushing in the right direction here:  Let’s agree
> on goals, and then see if we can craft a design that meets those
> goals.  I hear you saying that low overhead is a goal, which I think
> we can all agree on.  I also think I hear you saying that 1-1.5%
> overhead is too high, but it sounds like there’s less agreement on
> that.  You also raise an interesting point that some clients will care
> less about linkability because they will never intentionally migrate.
>

I think Victor said it best, "Obfuscation is just encryption done
poorly." Either we encrypt, or we don't. But there is no such thing as
an obfuscation middle-ground. Obfuscation would just be a speed bump,
easily broken. It won't provide privacy, and it won't prevent linkability.

As far as I am concerned, there are only two designs that prevent
linkability:

1) Martin's PR, in which the packet number is encrypted using sensible
crypto;

2) Or, leaving the packet number in the clear but using different
encryption contexts for each path, and using separate packet sequences
for each path.

The second option requires more computation when explicitly setting up a
new path. The connection ID would be used as part of the "salt" when
deriving connection contexts. The packet number on each path will start
at 1 (or maybe 0 ;-). This requires maintaining separate SACK dashboards
for each path/Connection-ID, and constraining ACK to be per/path, or
alternatively adding a PATH-ACK frame that indicates the path for which
packets are acked.

The pros and cons of the two options are easy to get:

1) Option 2 does not require the cost of encrypting every packet, thus
per packet processing will be maybe 1% faster than option 1.

2) Option 2 requires computing encryption contexts for each path, which
is more complex than option 1, but the cost will be amortized over the
duration of the path.

3) Option 2 requires more complex data structures and ACK management,
thus increasing the chances that something goes wrong.

4) Option 2 does nothing against packet number ossification. The packet
number effectively becomes part of the invariants.

In short, the only advantage of option 2 is to avoid a single symmetric
encryption operation per packet, while making the code significantly
more complex, and at the cost of not protecting against PN ossification.
The way I see that, there is hardly any debate.

-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 00:46:07 +0000

The input to the nonce will be the transformed value in cleartext.

From: Mike Bishop [mailto:[email-masked]]
Sent: Thursday, February 8, 2018 3:14 PM
To: Praveen Balasubramanian <[email-masked]>; Marten Seemann <[email-masked]>; huitema <[email-masked]>
Cc: [email-masked]
Subject: RE: Packet number encryption

From: QUIC [mailto:[email-masked]] On Behalf Of Praveen Balasubramanian
Sent: Wednesday, February 7, 2018 9:44 AM
To: Marten Seemann <[email-masked]<mailto:[email-masked]>>; huitema <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: Packet number encryption

Couple of things to add:

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 00:51:49 +0000

Also I think we are all mostly in agreement with the goals Jana stated. I was simply trying to find a less compute intensive solution. Yes migration is not universally applicable to all devices or connections and is not a mandatory part of the transport so the less any overhead needed for unlinkability by default the better.

From: QUIC [mailto:[email-masked]] On Behalf Of Praveen Balasubramanian
Sent: Thursday, February 8, 2018 4:46 PM
To: Mike Bishop <[email-masked]>; Marten Seemann <[email-masked]>; huitema <[email-masked]>
Cc: [email-masked]
Subject: RE: Packet number encryption

To: Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>; Marten Seemann <[email-masked]<mailto:[email-masked]>>; huitema <[email-masked]<mailto:[email-masked]>>

Couple of things to add:

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 01:50:46 +0000

>> and using separate packet sequences for each path
This alone prevents linkability right?

>> The packet number on each path will start at 1 (or maybe 0 ;-).
Is supporting multiple paths at the same time a goal? If yes, then yes all paths will need to have separate tracking and ACK state and congestion control etc. I thought this is not part of V1 RFC. If its just one additional path for connection migration scenario then all you need is a (random) jump to a offset in packet number space for the new path so that the few packets in the old path can continue for a little bit. This works with just one ACK space so no implementation complexity.

>> 2) Or, leaving the packet number in the clear but using different encryption contexts for each path, and using separate packet sequences for each path.
In addition if the value in the clear was obfuscated with an XOR transform it will help prevent ossification. The obfuscation is not meant to guard against later analysis that reveals the packet numbers - why do we care about that. It is meant to grease the space and not make the numbers serial so middleboxes cannot assume ordering etc.

All said if I am alone in being concerned about the perf hit then I’d say go on with option 1.

From: QUIC [mailto:[email-masked]] On Behalf Of Christian Huitema
Sent: Thursday, February 8, 2018 3:54 PM
To: [email-masked]
Subject: Re: Packet number encryption

I think Victor said it best, "Obfuscation is just encryption done poorly." Either we encrypt, or we don't. But there is no such thing as an obfuscation middle-ground. Obfuscation would just be a speed bump, easily broken. It won't provide privacy, and it won't prevent linkability.

As far as I am concerned, there are only two designs that prevent linkability:

1) Martin's PR, in which the packet number is encrypted using sensible crypto;

2) Or, leaving the packet number in the clear but using different encryption contexts for each path, and using separate packet sequences for each path.

The second option requires more computation when explicitly setting up a new path. The connection ID would be used as part of the "salt" when deriving connection contexts. The packet number on each path will start at 1 (or maybe 0 ;-). This requires maintaining separate SACK dashboards for each path/Connection-ID, and constraining ACK to be per/path, or alternatively adding a PATH-ACK frame that indicates the path for which packets are acked.

1) Option 2 does not require the cost of encrypting every packet, thus per packet processing will be maybe 1% faster than option 1.

2) Option 2 requires computing encryption contexts for each path, which is more complex than option 1, but the cost will be amortized over the duration of the path.

3) Option 2 requires more complex data structures and ACK management, thus increasing the chances that something goes wrong.

4) Option 2 does nothing against packet number ossification. The packet number effectively becomes part of the invariants.

In short, the only advantage of option 2 is to avoid a single symmetric encryption operation per packet, while making the code significantly more complex, and at the cost of not protecting against PN ossification. The way I see that, there is hardly any debate.

-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 05:40:04 +0000

Actually taking a step back what if we don’t we overloaded packet number as the nonce?

Can we:

  1.  Put packet number as a separate variable length field in the encrypted portion of the header. This allows for growth beyond 64 bits in future and completely avoids ossification. For short lived flows variable length encoding leads to minimal space overhead.
  2.  Put a separate plaintext nonce which starts out as a secure random number and monotonically increases as short random jumps (every packet or every few packets) and could then rollover. This will prevent linkability and scales to multi-path as well. No major compute overhead. We can also make the nonce optional for crypto algorithms that do not need a nonce saving more space.

Some MSB of the nonce could be used for signaling reorder to the network if needed. I don’t have the historical context for overloading of PN as the nonce so pardon me if un-overloading option was discussed and ruled out before.

From: QUIC [mailto:[email-masked]] On Behalf Of Praveen Balasubramanian
Sent: Thursday, February 8, 2018 5:51 PM
To: huitema <[email-masked]>; [email-masked]
Subject: RE: Packet number encryption

From: QUIC [mailto:[email-masked]] On Behalf Of Christian Huitema
Sent: Thursday, February 8, 2018 3:54 PM
To: [email-masked]<mailto:[email-masked]>
Subject: Re: Packet number encryption

-- Christian Huitema

---

From: Mike Bishop <[email-masked]>
Date: Fri, 9 Feb 2018 06:19:18 +0000

You can’t safely reuse the combination of nonce and key, IIUC.  Since key rollover is infrequent, the nonce has to change every packet and be guaranteed unique over the lifetime of, if not the connection, at least the current key.  Since packet numbers also don’t repeat over the lifetime of the connection, I think using one as the variable part of the other was just taking advantage of the alignment.

I don’t think having a randomized starting value and incrementing is necessarily any better than exposing the packet number.  Having a PRNG seed established during the handshake and making the visible thing the next random output seems potentially interesting, but I’d want our more crypto-minded friends to comment.  I think the real issue would be guarantees about never-repeating from the PRNG, particularly as you don’t have guarantees about your peer’s implementation.  A defined key derivation and a counter can be verified in the protocol, independent of how good the peer’s PRNG is.

From: QUIC [mailto:[email-masked]] On Behalf Of Praveen Balasubramanian
Sent: Thursday, February 8, 2018 9:40 PM
To: huitema <[email-masked]>; [email-masked]
Subject: RE: Packet number encryption

Can we:

From: QUIC [mailto:[email-masked]] On Behalf Of Praveen Balasubramanian
Sent: Thursday, February 8, 2018 5:51 PM
To: huitema <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: RE: Packet number encryption

-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 07:05:45 +0000

Does this mean we require a key rollover today for every 4 GB of data transfer since the PN is only a 4 byte field? Also I don’t see then how the PN encryption proposal is guaranteeing non-repeating output?

You can start with secure random number and then use PRNG for small jumps to prevent ossification. And depending on how often you jump you will then require key rollover early until you rollover and hit the starting nonce number. The other good part of un-overloading is that we can make the nonce optional when not needed needed by crypto.

From: Mike Bishop [mailto:[email-masked]]
Sent: Thursday, February 8, 2018 10:19 PM
To: Praveen Balasubramanian <[email-masked]>; huitema <[email-masked]>; [email-masked]
Subject: RE: Packet number encryption

Can we:

-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 07:08:28 +0000

Scratch the question on PN encryption proposal since it’s the plaintext that’s the nonce not the encrypted version.

From: QUIC [mailto:[email-masked]] On Behalf Of Praveen Balasubramanian
Sent: Thursday, February 8, 2018 11:06 PM
To: Mike Bishop <[email-masked]>; huitema <[email-masked]>; [email-masked]
Subject: RE: Packet number encryption

From: Mike Bishop [mailto:[email-masked]]
Sent: Thursday, February 8, 2018 10:19 PM
To: Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>; huitema <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: RE: Packet number encryption

Can we:

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 9 Feb 2018 02:22:04 -0500

You can’t safely reuse the combination of nonce and key, IIUC.  Since key
rollover is infrequent, the nonce has to change every packet and be
guaranteed unique over the lifetime of, if not the connection, at least the
current key.

For CTR mode encryption which AES-GCM is, it is extremely bad to reuse the
nonce. But it is still permitted if nonce only repeats with very low
probability, like 2^-32 or so, don’t recall. This means that using a crypto
hashed packet numbers as a nonce is an option, but a homegrown obfuscation
is not.

As to hashing the packet number instead of encrypting, there is hardly any
worthwhile hash that is faster than using AES, but precomputing the AES
over several blocks as Mike and I have suggested earlier, is an option
unless you expect large losses where a small lookup table is insufficient
on average.

I share the concern on relying on the a peers weak PRNG for packet jumps
etc. especially  if it is not the same as the crypto PRNG because if that
is broken, it doesn’t matter anyway.

As the the 1% number - it is the only real life data we have, but I already
argued that the overhead can much higher in low-latency operations with
short packets. It has also been argued that packet number is lower than
AEAD because it is shorter, but encrypting 64 bytes AEAD for a message is
the same as encrypting 16 bytes packet number because of instruction level
parallelism on current Intel hardware. You can see how the performance
increases with data length in appendix A

https://software.intel.com/sites/default/files/m/d/4/1/d/8/10TB24_Breakthrough_AES_Performance_with_Intel_AES_New_Instructions.final.secure.pdf

Mikkel

On 9 February 2018 at 07.19.32, Mike Bishop ([email-masked]) wrote:

You can’t safely reuse the combination of nonce and key, IIUC.  Since key
rollover is infrequent, the nonce has to change every packet and be
guaranteed unique over the lifetime of, if not the connection, at least the
current key.  Since packet numbers also don’t repeat over the lifetime of
the connection, I think using one as the variable part of the other was
just taking advantage of the alignment.

I don’t think having a randomized starting value and incrementing is
necessarily any better than exposing the packet number.  Having a PRNG seed
established during the handshake and making the visible thing the next
random output seems potentially interesting, but I’d want our more
crypto-minded friends to comment.  I think the real issue would be
guarantees about never-repeating from the PRNG, particularly as you don’t
have guarantees about your peer’s implementation.  A defined key derivation
and a counter can be verified in the protocol, independent of how good the
peer’s PRNG is.

*From:* QUIC [mailto:[email-masked]] *On Behalf Of *Praveen
Balasubramanian
*Sent:* Thursday, February 8, 2018 9:40 PM
*To:* huitema <[email-masked]>; [email-masked]
*Subject:* RE: Packet number encryption

Actually taking a step back what if we don’t we overloaded packet number as
the nonce?

Can we:

   1. Put packet number as a separate variable length field in the
   encrypted portion of the header. This allows for growth beyond 64 bits in
   future and completely avoids ossification. For short lived flows variable
   length encoding leads to minimal space overhead.
   2. Put a separate plaintext nonce which starts out as a secure random
   number and monotonically increases as short random jumps (every packet or
   every few packets) and could then rollover. This will prevent linkability
   and scales to multi-path as well. No major compute overhead. We can also
   make the nonce optional for crypto algorithms that do not need a nonce
   saving more space.

Some MSB of the nonce could be used for signaling reorder to the network if
needed. I don’t have the historical context for overloading of PN as the
nonce so pardon me if un-overloading option was discussed and ruled out
before.

*From:* QUIC [mailto:[email-masked] <[email-masked]>] *On
Behalf Of *Praveen Balasubramanian
*Sent:* Thursday, February 8, 2018 5:51 PM
*To:* huitema <[email-masked]>; [email-masked]
*Subject:* RE: Packet number encryption

>> The packet number on each path will start at 1 (or maybe 0 ;-).

Is supporting multiple paths at the same time a goal? If yes, then yes all
paths will need to have separate tracking and ACK state and congestion
control etc. I thought this is not part of V1 RFC. If its just one
additional path for connection migration scenario then all you need is a
(random) jump to a offset in packet number space for the new path so that
the few packets in the old path can continue for a little bit. This works
with just one ACK space so no implementation complexity.

>> 2) Or, leaving the packet number in the clear but using different
encryption contexts for each path, and using separate packet sequences for
each path.
In addition if the value in the clear was obfuscated with an XOR transform
it will help prevent ossification. The obfuscation is not meant to guard
against later analysis that reveals the packet numbers - why do we care
about that. It is meant to grease the space and not make the numbers serial
so middleboxes cannot assume ordering etc.

All said if I am alone in being concerned about the perf hit then I’d say
go on with option 1.

*From:* QUIC [mailto:[email-masked] <[email-masked]>] *On
Behalf Of *Christian Huitema
*Sent:* Thursday, February 8, 2018 3:54 PM
*To:* [email-masked]
*Subject:* Re: Packet number encryption

#2 won’t work.  The packet number is an input to the AEAD nonce, so you
have to get to the packet number before you can decrypt the payload.  It
can’t be based on anything internal to the encrypted packet.

In Martin’s proposal, it’s using a combination of the encrypted payload
(which requires nothing to access) and some stored state on each endpoint,
which requires only the Connection ID to look up.  You want the thing you
XOR by to be constantly changing, or it becomes easier to look at sequences
of obfuscated packet numbers and start drawing inferences about what the
XOR value is.  If overhead is the primary concern, I think there are a
couple approaches that might allow you to precompute values and amortize
the cost of doing the encryption step.  (At the cost of keeping the table
of precomputed values, of course.)

I still think Jana’s pushing in the right direction here:  Let’s agree on
goals, and then see if we can craft a design that meets those goals.  I
hear you saying that low overhead is a goal, which I think we can all agree
on.  I also think I hear you saying that 1-1.5% overhead is too high, but
it sounds like there’s less agreement on that.  You also raise an
interesting point that some clients will care less about linkability
because they will never intentionally migrate.

I think Victor said it best, "Obfuscation is just encryption done poorly."
Either we encrypt, or we don't. But there is no such thing as an
obfuscation middle-ground. Obfuscation would just be a speed bump, easily
broken. It won't provide privacy, and it won't prevent linkability.

2) Or, leaving the packet number in the clear but using different
encryption contexts for each path, and using separate packet sequences for
each path.

The second option requires more computation when explicitly setting up a
new path. The connection ID would be used as part of the "salt" when
deriving connection contexts. The packet number on each path will start at
1 (or maybe 0 ;-). This requires maintaining separate SACK dashboards for
each path/Connection-ID, and constraining ACK to be per/path, or
alternatively adding a PATH-ACK frame that indicates the path for which
packets are acked.

1) Option 2 does not require the cost of encrypting every packet, thus per
packet processing will be maybe 1% faster than option 1.

2) Option 2 requires computing encryption contexts for each path, which is
more complex than option 1, but the cost will be amortized over the
duration of the path.

3) Option 2 requires more complex data structures and ACK management, thus
increasing the chances that something goes wrong.

In short, the only advantage of option 2 is to avoid a single symmetric
encryption operation per packet, while making the code significantly more
complex, and at the cost of not protecting against PN ossification. The way
I see that, there is hardly any debate.

-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 07:25:42 +0000

The proposal is to keep the PN in encrypted portion of header and use a dedicated nonce in plain text. If non-repeating nonce is needed with that probability then we will need much larger nonce since 10 Gbps+ rates are common in the datacenter. A nonce can simply be generated using PRNG. Why does the peer’s PRNG matter? The peer uses a different PN space. The jumps are completely determined by the sender as needed and its just to prevent ossification of assuming that nonce always increments by 1.

From: Mikkel Fahnøe Jørgensen [mailto:[email-masked]]
Sent: Thursday, February 8, 2018 11:22 PM
To: Praveen Balasubramanian <[email-masked]>; Mike Bishop <[email-masked]>; [email-masked]; huitema <[email-masked]>
Subject: RE: Packet number encryption

You can’t safely reuse the combination of nonce and key, IIUC.  Since key rollover is infrequent, the nonce has to change every packet and be guaranteed unique over the lifetime of, if not the connection, at least the current key.
For CTR mode encryption which AES-GCM is, it is extremely bad to reuse the nonce. But it is still permitted if nonce only repeats with very low probability, like 2^-32 or so, don’t recall. This means that using a crypto hashed packet numbers as a nonce is an option, but a homegrown obfuscation is not.

As to hashing the packet number instead of encrypting, there is hardly any worthwhile hash that is faster than using AES, but precomputing the AES over several blocks as Mike and I have suggested earlier, is an option unless you expect large losses where a small lookup table is insufficient on average.

I share the concern on relying on the a peers weak PRNG for packet jumps etc. especially  if it is not the same as the crypto PRNG because if that is broken, it doesn’t matter anyway.

As the the 1% number - it is the only real life data we have, but I already argued that the overhead can much higher in low-latency operations with short packets. It has also been argued that packet number is lower than AEAD because it is shorter, but encrypting 64 bytes AEAD for a message is the same as encrypting 16 bytes packet number because of instruction level parallelism on current Intel hardware. You can see how the performance increases with data length in appendix A

https://software.intel.com/sites/default/files/m/d/4/1/d/8/10TB24_Breakthrough_AES_Performance_with_Intel_AES_New_Instructions.final.secure.pdf<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsoftware.intel.com%2Fsites%2Fdefault%2Ffiles%2Fm%2Fd%2F4%2F1%2Fd%2F8%2F10TB24_Breakthrough_AES_Performance_with_Intel_AES_New_Instructions.final.secure.pdf&data=04%7C01%7Cpravb%40microsoft.com%7C20e615aa80de43354ae108d56f8dd2c9%7Cee3303d7fb734b0c8589bcd847f1c277%7C1%7C1%7C636537577282331419%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-1&sdata=2Sh%2BzP6WC7fBGrKwUHUh4CEJTe2hWk3bR8ZAN%2BQcD1c%3D&reserved=0>

Mikkel

On 9 February 2018 at 07.19.32, Mike Bishop ([email-masked]<mailto:[email-masked]>) wrote:
You can’t safely reuse the combination of nonce and key, IIUC.  Since key rollover is infrequent, the nonce has to change every packet and be guaranteed unique over the lifetime of, if not the connection, at least the current key.  Since packet numbers also don’t repeat over the lifetime of the connection, I think using one as the variable part of the other was just taking advantage of the alignment.

Can we:

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 9 Feb 2018 02:43:47 -0500

Why does the peer’s PRNG matter? The peer uses a different PN space. The
jumps are completely determined by the sender as needed and its just to
prevent ossification of assuming that nonce always increments by 1.

Because linkability goes both ways.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 9 Feb 2018 02:50:33 -0500

On 9 February 2018 at 08.26.02, Praveen Balasubramanian ([email-masked])
wrote:

 If non-repeating nonce is needed with that probability then we will need
much larger nonce since 10 Gbps+ rates are common in the datacenter. A
nonce can simply be generated using PRNG.

It is decided by the lifetime of the key, not the bandwidth. The lifetime
is typically limited to about 2^32 bytes because AES-GCM says so though
other ciphers could allow for more.

The nonce can be generated by a PRNG, but incidentally a solid PRNG is
often implemented by encrypting a counter, so we are back to square one,
only now using more space and crypto.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 9 Feb 2018 03:25:07 -0500

The lifetime is typically limited to about 2^32 bytes because AES-GCM says
so though

There is a limit in bytes per packet as well, but I actually I meant to say
2^32 invocations, i.e. distinct packets and nonces based on NIST
requirements, FWTW. IETF TLS requires IV’s (aka nonce) to be unique whereas
NIST allows for random IV but it must then use 96 bits and still not exceed
2^32 invocations.

Since IETF does not permit reuse, that random construct is out, and it
would also be using too much space.

http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38d.pdf

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 9 Feb 2018 12:53:55 +0000

Can you please explain why you want the packet number in the clear?  Or post a link to the email archive if you’ve already done so.  Because, to my casual observer viewpoint, you are trying fairly hard to do this, including some tortured/questionable crypto; so, why?

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 9 Feb 2018 05:33:35 -0800

On Thu, Feb 8, 2018 at 11:22 PM, Mikkel Fahnøe Jørgensen <[email-masked]

> hashed packet numbers as a nonce is an option, but a homegrown obfuscation

I'm not sure where you are getting this from. Any nonce reuse at all with
AES-GCM is catastrophic:
http://csrc.nist.gov/groups/ ST/toolkit/BCM/documents/comments/800-38_
Series-Drafts/GCM/Joux_comments.pdf

-Ekr

Mikkel
> Some MSB of the nonce could be used for signaling reorder to the network
> if needed. I don’t have the historical context for overloading of PN as the

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 9 Feb 2018 05:51:47 -0800

On Thu, Feb 8, 2018 at 9:40 PM, Praveen Balasubramanian <[email-masked]

>    make the nonce optional for crypto algorithms that do not need a nonce
I'm not sure how to do #2 safely and efficiently. The primary concern is
that short random jumps don't actually prevent linkability because the high
order bits don't change and long random jumps are effectively randomly
sampling out of the nonce space, and you start to run into birthday
problems. Additionally, the cheapest PRNGs are basically ciphers in counter
mode, so at best you're going to get a small amount of overhead reduction
by getting multiple jump computations out of a single block, though if you
use 96-bit jumps, you only get 25% savings...

-Ekr

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 9 Feb 2018 09:05:36 -0500

On 9 February 2018 at 14.34.17, Eric Rescorla ([email-masked]) wrote:

I read it in a NIST paper I also linked to in other mail here, but it
appears they have gone back on that based on the Ferguson paper. It nows
says Legacy in url, but still shows up prominently in a google search.

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 9 Feb 2018 06:07:16 -0800

As far as I know, this is just NIST's judgement of how low a risk of
nonce-reuse has to be in order to be safe. To the best of my knowledge,
there's still no safe level of actual reuse. For that you'd need something
like AES-GCM-SIV.

-Ekr

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 9 Feb 2018 09:09:35 -0500

OK, but as I also pointed out in another recent mail, correcting myself,
IETF requires unique nonces in TLS, so the point is mood.

---

From: Ian Swett <[email-masked]>
Date: Fri, 9 Feb 2018 09:14:39 -0500

On Fri, Feb 9, 2018 at 8:51 AM, Eric Rescorla <[email-masked]> wrote:

>
>
> On Thu, Feb 8, 2018 at 9:40 PM, Praveen Balasubramanian <
>> Is supporting multiple paths at the same time a goal? If yes, then yes
>> all paths will need to have separate tracking and ACK state and congestion
>> In addition if the value in the clear was obfuscated with an XOR
>> transform it will help prevent ossification. The obfuscation is not meant
>> to guard against later analysis that reveals the packet numbers - why do we
>> care about that. It is meant to grease the space and not make the numbers
>> serial so middleboxes cannot assume ordering etc.
>>
>>
>>
>> poorly." Either we encrypt, or we don't. But there is no such thing as an
> I don't believe the second is necessary if we want to leave the PN in the
clear.  Avoiding nonce reuse is relatively easy.  Avoiding linkability is
more annoying, but doable.

On net, I think encryption may be the best approach, but I don't think not
doing encryption demands the complexity you indicate.

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 17:15:06 +0000

No I don’t want it in the clear. The proposal is to put PN in encrypted header and use a separate nonce field in the clear. Please ignore the whole obfuscation discussion for now, that was an attempt to produce a transform cheaper than crypto and doesn’t seem to be going anywhere.

From: Salz, Rich [mailto:[email-masked]]
Sent: Friday, February 9, 2018 4:54 AM
To: Praveen Balasubramanian <[email-masked]>; Mikkel Fahnøe Jørgensen <[email-masked]>; Mike Bishop <[email-masked]>; [email-masked]; huitema <[email-masked]>
Subject: Re: Packet number encryption

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 17:38:02 +0000

Let me try to summarize the proposal again:

  1.  Packet number is in the encrypted portion of the header and is no longer used as the nonce. It is encoded as a variable length number. Always starts at 0.
     *   No ossification of PN
     *   No linkability using PN
     *   Allows for growth beyond 64 bits if needed in future
  2.  The nonce is in the clear in plaintext header. It is generated per path as a secure random number. Call this the “starting value”. Both sides MUST initialize the “starting value” nonce per path. Within a path it grows monotonically (to avoid repeats just like PN today) and can include short random jumps periodically so that middleboxes cannot assume anything about the field. Since nonce has nothing to do with PN the jumps don’t add any complexity to transport logic. Jumps are for greasing and preventing ossification. When the nonce rolls over and reaches close to the “starting value”, key rollover must happen. At high data rates the nonce field can be 64-bits to prevent frequent key rollover. For crypto that doesn’t need nonces, the field can be omitted.
     *   No ossification because there is no pre-determined sequence to the nonce.
     *   In gQUIC and the RFC as defined the nonce is a simple sequence that increments by one, so this is no worse off in terms of security. It is non-repeating.
     *   No linkability because different paths start at a completely different random number. If repeating is a concern then different paths use different keys. Carries forward to not just connection migration but mutli-path.
     *   Supports key rollover just like gQUIC and currently defined draft RFC.
     *   Allows for omitting the nonce when we use crypto that doesn’t need it.
     *   Allows for larger nonces for high data rates.
     *   It doesn’t preclude any other type of non-repeating nonces without having to modify the RFC! So implementations that want to use cryptographic nonces can still do so.

The downsides:

  1.  Even with variable length encoding the extra PN field is more space used per packet when a nonce is required. I think we can do with 2 bytes in version 1.
  2.  It’s not perfect unlinkabiltiy but I have yet to see any solution that can provide this. A simple time based analysis using machine learning will allow flow correlation especially for connection migration. The IP addresses in the clear allow for a high degree of correlation. There is also the application sending pattern and other signals that could be used.

From: Eric Rescorla [mailto:[email-masked]]
Sent: Friday, February 9, 2018 5:52 AM
To: Praveen Balasubramanian <[email-masked]>
Cc: huitema <[email-masked]>; [email-masked]
Subject: Re: Packet number encryption

On Thu, Feb 8, 2018 at 9:40 PM, Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>> wrote:
Actually taking a step back what if we don’t we overloaded packet number as the nonce?

Can we:

I'm not sure how to do #2 safely and efficiently. The primary concern is that short random jumps don't actually prevent linkability because the high order bits don't change and long random jumps are effectively randomly sampling out of the nonce space, and you start to run into birthday problems. Additionally, the cheapest PRNGs are basically ciphers in counter mode, so at best you're going to get a small amount of overhead reduction by getting multiple jump computations out of a single block, though if you use 96-bit jumps, you only get 25% savings...

-Ekr

-- Christian Huitema

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 9 Feb 2018 17:42:23 +0000

Thanks for the summary.  Why do you want to do this?

---

From: Ian Swett <[email-masked]>
Date: Fri, 9 Feb 2018 13:20:52 -0500

One issue with the current encryption proposal is you can't use hardware
crypto offload for the bulk encryption because the packet number has to be
encrypted after the bulk data.

Any suggestions on how to fix that?

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 19:10:12 +0000

Because of the listed advantages. Clear separation of concerns results if we don’t overload the packet number as the nonce. Connection migration and multi-path are optional features and we pay no extra compute cost for most connections that are not impacted by unlnkability. Nonce field becomes optional for crypto that doesn’t need it and can grow to larger size as needed for bulk transfers to prevent frequent key rollovers (say in datacenter scenarios).

From: Salz, Rich [mailto:[email-masked]]
Sent: Friday, February 9, 2018 9:42 AM
To: Praveen Balasubramanian <[email-masked]>; Eric Rescorla <[email-masked]>
Cc: [email-masked]; huitema <[email-masked]>
Subject: Re: Packet number encryption

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 19:11:50 +0000

+1 that any proposed solution should not preclude crypto offload.

From: Ian Swett [mailto:[email-masked]]
Sent: Friday, February 9, 2018 10:21 AM
To: Salz, Rich <[email-masked]>
Cc: Praveen Balasubramanian <[email-masked]>; Eric Rescorla <[email-masked]>; [email-masked]; huitema <[email-masked]>
Subject: Re: Packet number encryption

One issue with the current encryption proposal is you can't use hardware crypto offload for the bulk encryption because the packet number has to be encrypted after the bulk data.

On Fri, Feb 9, 2018 at 12:42 PM, Salz, Rich <[email-masked]<mailto:[email-masked]>> wrote:
Thanks for the summary.  Why do you want to do this?

---

From: David Benjamin <[email-masked]>
Date: Fri, 09 Feb 2018 19:38:57 +0000

Your primary worry here was the extra block cipher operation, right? By
moving the packet number into the plaintext, this proposal increases the
length, which will cost that same one extra block cipher operation, no?

It is one block operation under the same key, rather than a different key
so I suppose there are minuscule cache footprint wins (one symmetric key is
16 or 32 bytes). But the cost is per-packet (!) bandwidth and complexity. I
do not see how this could be worth it. Note that, for AES, many CPUs have
dedicated instructions for it already.

> Clear separation of concerns results if we don’t overload the packet
number as the nonce. Connection migration and multi-path are optional
features and we pay no extra compute cost for most connections that are not
impacted by unlnkability. Nonce field becomes optional for crypto that
doesn’t need it and can grow to larger size as needed for bulk transfers to
prevent frequent key rollovers (say in datacenter scenarios).

You're not going to find AEADs that don't require a nonce. Encryption must
not produce the same ciphertext twice for the same plaintext, so you need
some input to diversify them. That input is the nonce. Nonces also are not
variable-length, so you don't get to arbitrarily grow them. (AES-GCM per
spec is variable length, but that's a spec bug. If you size it above 96
bits, it's as if you randomized it and your nonce space crashes per
birthday.)

It is also harmful to separate the two concerns. When you are building a
transport, derive the nonce from a sequence number. TLS 1.2 AES-GCM
separated them, which was a mistake. It's less efficient (the world wasted
quite a lot of bandwidth due to this), and more error-prone. There was
nothing stopping implementations from sending the same nonce or generating
nonces randomly (the AEADs we use don't have large enough nonces for that),
and indeed some did just that. If you use some sort of sequence number, you
can lean on existing work both to filter out the obvious implementation
mistakes and to avoid sending unnecessary data.

On Fri, Feb 9, 2018 at 12:38 PM Praveen Balasubramanian <[email-masked]>
wrote:

>    1. Packet number is in the encrypted portion of the header and is no
>    longer used as the nonce. It is encoded as a variable length number. Always
>    starts at 0.
>       1. No ossification of PN
>       2. No linkability using PN
>       3. Allows for growth beyond 64 bits if needed in future
>    2. The nonce is in the clear in plaintext header. It is generated *per
>    path* as a secure random number. Call this the “starting value”. Both
>    sides MUST initialize the “starting value” nonce *per path*. Within a
>    path it grows monotonically (to avoid repeats just like PN today) and can
>    include short random jumps periodically so that middleboxes cannot assume
>    anything about the field. Since nonce has nothing to do with PN the jumps
>    don’t add any complexity to transport logic. Jumps are for greasing and
>    preventing ossification. When the nonce rolls over and reaches close to the
>    “starting value”, key rollover must happen. At high data rates the nonce
>    field can be 64-bits to prevent frequent key rollover. For crypto that
>    doesn’t need nonces, the field can be omitted.
>       1. No ossification because there is no pre-determined sequence to
>       the nonce.
>       2. In gQUIC and the RFC as defined the nonce is a simple sequence
>       that increments by one, so this is no worse off in terms of security. It is
>       non-repeating.
>       3. No linkability because different paths start at a completely
>       different random number. If repeating is a concern then different paths use
>       different keys. Carries forward to not just connection migration but
>       mutli-path.
>       4. Supports key rollover just like gQUIC and currently defined
>       draft RFC.
>       5. Allows for omitting the nonce when we use crypto that doesn’t
>       need it.
>       6. Allows for larger nonces for high data rates.
>       7. It doesn’t preclude any other type of non-repeating nonces
>       without having to modify the RFC! So implementations that want to use
>       cryptographic nonces can still do so.
>
>
>
> The downsides:
>
>    1. Even with variable length encoding the extra PN field is more space
>    used per packet when a nonce is required. I think we can do with 2 bytes in
>    version 1.
>    2. It’s not perfect unlinkabiltiy but I have yet to see any solution
>    that can provide this. A simple time based analysis using machine learning
>    will allow flow correlation especially for connection migration. The IP
>    addresses in the clear allow for a high degree of correlation. There is
>    also the application sending pattern and other signals that could be used.
>
>
>
> *From:* Eric Rescorla [mailto:[email-masked]]
> *Sent:* Friday, February 9, 2018 5:52 AM
> *Cc:* huitema <[email-masked]>; [email-masked]
>
>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 19:58:38 +0000

With VLE its minimal overhead per packet so you are not encrypting 4 bytes more all the time even with 64 bit or 128 bit packet number . So its decidedly better. Also fewer calls into TLS, but I suppose one could do some refactoring of TLS API. Many CPUs yes but this is a general purpose transport and could run on CPUs that don’t.

Didn’t ask for nonce to be variable length. Fixed length for connection lifetime. Datacenters connections could use 64-bit nonce to avoid frequent key rollover.

Effectively the plain text nonce is a sequence number since it increases monotonically with small jumps as needed to avoid ossification. So no worse off than gQUIC or current draft. No more complex than using the packet number. Just the starting number is randomized. Also please notice that this proposal does not preclude using encrypted packet number as the nonce as proposed previously. Changing how the nonce works makes 0 impact on the transport logic. An implementation can choose to implement any form of non-repeating nonce without revving the RFC. This is the biggest advantage of separation of concerns IMHO.

From: David Benjamin [mailto:[email-masked]]
Sent: Friday, February 9, 2018 11:39 AM
To: Praveen Balasubramanian <[email-masked]>
Cc: Eric Rescorla <[email-masked]>; [email-masked]; huitema <[email-masked]>
Subject: Re: Packet number encryption

Your primary worry here was the extra block cipher operation, right? By moving the packet number into the plaintext, this proposal increases the length, which will cost that same one extra block cipher operation, no?

It is one block operation under the same key, rather than a different key so I suppose there are minuscule cache footprint wins (one symmetric key is 16 or 32 bytes). But the cost is per-packet (!) bandwidth and complexity. I do not see how this could be worth it. Note that, for AES, many CPUs have dedicated instructions for it already.

> Clear separation of concerns results if we don’t overload the packet number as the nonce. Connection migration and multi-path are optional features and we pay no extra compute cost for most connections that are not impacted by unlnkability. Nonce field becomes optional for crypto that doesn’t need it and can grow to larger size as needed for bulk transfers to prevent frequent key rollovers (say in datacenter scenarios).

You're not going to find AEADs that don't require a nonce. Encryption must not produce the same ciphertext twice for the same plaintext, so you need some input to diversify them. That input is the nonce. Nonces also are not variable-length, so you don't get to arbitrarily grow them. (AES-GCM per spec is variable length, but that's a spec bug. If you size it above 96 bits, it's as if you randomized it and your nonce space crashes per birthday.)

It is also harmful to separate the two concerns. When you are building a transport, derive the nonce from a sequence number. TLS 1.2 AES-GCM separated them, which was a mistake. It's less efficient (the world wasted quite a lot of bandwidth due to this), and more error-prone. There was nothing stopping implementations from sending the same nonce or generating nonces randomly (the AEADs we use don't have large enough nonces for that), and indeed some did just that. If you use some sort of sequence number, you can lean on existing work both to filter out the obvious implementation mistakes and to avoid sending unnecessary data.

On Fri, Feb 9, 2018 at 12:38 PM Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>> wrote:
Let me try to summarize the proposal again:

     *   No ossification of PN

  1.  The nonce is in the clear in plaintext header. It is generated per path as a secure random number. Call this the “starting value”. Both sides MUST initialize the “starting value” nonce per path. Within a path it grows monotonically (to avoid repeats just like PN today) and can include short random jumps periodically so that middleboxes cannot assume anything about the field. Since nonce has nothing to do with PN the jumps don’t add any complexity to transport logic. Jumps are for greasing and preventing ossification. When the nonce rolls over and reaches close to the “starting value”, key rollover must happen. At high data rates the nonce field can be 64-bits to prevent frequent key rollover. For crypto that doesn’t need nonces, the field can be omitted.

The downsides:

From: Eric Rescorla [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Friday, February 9, 2018 5:52 AM
To: Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>
Cc: huitema <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>

Can we:

-Ekr

-- Christian Huitema

---

From: "Deval, Manasi" <[email-masked]>
Date: Fri, 9 Feb 2018 20:55:24 +0000

Going back to Ian’s point here – Can you be explicit about the issue you see in the packet number encryption?

Thanks,
Manasi

From: QUIC [mailto:[email-masked]] On Behalf Of Praveen Balasubramanian
Sent: Friday, February 09, 2018 11:59 AM
To: David Benjamin <[email-masked]>
Cc: Eric Rescorla <[email-masked]>; [email-masked]; huitema <[email-masked]>
Subject: RE: Packet number encryption

     *   No ossification of PN

The downsides:

Can we:

-Ekr

-- Christian Huitema

---

From: "Deval, Manasi" <[email-masked]>
Date: Fri, 9 Feb 2018 21:16:35 +0000

Hi All,

The issue with TCP crypto offload does not directly apply to the QUIC Crypto offload. For example, received TCP frames need to be coalesced before decryption. This is a prohibitive operation to achieve in hardware but much simpler to achieve in the software. The QUIC data encryption and header update is therefore achievable.

Encrypting the packet number complicates the crypto offload, though. Is there another way to achieve the same security? In some early tests with the existing QUIC stacks, I see that the crypto cost is quite significant so high speed deployments may find use of crypto offload.

Thanks,

Manasi

From: QUIC [mailto:[email-masked]] On Behalf Of Deval, Manasi
Sent: Friday, February 09, 2018 12:55 PM
To: Praveen Balasubramanian <[email-masked]>; David Benjamin <[email-masked]>
Cc: Eric Rescorla <[email-masked]>; [email-masked]; huitema <[email-masked]>
Subject: RE: Packet number encryption

Thanks,
Manasi

From: QUIC [mailto:[email-masked]] On Behalf Of Praveen Balasubramanian
Sent: Friday, February 09, 2018 11:59 AM
To: David Benjamin <[email-masked]<mailto:[email-masked]>>
Cc: Eric Rescorla <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; huitema <[email-masked]<mailto:[email-masked]>>
Subject: RE: Packet number encryption

Cc: Eric Rescorla <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; huitema <[email-masked]<mailto:[email-masked]>>

     *   No ossification of PN

The downsides:

Can we:

-Ekr

-- Christian Huitema

---

From: Victor Vasiliev <[email-masked]>
Date: Fri, 9 Feb 2018 16:17:10 -0500

On Thu, Feb 8, 2018 at 5:32 PM, Praveen Balasubramanian <[email-masked]

> AFAICT the goal is to perform some transform for preventing middlebox
> ossification. Encryption is one possible transform but is it the only
> possible one?
>

The goal of the transform is to prevent the middleboxes from interpreting
the packet number, so yes, anything you'll propose would be encryption.

> The numbers I quoted were from a quick test by modifying the winquic
> implementation to add an additional encrypt / decrypt step on packet number
> size worth of data. If anyone else has any data please share.
>

I am very confused about the 1.25% number.  1280 bytes is about 80 AES
blocks, and one extra block is 1.25% on top of total *encryption* costs
(you don't have to MAC the PN, so that's even cheaper).  And the encryption
itself is fairly cheap compared to almost any other part of the QUIC stack
(sent packet tracking, ack processing, I/O, timers, etc).

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 9 Feb 2018 13:18:18 -0800

The obvious advantage of a clear text nonce is that it can be immediately
decrypted without extra processing and without modifying the packet
content, and it avoids latency from a separate packet number decryption
step.

However, having per path unique nonces per path jumping randomly in order
to gain an insignificant amount of anti-ossification is not every
appealing. Neither is the extra space required for having both packet
numbers and nonces, when they in the majority of cases is one and the same,
effectively. The random jumps are effectively a hidden encryption step
although cheaper than packet number encryption.

So, I propose having a nonce that counts from 0 per path per direction and
also having a unique connection identifier per path direction (asymmetric
CID). Lets call this a half-path context (CID+nonce counter). It could also
work shared path CID as today, but it is not as clean.

The nonce can at most count to 2^32-1 due to hard limits on AES-GCM and
possibly general TLS requirements.

This leaves the packet number encoding:

If we enumerate each half-path with a path ID PID as a simpler counter
starting at zero, we have a 1-1 mapping from PID to CID. The packet number
PN is then encoded as <PID * 2^32 + nonce counter>. The PN is used
explicitly and implicitly in ACK frames but is never carried directly on
the wire.

During path migration packet numbers in the same ACK frame may have large
predictable jumps, but the ACK frame already competently deals with this,
and the jumps are limited to the number of concurrently active paths.

The only loss is the ossification concern of having predictable per path
counters, but I think that is low price of admission. There are no
linkability concerns with this, and all the networking monitoring people
would be happy too.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 9 February 2018 at 21.55.33, Deval, Manasi ([email-masked])
wrote:

Going back to Ian’s point here – Can you be explicit about the issue you
see in the packet number encryption?

Thanks,

Manasi

*From:* QUIC [mailto:[email-masked]] *On Behalf Of *Praveen
Balasubramanian
*Sent:* Friday, February 09, 2018 11:59 AM
*To:* David Benjamin <[email-masked]>
*Cc:* Eric Rescorla <[email-masked]>; [email-masked]; huitema <
[email-masked]>
*Subject:* RE: Packet number encryption

With VLE its minimal overhead per packet so you are not encrypting 4 bytes
more all the time even with 64 bit or 128 bit packet number . So its
decidedly better. Also fewer calls into TLS, but I suppose one could do
some refactoring of TLS API. Many CPUs yes but this is a general purpose
transport and could run on CPUs that don’t.

Didn’t ask for nonce to be variable length. Fixed length for connection
lifetime. Datacenters connections could use 64-bit nonce to avoid frequent
key rollover.

Effectively the plain text nonce is a sequence number since it increases
monotonically with small jumps as needed to avoid ossification. So no worse
off than gQUIC or current draft. No more complex than using the packet
number. Just the starting number is randomized. Also please notice that
this proposal does not preclude using encrypted packet number as the nonce
as proposed previously. Changing how the nonce works makes 0 impact on the
transport logic. An implementation can choose to implement any form of
non-repeating nonce without revving the RFC. This is the biggest advantage
of separation of concerns IMHO.

*From:* David Benjamin [mailto:[email-masked]]
*Sent:* Friday, February 9, 2018 11:39 AM
*To:* Praveen Balasubramanian <[email-masked]>
*Cc:* Eric Rescorla <[email-masked]>; [email-masked]; huitema <
[email-masked]>
*Subject:* Re: Packet number encryption

   1. No ossification of PN

   1. The nonce is in the clear in plaintext header. It is generated *per

   1. No ossification because there is no pre-determined sequence to the
      nonce.
      2. In gQUIC and the RFC as defined the nonce is a simple sequence
      that increments by one, so this is no worse off in terms of
security. It is
      non-repeating.
      3. No linkability because different paths start at a completely
      different random number. If repeating is a concern then
different paths use
      different keys. Carries forward to not just connection migration but
      mutli-path.
      4. Supports key rollover just like gQUIC and currently defined draft
      RFC.
      5. Allows for omitting the nonce when we use crypto that doesn’t need
      it.
      6. Allows for larger nonces for high data rates.
      7. It doesn’t preclude any other type of non-repeating nonces without
      having to modify the RFC! So implementations that want to use
cryptographic
      nonces can still do so.

The downsides:

On Thu, Feb 8, 2018 at 9:40 PM, Praveen Balasubramanian <[email-masked]>
wrote:

Can we:

-Ekr

-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 21:20:03 +0000

Please ignore the previous proposal and look at the new proposal where PN is encrypted and decoupled from the nonce. I am abandoning the previous proposal that was trying to use obfuscation of PN in cleartext.

From: Victor Vasiliev [mailto:[email-masked]]
Sent: Friday, February 9, 2018 1:17 PM
To: Praveen Balasubramanian <[email-masked]>
Cc: Marten Seemann <[email-masked]>; [email-masked]; huitema <[email-masked]>
Subject: Re: Packet number encryption

On Thu, Feb 8, 2018 at 5:32 PM, Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>> wrote:
AFAICT the goal is to perform some transform for preventing middlebox ossification. Encryption is one possible transform but is it the only possible one?

The goal of the transform is to prevent the middleboxes from interpreting the packet number, so yes, anything you'll propose would be encryption.

The numbers I quoted were from a quick test by modifying the winquic implementation to add an additional encrypt / decrypt step on packet number size worth of data. If anyone else has any data please share.

I am very confused about the 1.25% number.  1280 bytes is about 80 AES blocks, and one extra block is 1.25% on top of total *encryption* costs (you don't have to MAC the PN, so that's even cheaper).  And the encryption itself is fairly cheap compared to almost any other part of the QUIC stack (sent packet tracking, ack processing, I/O, timers, etc).

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 9 Feb 2018 13:27:38 -0800

This is for the large file upload case. It does not consider low latency
applications with small messages and separate cores for different
responsibilities. It also does not consider that that a single block costs
the same as 4 blocks on Intel, nor does it consider that the packet cannot
be decoded in-place. But yes, in the common large file download case, the
cost is not very significant. But it can be significant enough to choose
another protocol in other scenarios. So there needs to be a really good
reason to add this extra latency step. Linkability might be an argument,
and ossification might be too, and the complexity of alternatives might as
well. However, I’m not convinced. I’m also not completely dismissing it,
but I would like to see alternatives explored more thoroughly before
throwing in a cost for no good reason.

---

From: Ian Swett <[email-masked]>
Date: Fri, 9 Feb 2018 16:46:28 -0500

I should say that I don't have a specific API, implementation, or NIC in
mind, so hopefully someone(Praveen?) can provide some context about what is
possible with existing APIs or what is relatively easy to add to them.

The API I had in mind was of the "encrypt this byte string" form, but in
the NIC.  ie: When I write a sequence of packets I supply crypto info, as
well as a minimum release time(or pacing rate).  The goal is to do the bulk
crypto as the network stack is packetizing and copying into the NIC
memory.  Ideally this can avoid an extra memcpy and avoids touching the
data to be sent until the last possible moment.

Hardware acceleration of AES in modern CPUs is extremely good, so another
approach is to do the above in a userspace networking stack or the kernel
and use the CPU's AES acceleration.  If the networking code that was
writing into NIC memory had a deep understanding of QUIC, possibly it could
do the packet number encryption as a second pass?

On Fri, Feb 9, 2018 at 3:17 PM, Christian Huitema <[email-masked]>
wrote:

> On 2/9/2018 8:20 AM, Ian Swett wrote:
>
> > One issue with the current encryption proposal is you can't use
> > hardware crypto offload for the bulk encryption because the packet
> > number has to be encrypted after the bulk data.
> >
> Ian, could you be a bit more explicit about the type of hardware offload
> that you have in mind?
>
> I can think of two extreme types. One is embedding crypto offload in the
> NIC, and is generally tied to TCP/TLS/TLO. That one type has no hope
> whatsoever of working with UDP/QUIC. The other type is simply providing
> an "encrypt this byte string" API, and can be trivially used with QUIC
> -- much like we can use Intel's AES extended instructions today. So
> obviously you have something else in mind, and maybe you should explain it.
>
> -- Christian Huitema
>
>
>
>

---

From: Ian Swett <[email-masked]>
Date: Fri, 9 Feb 2018 16:53:24 -0500

Good question.  I'm still trying to understand how crypto offload might
work for QUIC, but I tried to clarify what I was thinking on the side
thread Christian started.  But yes, the two step encryption was the
concern, particularly the need to use the results of the bulk crypto when
encrypting the packet number.

On Fri, Feb 9, 2018 at 4:16 PM, Deval, Manasi <[email-masked]>
wrote:

> Hi All,
>
>
>
> The issue with TCP crypto offload does not directly apply to the QUIC
> Crypto offload. For example, received TCP frames need to be coalesced
> before decryption. This is a prohibitive operation to achieve in hardware
> but much simpler to achieve in the software. The QUIC data encryption and
> header update is therefore achievable.
>
>
>
> Encrypting the packet number complicates the crypto offload, though. Is
> there another way to achieve the same security? In some early tests with
> the existing QUIC stacks, I see that the crypto cost is quite significant
> so high speed deployments may find use of crypto offload.
>
>
>
> Thanks,
>
> Manasi
>
>
>
>
>
> *From:* QUIC [mailto:[email-masked]] *On Behalf Of *Deval, Manasi
> *Sent:* Friday, February 09, 2018 12:55 PM
> *To:* Praveen Balasubramanian <[email-masked]>; David Benjamin <
> [email-masked]>
>
> *From:* David Benjamin [mailto:[email-masked]
> <[email-masked]>]
> On Fri, Feb 9, 2018 at 12:38 PM Praveen Balasubramanian <

---

From: Martin Thomson <[email-masked]>
Date: Sat, 10 Feb 2018 08:54:14 +1100

> encrypted after the bulk data.

Why?  Are you expecting the hardware to take the packet and run away
with it?  (I know that happens, but is that really a requirement?)

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 9 Feb 2018 13:58:38 -0800

I'm still trying to understand how crypto offload might work for QUIC,

You could start looking at the netmap API. It exposes a vector of fixed
size buffers than can originate directly from the NIC buffer memory. The
same API can also be used to offload work by other CPU cores because you
essentially commit a buffer and leave it for the next stage in a pipeline.

https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 23:13:05 +0000

Completely agree. In our current implementation crypto is a huge cost (we see a sizeable perf difference between a plaintext (test only) mode versus not) and we expect that to get even more pronounced as the main bottleneck as we continue to improve UDP perf.

From: Mikkel Fahnøe Jørgensen [mailto:[email-masked]]
Sent: Friday, February 9, 2018 1:28 PM
To: Victor Vasiliev <[email-masked]>; Praveen Balasubramanian <[email-masked]>
Cc: [email-masked]; huitema <[email-masked]>; Marten Seemann <[email-masked]>
Subject: Re: Packet number encryption

This is for the large file upload case. It does not consider low latency applications with small messages and separate cores for different responsibilities. It also does not consider that that a single block costs the same as 4 blocks on Intel, nor does it consider that the packet cannot be decoded in-place. But yes, in the common large file download case, the cost is not very significant. But it can be significant enough to choose another protocol in other scenarios. So there needs to be a really good reason to add this extra latency step. Linkability might be an argument, and ossification might be too, and the complexity of alternatives might as well. However, I’m not convinced. I’m also not completely dismissing it, but I would like to see alternatives explored more thoroughly before throwing in a cost for no good reason.

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 9 Feb 2018 23:16:41 +0000

But plaintext/encryption isn’t the real comparison, it’s encrypting A vs A-n, right?

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 23:33:29 +0000

Yep in context of the proposals yes and that is at about 1%+ (based on a quick test) and might get more pronounced when other bottlenecks disappear. I was merely commenting that crypto is already a large fraction of overall processing cost and will only become a larger factor with other improvements.

From: Salz, Rich [mailto:[email-masked]]
Sent: Friday, February 9, 2018 3:17 PM
To: Praveen Balasubramanian <[email-masked]>; Mikkel Fahnøe Jørgensen <[email-masked]>; Victor Vasiliev <[email-masked]>
Cc: Marten Seemann <[email-masked]>; [email-masked]; huitema <[email-masked]>
Subject: Re: Packet number encryption

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Feb 2018 23:35:29 +0000

We have precedence in Windows with IPsec task offload which is symmetric crypto offload. The goal is to not have to make separate API calls to process crypto. The main advantage of crypto offload results from being able to make it a inline step in the NIC packet processing engine as part of the data path. This allows you to offload more mundane packet processing tasks going forward like we do with TCP. Some programmable NICs might be able to do a second pass but I am not a hardware expert to say if all NICs will be able to. There is precedence of some form of hardware crypto offload for TCP/TLS both as a side function and inline in network card. Some links from a quick web search:
https://www.chelsio.com/crypto-offload/
https://aws.amazon.com/documentation/cloudhsm/
https://netdevconf.org/1.2/session.html?boris-pismenny
I cannot comment on whether requiring an extra pass will render existing hardware inapplicable for QUIC. I have been trying hard to see if we can avoid an extra step regardless of crypto offload.

From: Ian Swett [mailto:[email-masked]]
Sent: Friday, February 9, 2018 1:46 PM
To: huitema <[email-masked]>
Cc: Salz, Rich <[email-masked]>; Praveen Balasubramanian <[email-masked]>; Eric Rescorla <[email-masked]>; [email-masked]
Subject: Re: hardware offload (was: Packet number encryption)

I should say that I don't have a specific API, implementation, or NIC in mind, so hopefully someone(Praveen?) can provide some context about what is possible with existing APIs or what is relatively easy to add to them.

The API I had in mind was of the "encrypt this byte string" form, but in the NIC.  ie: When I write a sequence of packets I supply crypto info, as well as a minimum release time(or pacing rate).  The goal is to do the bulk crypto as the network stack is packetizing and copying into the NIC memory.  Ideally this can avoid an extra memcpy and avoids touching the data to be sent until the last possible moment.

Hardware acceleration of AES in modern CPUs is extremely good, so another approach is to do the above in a userspace networking stack or the kernel and use the CPU's AES acceleration.  If the networking code that was writing into NIC memory had a deep understanding of QUIC, possibly it could do the packet number encryption as a second pass?

On Fri, Feb 9, 2018 at 3:17 PM, Christian Huitema <[email-masked]<mailto:[email-masked]>> wrote:

-- Christian Huitema

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 9 Feb 2018 23:36:58 +0000

> Clear separation of concerns results if we don’t overload the packet number as the nonce. Connection migration and multi-path are optional features and we pay no extra compute cost for most connect…

David already replied pretty definitively, but for those not on his level …

It seems highly unlikely that we will want to use non-authenticated crypto – ie., AEAD or similar.  TLS 1.3 is all and only authenticated encryption.  The difference in cost is a few extra bytes of encryption, as opposed to bigger packets for plaintext and then encrypting the nonce, right?  That difference should be hard to notice.

If we assume it’s all authenticated crypto, and therefore we need a nonce, then separation of concerns becomes a non-issue. In fact, as David pointed out, a packet number *exactly* meets the requirements of an authenticated encrypted nonce. Not using it for that purpose is being wasteful for absolutely no good reason.

Do you have a non-authenticated crypto mechanism we should consider?

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 9 Feb 2018 15:56:14 -0800

On 9 February 2018 at 22.18.18, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Using a segment packet number space can also be used to deal with
optimistic ACK attacks:

Each path is given two connection ID’s which may very just at the lower
bit. Rather than randomly skipping some packet numbers, a few packets could
be sent on the alternative CID. This will introduce gaps but each nonce
range remains gapless which keeps the ACK frame defragmented and avoids
false loss signals. It does required that CID is present on the wire, but
if it is not, a single header bit could be used to imply which of packet
number segments is in use. On its own this is not a very attractive ack
attack solution, but it may be relevant if packet numbers are segmented,
and especially with a v2 multi-path solution.

---

From: Victor Vasiliev <[email-masked]>
Date: Fri, 9 Feb 2018 20:01:37 -0500

Large download case is the case in which the percent of time spent on
symmetric crypto is the greatest.  As your packets become smaller,
non-cryptographic overheads of sending a packet become bigger.

---

From: Christian Huitema <[email-masked]>
Date: Fri, 9 Feb 2018 10:17:52 -1000

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sat, 10 Feb 2018 05:18:20 +0000

98ns for 68 byte messages

https://www.solarflare.com/solarflare-lda-penguin-unleash-ultimate-trading-machine-stac-t0-benchmark

________________________________
From: Victor Vasiliev <[email-masked]>
Sent: Saturday, February 10, 2018 2:01:37 AM
To: Mikkel Fahnøe Jørgensen
Cc: Praveen Balasubramanian; [email-masked]; huitema; Marten Seemann
Subject: Re: Packet number encryption

Large download case is the case in which the percent of time spent on symmetric crypto is the greatest.  As your packets become smaller, non-cryptographic overheads of sending a packet become bigger.

On Fri, Feb 9, 2018 at 4:27 PM, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:
I am very confused about the 1.25% number.  1280 bytes is about 80 AES blocks, and one extra block is 1.25% on top of total *encryption* costs (you don't have to MAC the PN, so that's even cheaper).  And the encryption itself is fairly cheap compared to almost any other part of the QUIC stack (sent packet tracking, ack processing, I/O, timers, etc).

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sat, 10 Feb 2018 03:35:45 -0800

To put numbers into perspective using Intel 2015 data

https://software.intel.com/en-us/articles/improving-openssl-performance#_Toc416943490

A 64 byte message in AES-GCM AEAD in HW would use 1.03 cycles per byte or
66 cycles total, or 22ns on a 3GHz core.

For packet numbers we use the CBC encrypt numbers because here AES cannot
exploit block parallelism.
Here we see 4.44 cycles/byte in HW or 71 cycles per block. With a 3GHz
setup that would amount to about 24ns overhead for packet encryption.

Clearly it makes no sense that AES-GCM is faster than a single AES block
encryption, but these are only approximate numbers and CBC mode might have
a little overhead, so we clamp packet numbers to 22ns.

Taking the 98ns overhead by the Solarflare report we get a total
(simplified) processing time is 98ns non-crypto, 22ns for packet number,
and 22ns for AEAD totalling 142ns. So the packet number encryption overhead
would be 22/(98+22)*100% = 18%. The numbers ignore other QUIC bookkeeping,
but that can be done in other cores or outside the latency critical window.

This does not take into account that AEAD operation may operate less than
optimal because the packet number must be extracted first. On the other
hand, it is also not a disastrous overhead if no good alternative can be
found.

Earlier AES-NI I’ve seen from Intel doc suggests around 100cycles in HW for
a single AES-128 block which would be 33ns per packet number in the above
example.

On 10 February 2018 at 06.18.25, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

---

From: "Eggert, Lars" <[email-masked]>
Date: Sat, 10 Feb 2018 14:25:39 +0000

> Hardware acceleration of AES in modern CPUs is extremely good, so another approach is to do the above in a userspace networking stack or the kernel and use the CPU's AES acceleration.  If the netwo…

FWIW, this is exactly what I intend on doing with quant, which can sit on top of netmap. At the moment, I do the crypto on the same core as anything else (so I can run-to-completion w/o locks), but eventually I want to try to push things over to other cores for crypto.

I'm also chatting with the Cambridge NetFPGA people to see what type of crypto (and segmentation offload they could provide that would tie in well with a QUIC stack.

Lars

---

From: Praveen Balasubramanian <[email-masked]>
Date: Sat, 10 Feb 2018 17:12:50 +0000

Makes sense. The 1%+ overhead I had quoted was in comparison to full protocol processing all the way from app to NIC. This is with an unoptimized early QUIC implementation and not fully optimized UDP/IP stack (we have focused much much more on optimizing TCP in the past). After software stack optimizations, the crypto share of the cost will only keep going up for which we have no technique other than to offload when such support is available in hardware and offload is way more challenging when workload is hosted in VMs and containers.

From: Mikkel Fahnøe Jørgensen [mailto:[email-masked]]
Sent: Saturday, February 10, 2018 3:36 AM
To: Victor Vasiliev <[email-masked]>
Cc: Praveen Balasubramanian <[email-masked]>; [email-masked]; Marten Seemann <[email-masked]>; huitema <[email-masked]>
Subject: Re: Packet number encryption

https://software.intel.com/en-us/articles/improving-openssl-performance#_Toc416943490<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsoftware.intel.com%2Fen-us%2Farticles%2Fimproving-openssl-performance%23_Toc416943490&data=04%7C01%7Cpravb%40microsoft.com%7C947ea96d592c498130df08d5707a6db1%7Cee3303d7fb734b0c8589bcd847f1c277%7C1%7C0%7C636538593507129957%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-1&sdata=I1%2BnvvFBfsQ1P%2BwSI7riWr%2FkP9ydkAihxlpAgpFggEg%3D&reserved=0>

A 64 byte message in AES-GCM AEAD in HW would use 1.03 cycles per byte or 66 cycles total, or 22ns on a 3GHz core.

For packet numbers we use the CBC encrypt numbers because here AES cannot exploit block parallelism.
Here we see 4.44 cycles/byte in HW or 71 cycles per block. With a 3GHz setup that would amount to about 24ns overhead for packet encryption.

Clearly it makes no sense that AES-GCM is faster than a single AES block encryption, but these are only approximate numbers and CBC mode might have a little overhead, so we clamp packet numbers to 22ns.

Taking the 98ns overhead by the Solarflare report we get a total (simplified) processing time is 98ns non-crypto, 22ns for packet number, and 22ns for AEAD totalling 142ns. So the packet number encryption overhead would be 22/(98+22)*100% = 18%. The numbers ignore other QUIC bookkeeping, but that can be done in other cores or outside the latency critical window.

This does not take into account that AEAD operation may operate less than optimal because the packet number must be extracted first. On the other hand, it is also not a disastrous overhead if no good alternative can be found.

Earlier AES-NI I’ve seen from Intel doc suggests around 100cycles in HW for a single AES-128 block which would be 33ns per packet number in the above example.

On 10 February 2018 at 06.18.25, Mikkel Fahnøe Jørgensen ([email-masked]<mailto:[email-masked]>) wrote:
98ns for 68 byte messages

---

From: Praveen Balasubramanian <[email-masked]>
Date: Sat, 10 Feb 2018 17:18:59 +0000

>> I want to try to push things over to other cores for crypto
There might be context switch costs and cache misses. I don't know if this will be better than just doing it inline on current core. Run to completion makes for easier to maintain implementation and no locks is great. Plus the overall CPU cost of network stack processing will remain the same and maybe even go up a tad bit.

From: Eggert, Lars [mailto:[email-masked]]
Sent: Saturday, February 10, 2018 6:26 AM
To: Ian Swett <[email-masked]>
Cc: huitema <[email-masked]>; Salz, Rich <[email-masked]>; Eric Rescorla <[email-masked]>; [email-masked]; Praveen Balasubramanian <[email-masked]>
Subject: Re: hardware offload (was: Packet number encryption)

On 2018-2-9, at 13:46, Ian Swett <[email-masked]<mailto:[email-masked]>> wrote:
Hardware acceleration of AES in modern CPUs is extremely good, so another approach is to do the above in a userspace networking stack or the kernel and use the CPU's AES acceleration.  If the networking code that was writing into NIC memory had a deep understanding of QUIC, possibly it could do the packet number encryption as a second pass?

Lars

---

From: Praveen Balasubramanian <[email-masked]>
Date: Sat, 10 Feb 2018 18:22:42 +0000

>> a packet number *exactly* meets the requirements of an authenticated encrypted nonce. Not using it for that purpose is being wasteful for absolutely no good reason.
For better or worse we have chosen to tie the transport very tightly with TLS. This means extra work will be needed for unencrypted QUIC or QUIC over IPsec for example. Case in point transport parameters not being in the transport header but rather in the TLS header. In HTTPS over TCP, TLS will use its own sequence number for nonce and TCP will have its own sequence number. This is a clear separation of concerns. From what I can tell, since QUIC uses its own packet framing and the PN doesn’t repeat we have tried to optimize this by overloading it with the nonce.

That said nothing prevents an implementation from using the packet number as the nonce. By separating concerns one can pick whatever one wants for the nonce, the receiver doesn’t care about the nonce value other than for decryption i.e. no impact on transport logic. We can use a VLE field or other methods to reduce the extra space needed for the PN.

>> and then encrypting the nonce
You don’t have to but you can . The only functional requirement for the nonce is non-repeatability. Encryption is only being considered for ossification and unlinkability concerns. For connections that don’t care about unlinkability (no migration, no multi path) the overhead is not worth it. Ossification can be avoided by a random initial value and jumps in between since this is a form of greasing.

>> Do you have a non-authenticated crypto mechanism we should consider?
I am not a crypto expert. I don’t know what future crypto will look like. Making the nonce field optional or larger size when needed (but not variable for given connection) allows the transport protocol to be more future proof. Forcing the PN to be the nonce makes things more inflexible. I care much less about the nonce getting ossified compared to the PN.

From: Salz, Rich [mailto:[email-masked]]
Sent: Friday, February 9, 2018 3:37 PM
To: Praveen Balasubramanian <[email-masked]>
Cc: [email-masked]
Subject: Re: Packet number encryption

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sat, 10 Feb 2018 13:44:38 -0600

I wrote up my previous suggestion to use segmented packet numbers with
clear text nonce in an issue, and I added a comment where I propose a
variation that encrypts the nonce, but this encryption is 16 times faster
than the current packet number encryption scheme. In both cases there is no
duplication of nonce and packet numbers because they are the same modulo
path.

https://github.com/quicwg/base-drafts/issues/1105

On 10 February 2018 at 19.22.50, Praveen Balasubramanian (
[email-masked]) wrote:

>> a packet number **exactly** meets the requirements of an authenticated
encrypted nonce. Not using it for that purpose is being wasteful for
absolutely no good reason.

For better or worse we have chosen to tie the transport very tightly with
TLS. This means extra work will be needed for unencrypted QUIC or QUIC over
IPsec for example. Case in point transport parameters not being in the
transport header but rather in the TLS header. In HTTPS over TCP, TLS will
use its own sequence number for nonce and TCP will have its own sequence
number. This is a clear separation of concerns. From what I can tell, since
QUIC uses its own packet framing and the PN doesn’t repeat we have tried to
optimize this by overloading it with the nonce.

That said nothing prevents an implementation from using the packet number
as the nonce. By separating concerns one can pick whatever one wants for
the nonce, the receiver doesn’t care about the nonce value other than for
decryption i.e. no impact on transport logic. We can use a VLE field or
other methods to reduce the extra space needed for the PN.

>> and then encrypting the nonce

You don’t have to but you can . The only *functional* requirement for the
nonce is non-repeatability. Encryption is only being considered for
ossification and unlinkability concerns. For connections that don’t care
about unlinkability (no migration, no multi path) the overhead is not worth
it. Ossification can be avoided by a random initial value and jumps in
between since this is a form of greasing.

>> Do you have a non-authenticated crypto mechanism we should consider?

I am not a crypto expert. I don’t know what future crypto will look like.
Making the nonce field optional or larger size when needed (but not
variable for given connection) allows the transport protocol to be more
future proof. Forcing the PN to be the nonce makes things more inflexible.
I care much less about the nonce getting ossified compared to the PN.

*From:* Salz, Rich [mailto:[email-masked]]
*Sent:* Friday, February 9, 2018 3:37 PM
*To:* Praveen Balasubramanian <[email-masked]>
*Cc:* [email-masked]
*Subject:* Re: Packet number encryption

It seems highly unlikely that we will want to use non-authenticated crypto
– ie., AEAD or similar.  TLS 1.3 is all and only authenticated encryption.
The difference in cost is a few extra bytes of encryption, as opposed to
bigger packets for plaintext and then encrypting the nonce, right?  That
difference should be hard to notice.

If we assume it’s all authenticated crypto, and therefore we need a nonce,
then separation of concerns becomes a non-issue. In fact, as David pointed
out, a packet number **exactly** meets the requirements of an authenticated
encrypted nonce. Not using it for that purpose is being wasteful for
absolutely no good reason.

---

From: Victor Vasiliev <[email-masked]>
Date: Mon, 12 Feb 2018 02:19:06 -0500

I have no idea how you get that number (by which I mean, I have a lot of
guesses, but none of which are solid or useful here).  I looked at the
profiles of various workloads we're running in production, and I estimate
that none of them would be impacted by PN encryption by worse than 0.1%.

On Sat, Feb 10, 2018 at 12:12 PM, Praveen Balasubramanian <
[email-masked]> wrote:

> Makes sense. The 1%+ overhead I had quoted was in comparison to *full
> protocol processing all the way from app to NIC*. This is with an
> unoptimized early QUIC implementation and not fully optimized UDP/IP stack
> (we have focused much much more on optimizing TCP in the past). After
> software stack optimizations, the crypto share of the cost will only keep
> going up for which we have no technique other than to offload when such
> support is available in hardware and offload is way more challenging when
> workload is hosted in VMs and containers.
>
>
>
> *From:* Mikkel Fahnøe Jørgensen [mailto:[email-masked]]
> *Sent:* Saturday, February 10, 2018 3:36 AM
> *To:* Victor Vasiliev <[email-masked]>
> *Cc:* Praveen Balasubramanian <[email-masked]>; [email-masked];
> Marten Seemann <[email-masked]>; huitema <[email-masked]>
> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsoftware.intel.com%2Fen-us%2Farticles%2Fimproving-openssl-performance%23_Toc416943490&data=04%7C01%7Cpravb%40microsoft.com%7C947ea96d592c498130df08d5707a6db1%7Cee3303d7fb734b0c8589bcd847f1c277%7C1%7C0%7C636538593507129957%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-1&sdata=I1%2BnvvFBfsQ1P%2BwSI7riWr%2FkP9ydkAihxlpAgpFggEg%3D&reserved=0>
>
>
>
> Earlier AES-NI I’ve seen from Intel doc suggests around 100cycles in HW
> for a single AES-128 block which would be 33ns per packet number in the
> above example.
>
>
>
> On 10 February 2018 at 06.18.25, Mikkel Fahnøe Jørgensen (
> [email-masked]) wrote:
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 12 Feb 2018 00:15:28 -0800

Running the math with 0.1% PN encrypt overhead and an assumed 1200 byte
packet size

We get 66 cycles for PN encrypt (lower bound) and 66000 cycles for total
processing. AEAD is 1200 * 1,03 = 1236 cycles.
This means non-crypto processing is 66000 - 66 - 1236 = 64698 cycles.
1200/

On a 3GHz core this is 21566ns.
The Solarflare 98 ns/68 bytes result scaled to 1200 bytes is 1729 ns, which
suggests there is a theoretical 10x optimisation potential (which of course
is not realistic in a large scale production environment).

In packets per second this is 1e9/21566 packets/s = 46369pps
With 1200 bytes / packet this is 55,64 MiB/s.

This is roughly 1/2 Gbps which could be related to the network throughput
including loss and retransmission, context switches, and cache effects.

To compare with raw processing time, 55MiB/s is more than 10x slower than a
JSON parser that I have.

On 12 February 2018 at 08.19.07, Victor Vasiliev ([email-masked]) wrote:

 I looked at the profiles of various workloads we're running in production,
and I estimate that none of them would be impacted by PN encryption by
worse than 0.1%.

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 12 Feb 2018 20:43:59 +0900

Victor, Mikkel, thank you for the estimations.

To me it seems that the estimated overhead of 0.1% seems like a
natural number that we would see on production environment, where the
average packet size is not small.

I also agree that it would be worthwhile to look the case where small
packets are exchanged. The numbers deducted from Solarflare report is
interesting, however I am not sure if that reflects the ordinary use
of a protocol; my understanding is that HFT is exceptional.

One benchmark that might give us a more meaningful number is the DNS benchmark.

https://www.knot-dns.cz/benchmark/ lists a benchmark of several
authoritative DNS servers. The TLD benchmark and the Hosting (10k)
benchmark gives us the rough estimation on how much PPS we can achieve
when exchanging small packets.

The numbers we see on the benchmark is roughly 2M RPS (4M pps) on a 16
core (8x2) Intel Xeon processer running with HP enabled, when the
fastest DNS server is being used.

Running `openssl speed -evp` on a similar CPU gives me the following result:

    $ /usr/local/openssl-1.1.0/bin/openssl speed -evp aes128
    Doing aes-128-cbc for 3s on 16 size blocks: 133387877 aes-128-cbc's in 3.00s

This shows that 40M AES block operations can be run on a single core,
per second, which in turn means that a server with 16 cores can
perform 640M AES block operations per second (or 1,280M if HT is
enabled and if AES is not the bottleneck).

To summarize, if DNS had packet encryption, the overhead would be
somewhere from 0.3% to 0.6%.

Considering that, I would anticipate that the overhead of packet
number encryption will be neglible even for short packet workloads.

2018-02-12 16:19 GMT+09:00 Victor Vasiliev <[email-masked]>:
> On Sat, Feb 10, 2018 at 12:12 PM, Praveen Balasubramanian
>> Makes sense. The 1%+ overhead I had quoted was in comparison to full
>> protocol processing all the way from app to NIC. This is with an unoptimized
>> early QUIC implementation and not fully optimized UDP/IP stack (we have
>> focused much much more on optimizing TCP in the past). After software stack
>> optimizations, the crypto share of the cost will only keep going up for
>> which we have no technique other than to offload when such support is
>> available in hardware and offload is way more challenging when workload is
>> hosted in VMs and containers.
>>
>>
>>
>> Cc: Praveen Balasubramanian <[email-masked]>; [email-masked]; Marten
>> Seemann <[email-masked]>; huitema <[email-masked]>
>> encryption, but these are only approximate numbers and CBC mode might have a
>> little overhead, so we clamp packet numbers to 22ns.
>>
>>
>>
>> (simplified) processing time is 98ns non-crypto, 22ns for packet number, and
>> 22ns for AEAD totalling 142ns. So the packet number encryption overhead
>> On 10 February 2018 at 06.18.25, Mikkel Fahnøe Jørgensen
>> ([email-masked]) wrote:
>>

-- 
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 12 Feb 2018 03:49:38 -0800

I can add some more numbers here
https://github.com/openssl/openssl/blob/master/crypto/aes/asm/aesni-x86_64.pl

The numbers deducted from Solarflare report is
interesting, however I am not sure if that reflects the ordinary use
of a protocol; my understanding is that HFT is exceptional.

HFT is the extreme case which is why the benchmark is interesting, not
necessarily the use case itself.

The same concern applies to how fast a distributed database can achieve
consensus, which is something I do care about.

On 12 February 2018 at 12.44.01, Kazuho Oku ([email-masked]) wrote:

One benchmark that might give us a more meaningful number is the DNS
benchmark.

Running `openssl speed -evp` on a similar CPU gives me the following
result:

$ /usr/local/openssl-1.1.0/bin/openssl speed -evp aes128
Doing aes-128-cbc for 3s on 16 size blocks: 133387877 aes-128-cbc's in
3.00s

>> (simplified) processing time is 98ns non-crypto, 22ns for packet number,
and

-- 
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 21 Feb 2018 07:42:12 -0500

Maybe I’m missing something (such as exactly how TLS/QUIC maps encryption
headers to packets:
https://tools.ietf.org/html/draft-ietf-quic-tls-08#section-5.3

but it appears that the packet number encryption / linkage discussion is
focused entirely on AES-GCM.

The CCM mode which is part of TLS 1.3 has a header of 16 octets that
includes a nonce. This nonce would appear to the packet number.

https://tools.ietf.org/html/rfc3610

In summary: CCM has the unencrypted form H, LA, A, PA, LM, M, PM, T where H
is a 16 octets header including nonce, LA is a length or the authenticated
data A, LM is the length of the plain text data M, PA and PM is 0-padding
up to block length, and T is tag of up to 16 octets. Some fields optional
depending on first octet (flag) in H. T is computed as a chain of AES
encryptions. Encryption is applied to LM, M, PM, T in CTR mode. (I suppose
there is also padding after A).

CCM is relevant to devices that have AES acceleration but no CLMUL
instruction for GHASH needed by efficient AES-GCM implementations. E.g.
ESP32 microcontrollers:
https://www.ietf.org/mail-archive/web/tls/current/msg22495.html

Of course the CCM header could partly or fully be removed if the
authenticated data and encrypted data lengths can be implied otherwise, or
part of it could be encrypted.

On 12 February 2018 at 12.49.38, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

and

--
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 21 Feb 2018 08:53:22 -0500

I suppose that CCM is not a major concern since the nonce would be computed
from the encrypted packet number and thus not reveal anything surprising.
It does, however, add a fair bit of overhead:

16 bytes of CCM header including nonce and length data. 0 padded
authenticated data so the packet header will consume 16 octets. This
includes the now redundant encrypted packet number. At the is a 0-15 octet
padding which on might be 7.5 octets on average, or 0 if the packet is
filled perfectly, and of course the 16 octet auth tag.

This means that CCM mode will use at least 3x16 octets and addition to
encrypted payload and UDP headers.

It would probably be easy to remove 32 octets of that overhead, but it
would require special adaptation of CCM for QUIC that might not be
worthwhile - as we wait for devices that include GCM hardware support.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 21 February 2018 at 13.42.12, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

and

--
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 21 Feb 2018 09:09:16 -0500

If the encrypted packet number is truncated, e.g. wrapping after 4 octets,
it would not be possible to use the encrypted packet number as a nonce
unless it is acceptable to use that octets that follow, which could be hard
to justify. The alternative is to either a) have full length packet
numbers, b) renegotiate keys before wrap, c) drop the CCM header and
compute it based on the decrypted packet number - which saves space as
discussed below, but which also interferes with plain TLS CCM operation.

As I have argued before, it might be simpler to simply have a 32-bit
sequential counter and restrict it to a single path, then user the upper
implicit bits for a path identifier (segmented packet numbers). In this
case there is no encryption overhead and the CCM nonce is trivially
available. The only remaining question is if the packet number should be
redundant in both the QUIC header and the CCM header. This could further be
generalised by moving the packet number out of the QUIC and into an AEAD
specific header field - but that of course impacts monitoring and
ossification.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 21 February 2018 at 14.53.23, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Kind Regards,
Mikkel Fahnøe Jørgensen

and

--
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 21 Feb 2018 09:13:37 -0500

Sorry for the mail flux,
but using encrypted packet numbers as nonce would not even be possible with
the current proposal, because the nonce must be known before the packet
number is encrypted.

On 21 February 2018 at 15.09.16, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Kind Regards,
Mikkel Fahnøe Jørgensen

Kind Regards,
Mikkel Fahnøe Jørgensen

and

--
Kazuho Oku