Subject: Proposed changes to cleartext packets

From: Martin Thomson <[email-masked]>
Date: Wed, 3 May 2017 20:43:28 +1000

Issue: https://github.com/quicwg/base-drafts/issues/442

I've heard quite a few concerns about the current design for server
selection of the connection ID.  The primary concern is that the
choice of final connection ID is made on the same flight as the server
handshake messages - the same flight where the server is required to
commit non-trivial state and computation resources to the connection.

The current design is actually anathema to certain load balancer
designs.  If the node that receives a packet wants to get rid of it
before sending the final server flight, the load balancer can no
longer route statelessly.

The server folks I've spoken to want to pick a connection ID during
both version negotiation and stateless reject/address validation.
This allows them to do trivial or at least lightweight processing on
the node that receives the first packet AND steer the packet to the
right node.

I'm proposing that we move to a special "first client packet" approach
rather than a "last server handshake packet" approach.  This allows
servers to identify which packet has a client-selected connection ID
and create special logic for that.

Issue: https://github.com/quicwg/base-drafts/pull/482#discussion_r114445062

The second problem is one that arises from stateless rejects.  A few
people observed that there is a disconnect between what the server
sends in a stateless reject and what it subsequently sends.

For instance, the server can't generate a stateless reject then send
subsequent packets with the expected packet numbers.  The reason: it
might need to generate multiple stateless reject packets in response
to multiple handshake attempts from a client.  It can't save the
packet number it chose in the cookie it sends because that would mean
sending different values for the same octets on stream 0 and clients
could get confused by that.

Also, it was previously necessary to resort to shenanigans at the
server after a stateless reject.  The second ClientHello will start at
a non-zero offset on stream 0, but the server will effectively have
never received that data.  That means pretending that the data was
received and delivered already, but then being able to validate the
offset and back out the change if validation fails.

All in all, that's more special handling than is ideal.  Jana proposed
resetting the transport side of things when this happens.  That means
resetting the stream 0 offset and not getting bent out of shape when
the server chooses a non-contiguous packet number.  That seems like a
reasonable idea, and I'm proposing a new cleartext message type for
that so that it can be explicit when the server is doing this.  That
message comes with its own special treatment (it is used for
HelloRetryRequest only, and HelloRetryRequest has to use it, it has to
be one packet).

This externalizes some of the server costs for stateless rejects, and
it adds another special case packet type.  Neither of which I'm
entirely happy about, but it seems like a reasonable compromise.

These are two separate issues, but they are intermingled in annoying
ways, so I have a single draft PR for both changes:

   https://github.com/quicwg/base-drafts/pull/493

There is a question in the PR about two alternative designs:

1. Require the client to send an initial packet after version
negotiation on the basis that version negotiation is a complete
do-over.

2. Maybe also after a stateless reject, which would mean that all
packets containing the ClientHello would use that "initial" type. That
would mean that only the initial type would have packet size
restrictions, which is nice.

I think that these could be an improvement, but I'd like to hear from
server folks.  The cost here is that after version negotiation and
maybe also a stateless reject, you would route the packet from the
client as if it were a client-selected connection ID.  I'm not sure
how much that matters.  What you do get is up to three opportunities
to reroute a client (during version negotiation, optionally during
stateless reject, and during the handshake).

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 4 May 2017 00:37:56 -0700

I commented on the PR, but echoing it here as well -- I think this is a
very nice set of improvements.

On your alt designs: The server has no way of ensuring that the Client
Cleartext packet contains the server's chosen connection ID. A bold client
could start a connection with a random (or if nefarious, a targeted)
connection ID with a Client Cleartext packet, defeating one of the purposes
of server-chosen connection IDs.

I'll argue for two things to change. (i) I think we should make both the
changes you propose -- that a Client Initial packet be sent right after
both Version Negotiation and Stateless Retry packets. (ii) 0-RTT packets
MUST contain the same connection ID as in the client's most recent
handshake packet to make it possible for load balancers to work with them.

With these changes, a server treats all Client Initial packets similarly,
in that it responds to them with a new server-chosen connection ID. Load
balancers treat all Client Initial, Client Cleartext, and 0-RTT packets as
containing a client-chosen connection ID, and other packets as containing
server-chosen connection IDs.

One of the purposes of this change is to enable a server to direct a client
towards a particular server via the connection ID in a Version Negotiation
or a Stateless Retry packet. A server can still do that by generating a
connection ID that it knows will hash to the right server. Basically, load
balancers will use two functions for hashing connection ID -- C() to hash
client-chosen connection IDs and S() for server-chosen ones. On a VN or SR
packet, the server needs to generate and send a connection ID so that
C(connection ID) sends the retrying client to the desired server. This
allows a server to redirect while also allowing the server infrastructure
to not trust that a client is doing the right thing.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 4 May 2017 19:05:46 +1000

On 4 May 2017 at 17:37, Jana Iyengar <[email-masked]> wrote:
> allows a server to redirect while also allowing the server infrastructure to
> not trust that a client is doing the right thing.

This convinced me.  I'd like to hear from Kazuho, Igor, and Subodh
about whether this works for them.  I don't see why it wouldn't, but I
don't know their infrastructure.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 4 May 2017 21:04:16 +1000

I have written up the proposed change and updated the PR.

>From my commit message, here's the summary:

* Client Initial packets are always a complete reset at the transport
layer, always contain a ClientHello, are always padded to the minimum
size, and include a randomized packet number unless they are a
retransmission.  STREAM frames in these packets always start at an
offset of 0, even if the cryptographic handshake is continued.  All
necessary data has to fit into a single packet; it can't span several.
The server can propose a new connection ID in response to Client
Initial packets (giving the server up to three re-routing options).

* Server Stateless Retry packets cause the client to start over,
retaining only cryptographic handshake state.  The contents of this
also can't span two packets.  This includes a server-proposed
connection ID and an echo of the client packet number.  This is
because the server can't be expected to remember what it chose when it
sends subsequent packets.

* Version Negotiation packets also cause a reset.  This has the same
rules for connection ID and packet number as the Server Stateless
Retry.

* Client Cleartext and Server Cleartext contain the other handshake
messages. These aren't really special in any way.

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 4 May 2017 19:23:25 +0200

Quick question, and maybe I lost the context here, but

On 04.05.2017 13:04, Martin Thomson wrote:
> retransmission.

how can you do a retransmit if you cleared all transport state?

Mirja

---

From: Ted Hardie <[email-masked]>
Date: Thu, 4 May 2017 13:49:02 -0700

On Thu, May 4, 2017 at 12:37 AM, Jana Iyengar <[email-masked]> wrote:

> (ii) 0-RTT packets MUST contain the same connection ID as in the client's
> most recent handshake packet to make it possible for load balancers to work
> with them.
>

I may be misunderstanding something, but I don't see the advantage of using
the same connection ID as in the most recent handshake packet over using
any of the Connection IDs that a server has provided (presuming that the
change that allows a list to be returned for use in connection migration
goes forward).  Presumably, any of the server supplied will work through
load balancers to reach the right server (or the server is implicitly okay
with stateless operation).

To me, at least, it would make for easier client logic if it always used
the next one in the list whether it was migrating a connection because of a
known network change or using 0-RTT because of a period where the transport
wasn't up.  It also might improve privacy, if linking flow state across
0-RTT restarts is a concern.

Am I missing a harm in using one of the later connection IDs, if they were
supplied?

regards,

Ted

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 4 May 2017 16:23:02 -0700

After some discussion with Ryan and some thinking, I'm going to end up
arguing against my previous self (whee!). I'll also suggest a few
modifications.

1. Sending server-proposed connection ID on only Stateless Retry:

I don't think a new connection ID on a Version Negotiation (VN) packet is
particularly useful. Specifically, the issue that I believe this was
expected to solve (Subodh and/or Kazuho should confirm) was if a server
does not support the client's requested version but knows of a different
server that does, it should be able to redirect the client to the correct
server by sending an appropriate connection ID on the VN packet. This
cannot work because the VN packet would contain the client's chosen version
and would therefore be ignored by the client (as it should be.) A server
can send a Stateless Retry (SR) packet with a server-chosen connection ID
when shedding load to a different server. If this is the only case that
needs supporting, a client's use of the server-chosen ID in a subsequent
Client Cleartext packet can be verified by simply adding the server-chosen
ID in the cookie that is sent with the Retry packet.

2. Co-ordinating connection ID change mechanisms:

This mechanism (and the previous one in the current spec) have both the
server and the client changing connection IDs -- the server does it during
handshake and the client does it after it receives one or more
NEW_CONNECTION_ID frames. I think we can make this simpler (both in
protocol and impl) with only the client initiating this change by sending a
packet with a server-chosen connection ID first. If the client were the
only one initiating the change, this would allow the server to not care
about doing any changes mid-connection, and it could simply use the
connection ID in the largest received packet as the outgoing one.

This implies that the server somehow signals the new connection ID to the
client during handshake, which the client MUST use on all 1-RTT encrypted
packets (since we are designing for a server-side load balancer, we can
allow the server to send 1-RTT encrypted packets with the client-chosen
connection IDs to accomodate the 0.5RTT case.) We can achieve this
signaling in one of two ways:
(ii) The server sends a NEW_CONNECTION_ID frame bundled in the ServerHello
packet, but there are corner cases with and without bundling that might
make it tricky. Additionally, the NEW_CONNECTION_ID frame sent along with
the ServerHello could just be the general one sent for a migration event,
and this would be ambiguous signaling. We could fix that by adding a bit
next to each connection ID, but then this bit is unused for all but the
first connection ID.
(i) The server sends a new connection ID as a transport param in the
ServerHello, instead of splitting NEW_CONNECTION_ID frame elements into two
types. NEW_CONNECTION_ID continues to carry IDs for future use.

Explicitly indicating the new connection ID gives the client an explicit
correlator, which is better than assuming that the 4-tuple will only carry
packets for this connection. Since the server also needs to convey this new
connection ID in a Stateless Retry, an SR packet could carry a
NEW_CONNECTION_ID frame.

This leaves us with two protocol mechanisms for carrying a new connection
ID: transport param and NEW_CONNECTION_ID. If we ditch the transport param,
then it's two types of connection IDs within a NEW_CONNECTION_ID frame,
which seems equivalent.

A nice side-effect is that a server that is happy with client-chosen
connection IDs simply does not send this param / frame during handshake.

If this discussion above makes sense, we would need to:
(i) stick with the packet types in the previous version of this patch, and
(ii) decide and add a mechanism for conveying connection ID to client on
handshake.

Thoughts?

On Thu, May 4, 2017 at 1:49 PM, Ted Hardie <[email-masked]> wrote:

> I may be misunderstanding something, but I don't see the advantage of
> using the same connection ID as in the most recent handshake packet over
> using any of the Connection IDs that a server has provided (presuming that
> the change that allows a list to be returned for use in connection
> migration goes forward).  Presumably, any of the server supplied will work
> through load balancers to reach the right server (or the server is
> implicitly okay with stateless operation).
>

---

From: Martin Thomson <[email-masked]>
Date: Fri, 5 May 2017 09:30:16 +1000

> how can you do a retransmit if you cleared all transport state?

The idea here is that you reset transport state when you receive
Version Negotiation or Server Stateless Retry.  When you send the next
Client Initial packet, that needs to be retransmitted.

(Note that you can't reset absolutely everything.  You probably want
to keep the RTT estimate and any congestion signals you have managed
to eke out.)

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 4 May 2017 16:41:43 -0700

> Am I missing a harm in using one of the later connection IDs, if they were

This design is only for the transition from a client-chosen to a
server-chosen connection ID, and not for subsequent ID changes... does that
make sense? The requirement you highlighted accomodates load balancers that
use different hashing algorithms on client-chosen vs server-chosen
connection IDs. For instance, a load balancer might use simple ECMP hashing
of the client-chosen connection IDs and treat server-chosen connection IDs
as containing structure or routing information. Changing the connection ID
on an externally visible transition allows such load balancers to use the
different hashing algorithms correctly on the connection IDs.

- jana

---

From: Martin Thomson <[email-masked]>
Date: Fri, 5 May 2017 10:09:19 +1000

I don't think that either suggestion works particularly well.  I'll
explain below.

On 5 May 2017 at 09:23, Jana Iyengar <[email-masked]> wrote:
> expected to solve (Subodh and/or Kazuho should confirm) was if a server does
> not support the client's requested version but knows of a different server
> that does, it should be able to redirect the client to the correct server by
> sending an appropriate connection ID on the VN packet.

I believe that this was one of the cases that was identified.  And you
are correct that a stateless reject is the only way to do what was
intended here.  Either the version negotiation packet includes the
client's chosen version and is ignored, or it doesn't match the
version that the other server will ultimately advertise.  BTW, I've
opened an issue for documenting how a server cluster might be upgraded
(and rolled back): https://github.com/quicwg/base-drafts/issues/504

The use case you are missing is one where the server receives an
unsupported version AND it wants to send the client to another server.
One reason you might want to do that is to avoid a second round trip.
I'll concede that is a little unlikely given the potential need for
address validation, so it's a pretty minor gain, but a gain
nonetheless.

Can you demonstrate something that we would lose by allowing the
server to propose a connection ID here?  The gain might be small, but
I'm not seeing any downside to allowing it.

> NEW_CONNECTION_ID frames. I think we can make this simpler (both in protocol
> and impl) with only the client initiating this change by sending a packet
> with a server-chosen connection ID first.

Let me try to restate your proposal.

You would have the entire handshake complete with the same connection
ID, then switch to a different connection ID once the handshake
completes.  A stateless reject would be allowed to cause re-routing.

I'm not sure that I understand the advantages that this design has.
You claim that it reduces the number of mechanisms to two, but I think
that you end up with three because you need to use a transport
parameter.  (FWIW, the current writeup essentially only has two
mechanisms: NEW_CONNECTION_ID and server handshake packets, though
I'll concede that there might be three of those).  It has essentially
the same properties as what I've written up, but it's more complicated
in several ways.

The NEW_CONNECTION_ID frame implies a packet number gap.  With client
authentication, the server might receive encrypted packets before the
handshake completes.  If we take ekr's pull request for
NEW_CONNECTION_ID, the server can't know what the gap is without
receiving the handshake packets.

This would create more exceptions for what can be sent in the clear,
which requires more care.  If, as proposed, we allow multiple new
connection IDs to be sent in the same frame, we wouldn't want that to
happen here, so that's another new special case to test.

That suggests a transport parameter might be a better design, but we
don't send transport parameters in the HelloRetryRequest.  We'd lose
confidentiality for those unless we created a new set of rules for
what transport parameters can be sent in a HelloRetryRequest.  That
generates more special rules.

I think that you raise an important point here:

> Changing the connection ID on an externally visible transition allows such load balancers to use the different hashing algorithms correctly on the connection IDs.

The current proposal has equally simple rules:

if packet[0] == 0x82 or packet[0] == 0x86:
  routing = client_selected
else:
  routing = server_selected

That is, Client Initial and 0-RTT get routing as though the client chose the ID.

You would instead have us switch the test to packet[0] & 0x80 == 0x80
and packet[0] != 0x87 and packet[0] != 0x88, that is every long form
packet except the 1-RTT ones.  I don't see any advantage there.

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 5 May 2017 18:06:58 -0700

On Thu, May 4, 2017 at 5:09 PM, Martin Thomson <[email-masked]>
wrote:

> does

So an example would be:
Client sends CHLO in QUICv2
Server only speaks QUICv1, and wants to shed load to another that speaks
QUICv1
Server sends VN with Connection ID that leads client to other server on
retry

This will be a super rare case: VN itself should be rare to begin with,
given Alt-Svc. Within that probability, this is a server that wants to shed
load and knows which server to shed to. And for those connections, we're
saving 1 RTT. This one  RTT will be amortized if there are subsequent
connections, since subsequent connections should learn about the new
version and not incur the VN delay. This optimization is super unlikely to
affect performance for applications or for servers.

> I'm not seeing any downside to allowing it.

I don't think we should add complexity to optimize an RTT for super rare
cases. If we could do it for no cost, that's fine. But I don't think it
makes sense to consider a design that gets limited because of this rare
case. The fewer places that we have code doing the same or similar things,
the better, so if there's a mechanism that works well for the rest of the
cases and VN fits in seamlessly, then sure, let's do it.

> protocol
> completes.  A stateless reject would be allowed to cause re-routing
>

Yes, that's correct.

> receiving the handshake packets.

That's a single bit of information we can add to the frame.

> happen here, so that's another new special case to test.

So you'd send a transport param in the Server Hello (not in the clear) or a
frame.
If you're sending the frame in response to an SR, it will not contain any
information that you need to be confidential.
If you're sending the frame after the ServerHello, it will be encrypted
with 1-RTT keys.
I don't think you need special rules here.

> generates more special rules.

You'd use a frame. Since HRR is a STREAM frame, and should be a small one,
bundling a NEW_CONNECTION_ID frame with it should be relatively easy (in
code.)

> > Changing the connection ID on an externally visible transition allows
> such load balancers to use the different hashing algorithms correctly on
> the connection IDs.
>

> packet except the 1-RTT ones.  I don't see any advantage there.

That's not right. Your logical expression reduces to:
if packet[0] != 0x87 and packet[0] != 0x88:
  routing = client_selected
else:
  routing = server_selected

... which is the same complexity, but it isn't the correct filter anyways.
There's a related problem that I realized with this most recent scheme:
Connection ID in 0-RTT packets are ambiguous. If they follow a Client
Initial, they're client-chosen, but if they follow a Client Cleartext,
they're server-chosen. I think this is a problem in your earlier version
too. So that's a bummer.

You say you see no advantages. I wrote out the advantages in my previous
email, but perhaps you didn't see them?

1. Explicitly indicating the new connection ID gives the client an explicit
correlator from the client's ID to the new server one, which is better than
assuming that the 4-tuple will only carry packets for this connection.
Currently,
the client will receive the first server packet with an unknown Connection
ID and the only correlator is the fact that it was received on the same
4-tuple that the Client Initial was sent on. If you have multiple
connections on the same 4-tuple, there's no explicit correlator.

2. This mechanism (and the previous one in the current spec) have both the

These are still valuable if we can have them.

I'll think about this and get back on Monday. I think the current mechanism
in the PR is sound; I'd like to see if we can address the two issues I
mention above.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 8 May 2017 09:31:05 +1000

On 6 May 2017 at 11:06, Jana Iyengar <[email-masked]> wrote:
> cases. If we could do it for no cost, that's fine.

I guess that it's a matter of perspective. I consider the less complex
case to be where the server's connection ID is always correct and the
server can respond with a new connection ID if the client sends Client
Initial.  That means that having Version Negotiation contain a
server-selected connection ID is the cheap option.

> That's a single bit of information we can add to the frame.

Yep, more complexity.

>> You would instead have us switch the test to packet[0] & 0x80 == 0x80>> and packet[0] != 0x87 and packet[0] != 0x88, that is every long form

Read it again, it's not the same.  It says everything with long form
except those two.  Though the point was poorly made: the logic isn't
particularly relevant.

> Currently, the client will receive the first server packet with an unknown
> Connection ID and the only correlator is the fact that it was received on
> the same 4-tuple that the Client Initial was sent on. If you have multiple

OK, that didn't seem like a real advantage to me, just a rearrangement
of the furniture.  Packets reach the client based on port/IP pairs.

> with a server-chosen connection ID first. If the client were the only one
> initiating the change, this would allow the server to not care about doing
> any changes mid-connection, and it could simply use the connection ID in the
> largest received packet as the outgoing one.

Yes, my assertion was that this isn't in fact simpler due to the need
for special rules during the handshake.

If you wanted to talk about how we might, for example, use a different
connection ID in each direction, then I might concede that there are
advantages in using the frame type.

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 8 May 2017 19:04:16 -0700

There are two issues that are I brought up; for clarity's sake, I'll limit
this discussion to the immediate one. I'm summarizing the two proposals,
because it helped clarify my thinking, so I hope it helps others as well.
Please correct / add anything you think is incorrect / missing. I'm back in
favor of the newer proposal, FWIW, see below.

*Proposal #1 (first proposal):*
Client sends Client Initial
Server sends ServerHello/Stateless Retry/Version Negotiation with new
connection ID

If server sends SHLO:
Client continues with connection, uses server-supplied connection ID
Server continues using same connection ID

If server sends SR / VN:
Client resets connection
Client sends Client Cleartext with server-supplied connection ID
Server continues using same connection ID.

Load balancer rules: Complicated; 0-RTT packets from client following
Client Initial carry client-chosen connection ID and those following the
Client Cleartext carry server-chosen connection ID.

Issues:
- no rules specified for the server to verify that the client is indeed
using the server's supplied connection ID.
- no rules specified for client to verify that the server saw its Client
Initial before sending SR/VN. (I see now that the packet number is echoed,
but we should turn this into a requirement for the client to verify. This
is what I meant by a "correlator" for the client to use to tie the
client-chosen ID to the server-chosen one.)

*Proposal #2 (newer proposal):*
Client sends Client Initial
Server sends ServerHello/Stateless Retry/Version Negotiation with new
connection ID

If server sends SHLO:
Client continues with connection, uses server-supplied connection ID
Server continues using same connection ID

If server sends SR / VN:
Client resets connection
Client sends Client Initial with server-supplied connection ID #1 (SCID1)
Server sends SHLO with new server-chosen connection ID #2 (SCID2).
Client continues sending Client Cleartext and 0-RTT packets with SCID1
When SHLO is fully received, client switches to 1-RTT packets and SCID2

Load balancer rules: Simple; 1-RTT packets use server-chosen connection
IDs, everything else uses client-chosen IDs (from the load-balancer's point
of view, even though SCID1 is actually server-chosen.)

Issue (repeating for clarity):
- no rules specified for client to verify that the server saw its Client
Initial before sending SR/VN. (I see now that the packet number is echoed,
but we should turn this into a requirement for the client to verify. This
is what I meant by a "correlator" for the client to use to tie the
client-chosen ID to the server-chosen one.)

Proposal #2 additionally has the value that no rules are required for the
server to verify that the client is indeed using the server's supplied
connection ID, since the load balancer treats SCID1 as client-chosen
anyways.

If my understanding is correct, I am back in favor of Proposal #2, because
I cannot resolve the load balancer question of 0-RTT packets with Proposal
#1. Laying it out like this makes it clearer, and I wonder if some of it
can make it into the draft. Currently, the draft does not really specify
how a server might use this degree of freedom of specifying server-chosen
connection ID in SR packets (especially with load balancing rules the way
they are intended to be), and I think it may be valuable to lay out what a
server might want to do. Should this go in the applicability draft perhaps?

- jana

On Sun, May 7, 2017 at 4:31 PM, Martin Thomson <[email-masked]>
wrote:

> >> You would instead have us switch the test to packet[0] & 0x80 == 0x80>>

---

From: Martin Thomson <[email-masked]>
Date: Tue, 9 May 2017 12:12:37 +1000

On 9 May 2017 at 12:04, Jana Iyengar <[email-masked]> wrote:
> Load balancer rules: Complicated; 0-RTT packets from client following Client
> Initial carry client-chosen connection ID and those following the Client
> Cleartext carry server-chosen connection ID.

No, this isn't right.  I already agreed to the client using the same
connection ID marking 0-RTT packets, the PR right now explicitly
requires the client to keep the same connection ID.

It looks like the only disagreement here is over a) whether a Version
Negotiation packet can include a new server-chosen connection ID, and
b) whether the server-chosen connection ID is carried in the
connection ID field, in a frame, or in a transport parameter.

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 8 May 2017 19:18:14 -0700

Did you see proposal #2?

On May 8, 2017 7:12 PM, "Martin Thomson" <[email-masked]> wrote:

> Client

---

From: Martin Thomson <[email-masked]>
Date: Tue, 9 May 2017 12:22:32 +1000

> Did you see proposal #2?

Ahh, you changed your mind and I missed the /VN bit.

Ahh, so we only disagree on when the client switches to using the
server selection.  That clearly takes any options away regarding how
the connection ID is carried.

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 8 May 2017 19:36:11 -0700

On May 8, 2017 7:22 PM, "Martin Thomson" <[email-masked]> wrote:

I did change my mind after thinking about load balancer issues a bit more
as I've tried to articulate, but I think there was a lot of confusion
because of all the different moving parts.

Do we? Assuming that you agree on proposal #2, how would you change
proposal #2?

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 8 May 2017 19:41:27 -0700

On May 8, 2017 7:36 PM, "Jana Iyengar" <[email-masked]> wrote:

Once we agree on this so far, I'd like to bring up the other questions I'm
considering. They may not be worth it, but it'll be easier to discuss them
once we're on the same page.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 9 May 2017 14:05:01 +1000

On 9 May 2017 at 12:36, Jana Iyengar <[email-masked]> wrote:
> Do we? Assuming that you agree on proposal #2, how would you change proposal
> #2?

I don't have enough info, but I think that there might be a problem.

Let's say that the server does bump the client twice, either a
stateless reject or version negotiation (or both).  That way, when the
ClientHello that the server will consume properly will have a
server-selected connection ID.

Then you have the 1-RTT case with rejections:

<some rejections>
Client Initial, CCID, ClientHello ->
  <- Stateless Retry or Version Negotiation, SCID1, [HelloRetryRequest]
Client Initial, SCID1, ClientHello ->
  <- Server Cleartext, SCID2, ServerHello (etc)
Client Cleartext, SCID1**?, Finished ->
1-RTT Data, SCID2 ->

<no rejections>
Client Initial, CCID, ClientHello ->
  <- Server Cleartext, SCID2, ServerHello (etc)
Client Cleartext, SCID2**?, Finished ->
1-RTT Data, SCID2 ->

The change you propose is on the Client Cleartext line in the "some
rejections" case. If I've understood you correctly, rather than use
SCID2 here (as included in what I wrote up), you want to keep using
SCID1.

A few concerns.  If this is right, I'm not sure how the server
determines that a client change is OK in the "no rejections" case.

I'm assuming here that once a server node sends a ServerHello, it
really wants to at least be able to ensure that client packets reach
that node again.  As you said previously, if you have route_c() and
route_s() and you only apply route_c() when the client might have
chosen the connection ID.  Here, you are proposing that we change that
logic to use route_c() for all cleartext packets from a client.

I'm still not sure what you are trying to gain by the change.  Is the
intent to ensure that the handshake all ends up in the same place as
0-RTT data?  I've been thinking that the first priority would be to
ensure that 1-RTT data and the handshake arrive in the same place and
that 0-RTT data might take extra effort to send to the right place
(but since it is bounded in size, you might reasonably be able to
manage that by forwarding or other such horrors).

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 8 May 2017 23:05:40 -0700

(I'd like Subodh and Kazuho to weigh in, since they were interested in
seeing solutions to this problem.)

Ah -- you are right of course! I mis-wrote one part, which I'll correct
below. Apologies for seeming like I'm pushing a change -- I wasn't. I was
partly trying to think through, and partly mis-articulating, and I think
what you just wrote helped clear out the final cobwebs in my addled brain.
So here's my corrected proposal, and below it is what I now understand to
be the proposal in the PR. In summary, either design works for me. I don't
have a strong argument for either.

Now that I understand, the text fits fine. It might just be me, but if
others find this non-trivial to reason about -- not because the text isn't
clear, but because this may just be non-trivial -- I wonder if it will help
to explain this flow of packets somewhere... perhaps in the Applicability
draft?

----------------
*Proposal #2 (newer proposal), my corrections:*
Client sends Client Initial
Server sends ServerHello/Stateless Retry/Version Negotiation with new
connection ID

If server sends SHLO:
Client continues with connection, uses client-chosen connection-ID for
Client Cleartext and 0-RTT packets *(<-- corrected)*
Client switches to server-supplied connection ID for 1-RTT packets. *(<--
corrected)*
Server continues using same connection ID

If server sends SR / VN:
Client resets connection

---------------------

I believe what you're saying is, use client-chosen connection ID on 0-RTT,
but not on Client Cleartext packets, since Client Cleartext packets are
sent only after the client has heard the first ServerHello packet. Which
would change the load balancer rules, as you articulated earlier: Client
Initial and 0-RTT packets use client-chosen connection IDs, everything else
uses server-chosen connection IDs.

----------------
*Proposal in the PR:*

If server sends SHLO:
Client continues with connection, uses client-chosen connection-ID for
0-RTT packets
Client switches to server-supplied connection ID for Client Cleartext and
1-RTT packets
Server continues using same connection ID

If server sends SR / VN:
Client resets connection
Client sends Client Initial with server-supplied connection ID #1 (SCID1)
Server sends SHLO with new server-chosen connection ID #2 (SCID2).
Client continues sending 0-RTT packets with SCID1
When first packet of SHLO is received, client switches to SCID2 for Client
Cleartext and 1-RTT packets

Load balancer rules: Simple; Client Initial and Client Cleartext packets
use client-chosen connection IDs, everything else uses server-chosen IDs
(from the load-balancer's point of view, even though SCID1 is actually
server-chosen.)

---------------------

On Mon, May 8, 2017 at 9:05 PM, Martin Thomson <[email-masked]>
wrote:

> proposal

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 8 May 2017 23:21:00 -0700

Let me know if we are on the same page.

I'd still like to see if there's anything we can do about localizing
connection ID changes to the client, but I think it's probably best if we
discuss that separately and not on this PR.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 9 May 2017 17:05:31 +1000

> Let me know if we are on the same page.

Yep, I think that's the essence of the choice we have in front of us.
Both alternatives are relatively simple once we get over the hurdle of
actually understanding each other :)

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 9 May 2017 00:10:32 -0700

> actually understanding each other :)

We usually get there -- just needed some patience and time... and less
bungling :-)

---

From: Martin Thomson <[email-masked]>
Date: Thu, 11 May 2017 10:22:34 +1000

>    https://github.com/quicwg/base-drafts/pull/493

After a bunch of discussion, the editors at least agree that this is
an improvement.  I have just merged this change into the editor's
copy.

We will put this on the agenda for the interim to discuss with a larger group.