Subject: Swimming in a sea of CIDs

From: Dmitri Tikhonov <[email-masked]>
Date: Fri, 9 Nov 2018 15:01:49 -0500

Hello,

I wonder about a scenario in which one endpoint (A) issues more connection
IDs than the other endpoint (B) is willing to store.  If A issues N
CIDs, but B only has room for N/2 CIDs, [the second] half of the CIDs
will be dropped.  If B tells A that CIDs N/2 + 1 through N are retired,
A will simply issue N/2 more CIDs, so B should not do that.  Instead, B
should remember that while it only has N/2 CIDs, N were actually issued.
If B needs to use a new CID, how many CIDs should it retire?  If it
only retires one CID, A might figure that B still has N - 1 CIDs and
not issue another one.

Should B retire a proportion of CIDs A issued?  For example, in our
scenario, N divided by N/2 is 2, so B would retire 2 CIDs -- one from
the stored half and one from dropped half -- so that the rate of CID
consumption is the same at A and B.

That last approach is workable, but seems a bit of a pain (the proportion
may not turn out so convenient as in the example above and one has to
keep track of the two sets of CID sequence IDs).  What should B do?

Seeking guidance,

  - Dmitri.

---

From: Ian Swett <[email-masked]>
Date: Sat, 10 Nov 2018 17:37:10 -0500

When I discussed this with people previously, it seemed like the obvious
solution was flow control(ie: MAX_CONNECTION_ID?), but most thought that
was overkill.

A transport parameter would be a simple way to indicate how many a peer is
willing to store, but if we're not going to change anything, I think some
clearer guidance on best practice would be nice.

---

From: Eric Kinnear <[email-masked]>
Date: Sun, 11 Nov 2018 11:15:17 +0900

Agreed, I think the most explicit way to handle this was a TP for how many connection IDs the receiving endpoint (vs the issuing one) was willing to store. My recollection is that we started with this TP in the PR, took it out and added it back, and then took it out for what finally got merged. 

Generally, in terms of best practice, I think just ignoring the “extra” CIDs is the way to go. Using A and B from the example below: 

A issues more CIDs than B is willing to store, say 64. 
In the example, that means that B has room for 32 CIDs. (Chose random numbers here)
Retiring the unused 32 will just result in more being sent, so B should just ignore the CIDs beyond which it’s willing to store. 

From there, B should consume and retire CIDs as normal, and A should replace any of the CIDs retired. 
In other words, a bad strategy for A would be to flood the other endpoint with CIDs and then wait until it only has a few left before replenishing them—A should make no assumptions about how many CIDs B is holding onto and instead respond to retire messages with new CIDs. 

A TP is a rather nice way to prevent A from having to keep state for more CIDs than B is actually ever going to use, but it may not actually change much—B should still expect A to replenish upon retirement, so perhaps it’s worth just highlighting that strategy on the issuer side rather than adding more machinery around this. 

Thanks,
Eric

> On Nov 11, 2018, at 7:37 AM, Ian Swett <ianswett=[email-masked]> wrote:
> 
> When I discussed this with people previously, it seemed like the obvious solution was flow control(ie: MAX_CONNECTION_ID?), but most thought that was overkill.
> 
> A transport parameter would be a simple way to indicate how many a peer is willing to store, but if we're not going to change anything, I think some clearer guidance on best practice would be nice.
> 
> On Fri, Nov 9, 2018 at 3:02 PM Dmitri Tikhonov <[email-masked] <mailto:[email-masked]>> wrote:
> Hello,
>

---

From: Dmitri Tikhonov <[email-masked]>
Date: Sun, 11 Nov 2018 02:42:39 +0000

Just retiring one CID does not mean that the peer will replace it immediately; it is not mandated anywhere in the doc. All that is said is that one must endure that the other end has sufficient CIDs. If you ignore CIDs, the issuer may think you still have them.

________________________________
From: QUIC <[email-masked]> on behalf of Eric Kinnear <[email-masked]>
Sent: Saturday, November 10, 2018 21:15
To: Ian Swett
Cc: IETF QUIC WG
Subject: Re: Swimming in a sea of CIDs

Thanks,
Eric

On Nov 11, 2018, at 7:37 AM, Ian Swett <ianswett=[email-masked]<mailto:ianswett=[email-masked]>> wrote:

On Fri, Nov 9, 2018 at 3:02 PM Dmitri Tikhonov <[email-masked]<mailto:[email-masked]>> wrote:
Hello,

Seeking guidance,

  - Dmitri.

---

From: Eric Kinnear <[email-masked]>
Date: Sun, 11 Nov 2018 11:53:34 +0900

That is correct, it is however suggested as the only example of how to satisfy the language here: 

"An endpoint SHOULD ensure that its peer has a sufficient number of available and unused connection IDs. While each endpoint independently chooses how many connection IDs to issue, endpoints SHOULD provide and maintain at least eight connection IDs. The endpoint can do this by always supplying a new connection ID when a connection ID is retired by its peer or when the endpoint receives a packet with a previously unused connection ID.”

Perhaps either “replace upon retirement” should be a SHOULD, or there should be some more language here to indicate that the endpoint cannot assume its peer has retained all issued CIDs and should not make decisions based upon an estimation of how many the other end has (note that this isn’t specific to the current strategy, reordering etc. also make it difficult for the endpoint to assume things about the state on the other end, so that’s likely a good plan regardless)?

Thanks,
Eric

> On Nov 11, 2018, at 11:42 AM, Dmitri Tikhonov <[email-masked]> wrote:
> 
>> On Nov 11, 2018, at 7:37 AM, Ian Swett <ianswett=[email-masked] <mailto:ianswett=[email-masked]>> wrote:
>>

---

From: Dmitri Tikhonov <[email-masked]>
Date: Sun, 11 Nov 2018 00:22:54 -0500

On Sun, Nov 11, 2018 at 11:53:34AM +0900, Eric Kinnear wrote:
> Perhaps either “replace upon retirement” should be a SHOULD, or
> there should be some more language here to indicate that the endpoint
> cannot assume its peer has retained all issued CIDs and should not
> make decisions based upon an estimation of how many the other end has
> (note that this isn’t specific to the current strategy, reordering
> etc. also make it difficult for the endpoint to assume things about
> the state on the other end, so that’s likely a good plan regardless)?

I think something is needed.

Why was the transport parameter dropped?  It seems to be the simplest.
(Certainly simpler than specifying another flow control mechanism and
the accompanying framing.)

  - Dmitri.

---

From: Eric Kinnear <[email-masked]>
Date: Sun, 11 Nov 2018 15:35:42 +0900

> On Nov 11, 2018, at 2:22 PM, Dmitri Tikhonov <[email-masked]> wrote:
> 
Agreed! Probably best not to leave people guessing here.

> Why was the transport parameter dropped?  It seems to be the simplest.

My recollection here is that it was for much the same reasons that Ian outlined. 

Even with a TP, it’s not really a good idea for the issuer to try to keep track of what it thinks the other side has, since that starts to involve all sorts of                                                                                                                                                                                                                                                             synchronization problems—instead it can just keep track of what it knows it’s promised to the other endpoint. 

So, even with a TP, it might be worth adding some explanatory text to the section above or making the replenishment a SHOULD. 

Thanks,
Eric

---

From: Dmitri Tikhonov <[email-masked]>
Date: Sun, 11 Nov 2018 02:15:45 -0500

On Sun, Nov 11, 2018 at 03:35:42PM +0900, Eric Kinnear wrote:
> Even with a TP, it’s not really a good idea for the issuer to try
> to keep track of what it thinks the other side has, since that starts
> to involve all sorts of synchronization problems—instead it can just
> keep track of what it knows it’s promised to the other endpoint.

The issuer only has to keep track of the size of the current set, which
the TP would specify.  The size of the current set would thus be the
minimum of the number of CIDs an endpoint is willing to provide and the
number of CIDs its peer is willing to track.

> So, even with a TP, it might be worth adding some explanatory text to
> the section above or making the replenishment a SHOULD.

Agreed.

  - Dmitri.

---

From: Ian Swett <[email-masked]>
Date: Sun, 11 Nov 2018 03:31:05 -0500

A TP would also avoid the current texts suggestion of making 8 available,
which is more than I expect my implementation to request.

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 12 Nov 2018 16:29:51 +0700

We discussed this in the past, specifically in NYC. The agreement was that
a RETIRE_CONNECTION_ID would have the semantic that the peer would
replenish it, if it wanted to ensure that the peer had enough CIDs. A peer
that didn't have enough memory to store the issued CIDs would expect to
receive a NEW_CONNECITON_ID on sending a RETIRE_CONNECTION_ID.

Eric pointed exactly to the text we introduced to explain this. I thought
we had adequate text to cover this (in 5.1.1 and 5.1.2), but it might be
worth saying explicitly that an endpoint SHOULD NOT assume that the peer is
storing all of the CIDs that it issued. I'll do this.

I don't have a strong opinion on the TP, but I agree with Eric that I don't
think a TP changes much once the semantics of RETIRE_CONNECTION_ID are made
clear. We do have that SHOULD around 8, but 8 was a bit arbitrary, and we
can change that. Or we can add a TP. If you care strongly enough, I'll ask
you to open an issue and drive it.

On Sun, Nov 11, 2018 at 3:31 PM Ian Swett <ianswett=
[email-masked]> wrote:

> On Sun, Nov 11, 2018 at 2:15 AM Dmitri Tikhonov <
> [email-masked]> wrote:
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 12 Nov 2018 02:18:52 -0800

Also I also commented on Jana’s recent PR on topic:

I think it is messy to have an endpoint discard excess CID’s since the
sender may also need to keep state. But since the sender can flood the
receiver, something needs to be done.

I believe the simplest solution is to have a TP with a fixed count, for
example for 5 or 8 chosen at connection time. CID’s are replenished when
one is retired.

Mikkel

On 12 November 2018 at 10.30.19, Jana Iyengar ([email-masked]) wrote:

replenish it, if it wanted to ensure that the peer had enough CIDs.. A peer

---

From: Rui Paulo <[email-masked]>
Date: Mon, 12 Nov 2018 11:47:15 -0800

Just wanted to say that this was brought up on issue 1924:

https://github.com/quicwg/base-drafts/issues/1924

> On Nov 12, 2018, at 02:18, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> 
> I think it is messy to have an endpoint discard excess CID’s since the sender may also need to keep state. But since the sender can flood the receiver, something needs to be done.
> 
> I believe the simplest solution is to have a TP with a fixed count, for example for 5 or 8 chosen at connection time. CID’s are replenished when one is retired.
> 
> Mikkel
> 
>> We discussed this in the past, specifically in NYC. The agreement was that a RETIRE_CONNECTION_ID would have the semantic that the peer would replenish it, if it wanted to ensure that the peer had enough CIDs.. A peer that didn't have enough memory to store the issued CIDs would expect to receive a NEW_CONNECITON_ID on sending a RETIRE_CONNECTION_ID.
>> 
>> Eric pointed exactly to the text we introduced to explain this. I thought we had adequate text to cover this (in 5.1.1 and 5.1.2), but it might be worth saying explicitly that an endpoint SHOULD NOT assume that the peer is storing all of the CIDs that it issued. I'll do this.
>> 
>> I don't have a strong opinion on the TP, but I agree with Eric that I don't think a TP changes much once the semantics of RETIRE_CONNECTION_ID are made clear. We do have that SHOULD around 8, but 8 was a bit arbitrary, and we can change that. Or we can add a TP. If you care strongly enough, I'll ask you to open an issue and drive it.
>> 
>> On Sun, Nov 11, 2018 at 3:31 PM Ian Swett <ianswett=[email-masked]> wrote:
>> A TP would also avoid the current texts suggestion of making 8 available, which is more than I expect my implementation to request.
>> 
>> On Sun, Nov 11, 2018 at 2:15 AM Dmitri Tikhonov <[email-masked]> wrote:

—
Rui Paulo