Subject: reserved bits for spin bit, etc.

From: Christian Huitema <[email-masked]>
Date: Wed, 22 Nov 2017 10:50:47 -0800

We cannot quite agree on the spin bit, let alone further management
bits. On the other hand, we see that there is clear interest from
something like that with at least part of the WG, we want to see some
experimentation, and we understand that the specification can proceed
independently from the base protocol development. Hence, I would like to
make a modest proposal.

Can we mark one or possibly two bits as reserved in the first byte of
the QUIC header? Specify them as SHOULD be zero when sending, SHOULD be
ignored when receiving, MAY be used for experiments between consenting
nodes?

-- Christian Huitema

---

From: Ted Hardie <[email-masked]>
Date: Wed, 22 Nov 2017 11:22:18 -0800

> ignored when receiving, MAY be used for experiments between consenting
While I'd personally be happy with reserving some bits for this now, I
think marking them as experimental is worse than setting them aside as
"reserved" (or whatever the shorthand is for "still under discussion").  If
folks start running multiple different experiments across full-path
connections, interpreting the data you get will be much harder.  Unless
there is some way to connect the experiment's semantics with the bit, it
seems to make a garbage bit likely.

Possibly I am too pessimistic, but that's my initial reaction.

Ted

---

From: "Eggert, Lars" <[email-masked]>
Date: Wed, 22 Nov 2017 19:57:46 +0000

On 2017-11-22, at 20:22, Ted Hardie <[email-masked]> wrote:
> 
> While I'd personally be happy with reserving some bits for this now, I think marking them as experimental is worse than setting them aside as "reserved" (or whatever the shorthand is for "still under discussion").  If folks start running multiple different experiments across full-path connections, interpreting the data you get will be much harder.  Unless there is some way to connect the experiment's semantics with the bit, it seems to make a garbage bit likely.

(chair hat off)

Agree with Ted. Am OK with reserving, but we'd need to grease them while they remain reserved, so they remain useable upon assignment.

Lars

---

From: Marten Seemann <[email-masked]>
Date: Wed, 22 Nov 2017 15:01:24 -0500

I like the idea to get some real-world data on the impact of exposing
further information. The problem with reserving a couple of bits in the
packet header for this purpose is that it’s not sufficient for the two end
points to agree on the meaning of those bits, the middle boxes must be
aware of it as well.
Maybe I’m missing an option here, but it seems that in order to perform
such an experiment, both endpoints as well as the middle boxes have to be
controlled by the same operator. In this case, it wouldn’t be necessary to
reserve any bits, since the end points are free to expose additional
information in any way they see fit.

On 22. November 2017 at 11:23:09, Ted Hardie ([email-masked]) wrote:

Ted

---

From: Martin Thomson <[email-masked]>
Date: Thu, 23 Nov 2017 09:51:19 +1100

Would you propose making this reservation in QUICv1 or QUICvAll?

---

From: Christian Huitema <[email-masked]>
Date: Wed, 22 Nov 2017 16:13:28 -0800

> Would you propose making this reservation in QUICv1 or QUICvAll?

Do you mean, just for the current version, or invariant across all
versions? My idea is to reserve them for the current version. If the
experiments let us firmly conclude one way or the other by the time we
publish V1, the bits will cease to be reserved, and will have an
explicit specification instead. In any case, "reserved" means that it is
available for use in a further version.

-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Wed, 22 Nov 2017 16:15:51 -0800

On 11/22/2017 11:57 AM, Eggert, Lars wrote:

> On 2017-11-22, at 20:22, Ted Hardie <[email-masked]
> <mailto:[email-masked]>> wrote:
>>
>> While I'd personally be happy with reserving some bits for this now,
>> I think marking them as experimental is worse than setting them aside
>> as "reserved" (or whatever the shorthand is for "still under
>> discussion").  If folks start running multiple different experiments
>> across full-path connections, interpreting the data you get will be
>> much harder.  Unless there is some way to connect the experiment's
>> semantics with the bit, it seems to make a garbage bit likely.
>
> (chair hat off)
>
> Agree with Ted. Am OK with reserving, but we'd need to grease them
> while they remain reserved, so they remain useable upon assignment.

Fair point. So the first bit should be defined as "reserved for the spin
bit experiment". The second bit, if we decide to have a second one,
should be strictly designed as "reserved" for now.

-- Christian Huitema

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 23 Nov 2017 06:41:02 +0000

On 23/11/2017, 00:15, Christian Huitema wrote:
>>> I think marking them as experimental is worse than setting them 
>>> aside as "reserved" (or whatever the shorthand is for "still under 
> Fair point. So the first bit should be defined as "reserved for the 
> spin bit experiment". The second bit, if we decide to have a second 
> one, should be strictly designed as "reserved" for now.
>
> -- Christian Huitema
I'm preetty sure that if we had a set of bits that were declared in the 
header, the IETF could define a mechanism that offers the features we 
want. I think the IETF transport people are quite capable of proposing a 
mechanism(s) if we spend the time discussing the mechanisms, rather than 
whether the bit field can be present. Sure, I'd agree the choice of 
which to use could be experimental in the first release - and become 
"standardised" in later releases.

I do suggest we consider two bits for the "spin", rather than one, 
because of the ability to robustly measure in the presence of 
loss/reordering - the greatest value is a system that continues to offer 
useful data when the path is under stress, so that the mechanisms can be 
used to figure out how to fix/analyse problems. Maybe it may be useful 
to also be able to quantify loss/congestion/reodering (even when on the 
return path) ... but I do think we need to first look at robust latency 
measurement.

Gorry

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Thu, 23 Nov 2017 09:02:14 +0100

+1 to Gorry. 

I can think of good uses off the top of my head for three bits in the short header simultaneously, even four if we can have them, and as it looks like short header packet number varints are a good idea, we’ll get the whole type field (5bits) back. 

However, two bits provides us enough to experiment with a two-bit spin versus spin-and-loss versus spin-and-block, and leaves us with three bits for future type signaling on the short header. It’s a good compromise.

As to whether these are reserved in v1 versus reserved forever, I note that once these deploy in v1 they are probably effectively reserved forever, but we don’t have to make that choice until they deploy. So reserved in v1 is fine for now and would indeed help with experimentation: Piet’s minq fork does all this in a measurement byte prepended to the header for now.

As to marten’s point: yes, for the time being experiments will have to synchronize the meaning among the endpoints and observers. We could use version negotiation to do this (indeed, Piet already blocked out 16 version numbers for our own experiments in the wiki for exactly this purpose), but I expect v1 to have a single meaning for these bits. Consider this an opportunity to exercise version negotiation (and, for observers, version tracking) during interop.

Cheers, 

Brian

Sent from my iPhone

> On 23 Nov 2017, at 07:41, Gorry Fairhurst <[email-masked]> wrote:
> 
>>>> On 2017-11-22, at 20:22, Ted Hardie <[email-masked] <mailto:[email-masked]>> wrote:
>>>> 
>> Fair point. So the first bit should be defined as "reserved for the spin bit experiment". The second bit, if we decide to have a second one, should be strictly designed as "reserved" for now.
>> 
>> -- Christian Huitema
> I'm preetty sure that if we had a set of bits that were declared in the header, the IETF could define a mechanism that offers the features we want. I think the IETF transport people are quite capable of proposing a mechanism(s) if we spend the time discussing the mechanisms, rather than whether the bit field can be present. Sure, I'd agree the choice of which to use could be experimental in the first release - and become "standardised" in later releases.
> 
> I do suggest we consider two bits for the "spin", rather than one, because of the ability to robustly measure in the presence of loss/reordering - the greatest value is a system that continues to offer useful data when the path is under stress, so that the mechanisms can be used to figure out how to fix/analyse problems. Maybe it may be useful to also be able to quantify loss/congestion/reodering (even when on the return path) ... but I do think we need to first look at robust latency measurement.
> 
> Gorry
> 
>

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 24 Nov 2017 22:06:42 +0100

Christian,

> ignored when receiving, MAY be used for experiments between consenting

Having some spare bits in the base header will clearly be useful. In 
addition to the discussion on the spin bit, let me also point that a 
proposed design for multipath quic also uses one of the spare bits in 
the public header

https://tools.ietf.org/html/draft-deconinck-multipath-quic-00

Olivier

-- 

------------------------------
DISCLAIMER.
This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. If you are not the intended recipient 
you are notified that disclosing, copying, distributing or taking any 
action in reliance on the contents of this information is strictly 
prohibited.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 27 Nov 2017 10:33:57 +1100

On Sat, Nov 25, 2017 at 8:06 AM, Olivier Bonaventure
<[email-masked]> wrote:
> proposed design for multipath quic also uses one of the spare bits in the
> public header
>

I don't think that this is a good design.  I think that we definitely
need to discuss the separation of the packet number space when it
comes to multipath, using the connection ID is superior to adding
another byte to the header.  For one, it's something we want to change
anyway.

---

From: Jana Iyengar <[email-masked]>
Date: Sun, 26 Nov 2017 22:37:00 -0800

+1 to what Lars said -- we can reserve a bit but if we do, we must GREASE
to ensure that this bit remains usable in the future. That said ...

If this is for experimental use, what precisely is the experiment that we
expect to get conducted? Bits if offered can and will be used, so what
potential outcomes are we expecting? Specifically, are we trying to figure
out how to use these bits, whether to use these bits, or how many bits are
needed? The answers to these questions will have bearing on what it means
to reserve one or more bits.

If the answer is "all of the above", then I'd like some more clarity on the
framing of the problem before reserving anything.

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Mon, 27 Nov 2017 16:03:10 +0900

hi Jana,

My take on this:

> On 27 Nov 2017, at 15:37, Jana Iyengar <[email-masked]> wrote:
> 
> +1 to what Lars said -- we can reserve a bit but if we do, we must GREASE to ensure that this bit remains usable in the future. That said ...
> 
> If this is for experimental use, what precisely is the experiment that we expect to get conducted? Bits if offered can and will be used, so what potential outcomes are we expecting? Specifically, are we trying to figure out how to use these bits, whether to use these bits, or how many bits are needed? The answers to these questions will have bearing on what it means to reserve one or more bits.
> 
> If the answer is "all of the above", then I'd like some more clarity on the framing of the problem before reserving anything.

It seems to me we have a concrete proposal on the table for exposing latency information, a privacy analysis showing negligible marginal risk, pretty clear indication of its utility in the Singapore and post-Singapore discussion, and initial experimental indications from at least one implementation that show that it does what it's supposed to. The single implementation we have (Piet's) hacks a "measurement byte" to the back of every header, short or long, which is not great for interop. It would be nice to be able to interop implementations of the spin bit (and spin-bit-consuming measurement tools) with some agreement as to which bit we're flipping, so we can do more exploration of the corner cases at a larger scale than is possible with a single forked implementation. There's a higher bar for wire-image-impacting changes, now, and it would be nice to collect larger-scale data on how the spin bit works concurrent with the work we're doing to pull together the requested document (the appearing-real-soon-now draft-trammell-quic-spin), and subsequent discussion thereof.

It also seems to me that there are several candidates for a second (or even third) measurement bit: some form of ConEx-like refeedback, some indication as to whether a sender is blocked (diagnostically useful in its own right, as well as fixing one of the spin bit corner cases), perhaps a second spin bit (though Piet's work to date indicates that looking at the bottom N bits of the packet number is better for correcting reordering error in spin bit measurement than a second bit)... here there is less clarity, so having reserved measurement bits for future experiments would be useful too.

There's a question as to how many bits: I'll note that if we go to varints for packet numbers -- which we really should -- then there are five bits free in the short header now. We'll probably want to reserve all of those for the future, regardless of how many of them eventually get used for measurability, so they'll need to be greased somehow anyway.

Mark and Lars also pointed out that we can define one or more of these bits for v1 fairly late in the game, especially for something like the spin bit which is specifically designed to integrate with none of the transport's machinery.

Cheers,

Brian
> 
> On Sun, Nov 26, 2017 at 3:33 PM, Martin Thomson <[email-masked]> wrote:

---

From: Olivier Bonaventure <[email-masked]>
Date: Mon, 27 Nov 2017 08:37:23 +0100

Martin,
> another byte to the header.  For one, it's something we want to change
The important point is to have different packet number spaces on the 
different paths. There are different ways to signal those number spaces.

The draft proposes an explicit signal which is present in the publich 
header of each packet. One benefit of this approach is that a 
load-balancing router could use this information to forward the packets 
belonging to different paths of a single QUIC session over different paths.

Using the connection id is a different way to distinguish different 
packet number spaces. In this case, there is no need for using space in 
the public header but we need to define QUIC frames to associate 
different connection ids to the same QUIC session. This seems possible 
as well, but in contrast with the above approach, a load-balancing 
router would not be able to easily load-balance the packets of a single 
QUIC session to different paths since there is no structure in the 
connection id.

There are tradeoffs in any multipath design. I'd suggest that when the 
WG starts to work on multipath we look at detailed proposals.

Olivier

--

---

From: Ian Swett <[email-masked]>
Date: Mon, 27 Nov 2017 09:42:56 -0500

Sorry to go off-topic, but...

I strongly agree with Martin that we want to use connection ID for routing,
and not add a path ID.  An explicit goal of QUIC multipath should be to
look as similar(or ideally be indistinguishable) from single path QUIC.
Adding a path ID makes it trivial to link paths, which creates a known
privacy issue and I believe is much more risky than any measurement bits
we've ever discussed exposing.

The details of how to divide up the packet number space can be discussed
later, but it's clear to me that's a viable solution, and QUIC already has
mechanisms for changing the connection ID mid-connection, so building upon
them is the right approach.

On Mon, Nov 27, 2017 at 2:37 AM, Olivier Bonaventure <
[email-masked]> wrote:

> Using the connection id is a different way to distinguish different packet
> number spaces. In this case, there is no need for using space in the public
> header but we need to define QUIC frames to associate different connection
> ids to the same QUIC session. This seems possible as well, but in contrast
> with the above approach, a load-balancing router would not be able to
> easily load-balance the packets of a single QUIC session to different paths
> since there is no structure in the connection id.
>
> There are tradeoffs in any multipath design. I'd suggest that when the WG
> starts to work on multipath we look at detailed proposals.
>
>
>
> Olivier
>
>
> --
>

---

From: Ted Hardie <[email-masked]>
Date: Mon, 27 Nov 2017 09:26:35 -0800

Quoting one tiny piece of this for emphasis:

On Sun, Nov 26, 2017 at 11:03 PM, Brian Trammell (IETF) <[email-masked]>
wrote:

> I'll note that if we go to varints for packet numbers -- which we really
> should -- then there are five bits free in the short header now. We'll
> probably want to reserve all of those for the future, regardless of how
> many of them eventually get used for measurability, so they'll need to be
> greased somehow anyway.
>

I think Brian has this exactly right:  if we reclaim these bits, then they
need to be greased as soon as possible.  Having the greasing algorithms
working and the peer behavior confirmed would be something I'd argue for as
a test for the first implementation draft after the change is made.

We've identified greasing as a strong requirement for avoiding early
ossification in QUIC, and I think this is a useful test of *that*,
regardless of the eventual use of the bits.

regards,

Ted