Subject: Spin bit decision

From: <[email-masked]>
Date: Tue, 2 Oct 2018 08:05:19 +0200

Hello,

After NYC, what are the remaining questions the WG is asking regarding the spin 
bit to educate the upcoming decision (Bangkok):

   (a) why is midpoint half-RTT measurement useful (in current practices) ?
   (b) is the spin bit up to the job?
   (c) isn't there any other way ?
   (d) other

?

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Lars Eggert <[email-masked]>
Date: Tue, 2 Oct 2018 09:51:33 +0300

Hi,

On 2018-10-2, at 9:05, [email-masked] wrote:
> After NYC, what are the remaining questions the WG is asking regarding the spin bit to educate the upcoming decision (Bangkok):
> 

the following is my personal feedback on this matter, i.e., I'm not speaking as chair:

I have little doubt that the single-bit spin bit variant is fit for purpose. I also think the DT has analyzed the privacy implications of the single-bit variant sufficiently for me to have no serious privacy concerns about the variant.

The key open question in my mind is this: on what fraction of the traffic would an operator need to see spinning and echoed bits in order to obtain useful network-wide or at least path specific latency information? In other words, how many browsers and servers-side deployments would need to opt in and spin or echo the bits? Would it be sufficient if one hyperscalar with a bowser and a server deployment opted in? Even if most of their traffic concentrated towards their peering links?

Regards,
Lars

---

From: <[email-masked]>
Date: Tue, 2 Oct 2018 10:28:41 +0200

On 10/02/18 08:51, Lars Eggert wrote:
>> After NYC, what are the remaining questions the WG is asking regarding the
>> spin bit to educate the upcoming decision (Bangkok):
>> 
>> (b) is the spin bit up to the job ? >> (c) isn't there any other way ?
>> (d) other >> ?
> 
> I have little doubt that the single-bit spin bit variant is fit for purpose.
> I also think the DT has analyzed the privacy implications of the single-bit
> variant sufficiently for me to have no serious privacy concerns about the
> variant.
> 
> The key open question in my mind is this: on what fraction of the traffic
> would an operator need to see spinning and echoed bits in order to obtain
> useful network-wide or at least path specific latency information? In other
> words, how many browsers and servers-side deployments would need to opt in
> and spin or echo the bits? Would it be sufficient if one hyperscalar with a
> bowser and a server deployment opted in? Even if most of their traffic
> concentrated towards their peering links?

Experience with debugging our networks in several countries shows that (1) all
segments may misbehave, and (2) sample size is very often an issue, in that some
problems are triggered by a non-obvious conjunction of parameters that is
extremely sensitive to sampling bias. So, the more support for measurement, the
better. TCP shines because clear headers are ubiquitous, not an opt-in debug
mode, nor a randomly enabled feature in X% of flows.

May I ask why this question is key ? Are you envisioning a situation where v1
would allow for the spin bit as a negotiated option instead of just requiring it
? Added complexity and inferior outcome: what is the motivation ?

---

From: Lars Eggert <[email-masked]>
Date: Tue, 2 Oct 2018 13:59:43 +0300

Hi,

again, as an individual:

> mode, nor a randomly enabled feature in X% of flows.

so what fraction of spin-enabled QUIC traffic would be sufficient? Maybe just a ballpark number; 1%, 10%, 100%?

> ? Added complexity and inferior outcome: what is the motivation ?

The WG hasn't really discussed yet what requirements level any inclusion of the spin bit into the main spec would have, and there could obviously be different levels of requirement for clients and servers.

But irrespective of what - if anything - the spec chooses to say, deployed implementations can obviously choose to do whatever they wish, since the spin bit is not affecting interoperability at all.

So for the spin bit to fulfill its intended purpose, at least some client deployments and at least some server deployments that spin and echo the bit are required. Enough on both sides so that a sufficient fraction of the traffic is spinning (which is what my first question was about.) 

Lars

---

From: Marcus Ihlar <[email-masked]>
Date: Tue, 2 Oct 2018 11:44:19 +0000

Hi,

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Lars Eggert
Sent: den 2 oktober 2018 13:00
To: [email-masked]
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: Spin bit decision

Hi,

again, as an individual:

On 2018-10-2, at 11:28, [email-masked] wrote:
> Experience with debugging our networks in several countries shows that
> (1) all segments may misbehave, and (2) sample size is very often an
> issue, in that some problems are triggered by a non-obvious
> conjunction of parameters that is extremely sensitive to sampling
> bias. So, the more support for measurement, the better. TCP shines
> because clear headers are ubiquitous, not an opt-in debug mode, nor a randomly enabled feature in X% of flows.

MI: It´s a difficult question to answer as there are different uses of the spin information ranging from active troubleshooting sessions for a small set of subscribers to passive monitoring on a large scale e.g for KPI dashboards.
>From a spin consumer perspective the bigger the fraction the better of course, but the pain thresholds would be different from usecase to usecase.
The active troubleshooting case would require either that a very large portion of traffic is spin-enabled or that it would be possible for a user to selectively enable the functionality.
The large scale monitoring case would likely be useful even if  only a few % of flows are spin-enabled.

> May I ask why this question is key ? Are you envisioning a situation
> where v1 would allow for the spin bit as a negotiated option instead
> of just requiring it ? Added complexity and inferior outcome: what is the motivation ?

Lars

---

From: Lars Eggert <[email-masked]>
Date: Tue, 2 Oct 2018 14:51:46 +0300

Hi,

> The large scale monitoring case would likely be useful even if  only a few % of flows are spin-enabled.

thanks, Marcus! That was what I was wondering about.

Lars

---

From: <[email-masked]>
Date: Tue, 2 Oct 2018 14:41:55 +0200

On 10/02/18 12:59, Lars Eggert wrote:
>  >> May I ask why this question is key ? Are you envisioning a situation where
>> v1 would allow for the spin bit as a negotiated option instead of just
>> requiring it ? Added complexity and inferior outcome: what is the
>> motivation ?
> 
> The WG hasn't really discussed yet what requirements level any inclusion of
> the spin bit into the main spec would have, and there could obviously be
> different levels of requirement for clients and servers.
> 
> But irrespective of what - if anything - the spec chooses to say, deployed
> implementations can obviously choose to do whatever they wish, since the spin
> bit is not affecting interoperability at all.

Well, if the spec says that en endpoint MUST copy of flip a bit, an implementor 
would be fairly ill-advised not do comply. And nothing prevents from adding a 
"does it spin ?" criterion to the interop test suite.

Clearly you now have this degree of freedom: weaken that MUST into a MAY or a 
SHOULD, separately for each role. But for what purpose ?

---

From: Lars Eggert <[email-masked]>
Date: Tue, 2 Oct 2018 15:55:50 +0300

Hi,

again, as an individual:

On 2018-10-2, at 15:41, <[email-masked]> <[email-masked]> wrote:
> Well, if the spec says that en endpoint MUST copy of flip a bit, an implementor would be fairly ill-advised not do comply.

sure, it may be "ill-advised", but in practice, it won't affect interop at all.

> And nothing prevents from adding a "does it spin ?" criterion to the interop test suite.

We can certainly consider this - it may give us some data about the current capabilities and configurations of the participating stacks.

> Clearly you now have this degree of freedom: weaken that MUST into a MAY or a SHOULD, separately for each role. But for what purpose ?

The point I'm trying to make is that it in practice doesn't matter if the spec says "MUST", "SHOULD" or "MAY", since it doesn't affect interop, and doesn't affect performance. Essentially, each stack can decide in isolation whether to spin, echo, do nothing, grease the bits, etc., no matter what language we put in the spec.

Lars

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 2 Oct 2018 05:59:38 -0700

Are there any concerns about impact and how to handle or ignore malicious
manipulation of the spin either by endpoint or a middlebox?

On 2 October 2018 at 14.56.16, Lars Eggert ([email-masked]) wrote:

Hi,

again, as an individual:

On 2018-10-2, at 15:41, <[email-masked]> <
[email-masked]> wrote:
> Well, if the spec says that en endpoint MUST copy of flip a bit, an
implementor would be fairly ill-advised not do comply.

sure, it may be "ill-advised", but in practice, it won't affect interop at
all.

> And nothing prevents from adding a "does it spin ?" criterion to the
interop test suite.

We can certainly consider this - it may give us some data about the current
capabilities and configurations of the participating stacks.

> Clearly you now have this degree of freedom: weaken that MUST into a MAY
or a SHOULD, separately for each role. But for what purpose ?

The point I'm trying to make is that it in practice doesn't matter if the
spec says "MUST", "SHOULD" or "MAY", since it doesn't affect interop, and
doesn't affect performance. Essentially, each stack can decide in isolation
whether to spin, echo, do nothing, grease the bits, etc., no matter what
language we put in the spec.

Lars

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Tue, 2 Oct 2018 15:01:40 +0200

Yes. See the last two paragraphs of section 6 of the current (TSVWG) full spin bit draft: https://tools.ietf.org/html/draft-trammell-tsvwg-spin-00.

> On 2 Oct 2018, at 14:59, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> 
> Are there any concerns about impact and how to handle or ignore malicious manipulation of the spin either by endpoint or a middlebox?
> 
> 
>

---

From: <[email-masked]>
Date: Tue, 2 Oct 2018 15:07:02 +0200

> We can certainly consider this - it may give us some data about the current capabilities and configurations of the participating stacks.

More than that: you seem to be implying that automated interop testing outweighs 
textual specification. No problem with that; let's focus on test definition then.

> The point I'm trying to make is that it in practice doesn't matter if the spec says "MUST", "SHOULD" or "MAY", since it doesn't affect interop, and doesn't affect performance. Essentially, each sta…

Sure. So, what is there to lose by saying "MUST" anyway ?

---

From: Lars Eggert <[email-masked]>
Date: Tue, 2 Oct 2018 17:00:17 +0300

Hi,

yet again as an individual.

On 2018-10-2, at 16:07, [email-masked] wrote:
> More than that: you seem to be implying that automated interop testing outweighs textual specification. No problem with that; let's focus on test definition then.

I'm not implying that at all, and I'm confused as to why you'd think so?

> Sure. So, what is there to lose by saying "MUST" anyway ?

This is a discussion the WG needs to have after there is consensus to merge the spin bit proposal.

The point I can't seem to be getting across is that irrespective of what the WG decides to require, there is no penalty for individual stacks at deployment time (or after) to do whatever they wish, since the spin bit by design has no impact on protocol operation and interop.

Lars

---

From: Nick Banks <[email-masked]>
Date: Tue, 2 Oct 2018 14:26:06 +0000

If we decide to make it a MUST, it seems to me that it would be pretty easy to add some logic to check that your peer is 'spinning' the bit. If they are not, after a couple of RTTs just to make sure, you could close the connection (add a new PEER_NOT_SPINNING error code). If just one or two major implementations implement the validation logic, then we could ensure interop and compliance for the most part.

- Nick

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Lars Eggert
Sent: Tuesday, October 2, 2018 7:00 AM
To: <[email-masked]> <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: Spin bit decision

Hi,

Lars

---

From: <[email-masked]>
Date: Tue, 2 Oct 2018 16:42:50 +0200

Exactly. It is only a question of deciding we want it.

---

From: Brian Trammell (IETF) <[email-masked]>
Date: Tue, 2 Oct 2018 16:45:11 +0200

Hi Marcus, Lars, all,

> On 2 Oct 2018, at 13:44, Marcus Ihlar <[email-masked]> wrote:
> 

(…note: not sure about any of the quoting below…)

> The large scale monitoring case would likely be useful even if  only a few % of flows are spin-enabled.

In any situation other than 100% participation, I think it will be necessary to have client opt-in be exposed up to the user for intraflow diagnostics using the spin bit.

(yes, once you’re opting in, quictrace gives you all that the spin bit does and more, but I don’t see it as a reasonable replacement for the spin bit: first, “send me a trace” scales way less well than automated “locking on” to an available one-bit signal; second, just exposing RTT is somewhat less risky than exposing the full operation of the transport to a partially-trusted operator. quictrace seems more like a debugging tool than an operational diagnostics tool to me. It’s a very cool one, and one we really need to standardize, hopefully after stripping the protobuf cruft off of it, but these questions are IMO completely separate…)

I’m more optimistic than Marcus is with respect to the utility of the spin bit for aggregate (link / path-level) measurements even with small amounts of deployment. It is relatively easy for a midpoint to recognize whether the bit spins or not, and to make those RTT samples available to the aggregate sample (or aggregate baseline / residual) on demand. So for a given aggregate, as long as there's one available active flow, the spin bit provides good RTT samples. This points to even one part in a thousand being useful in the Internet core.

IMO (and this is an opinion; I don't think anyone will have enough data to answer this question for real before Bangkok), as you say one "hyperscalar" turning the spin bit on -- even probabalistically, e.g. as Apple did (still does?) during transition to ECN-by-default on the client side -- would provide more than enough signal on the paths between the access networks of its users (most of them) and the network(s) through which it provides its services.

This would also be the case if one of the top-N CDNs and one major browser vendor did so (again, probabilistic activation suffices). The larger the set of the server networks implementing, the more coverage one would have for access and core network paths.

Cheers,

Brian

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 2 Oct 2018 14:48:04 +0000

>If we decide to make it a MUST, it seems to me that it would be pretty easy to
>add some logic to check that your peer is 'spinning' the bit. If they are not,
>after a couple of RTTs just to make sure, you could close the connection (add a
>new PEER_NOT_SPINNING error code). If just one or two major
>implementations implement the validation logic, then we could ensure
>interop and compliance for the most part.
>
>- Nick

A hypothetical spin stick.

With a cyncial endpoint operator hat on, I think the realisation spin bit benefits could be minimal. Leading to little incentive to implement spin bit. Using a stick, like causing connection termination, sounds like it will introduce fun and games for interop testing, and deployment on heterogenous real-world networks.

What's the carrot to offer a non-hyperscaler?

-----------------------------
http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and
may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in
error, please delete it from your system.
Do not use, copy or disclose the
information in any way nor act in reliance on it and notify the sender
immediately.
Please note that the BBC monitors e-mails
sent or received.
Further communication will signify your consent to
this.
-----------------------------

---

From: <[email-masked]>
Date: Tue, 2 Oct 2018 17:05:12 +0200

On 10/02/18 16:00, Lars Eggert wrote:
> 
> The point I can't seem to be getting across is that irrespective of what the
> WG decides to require, there is no penalty for individual stacks at
> deployment time (or after) to do whatever they wish, since the spin bit by
> design has no impact on protocol operation and interop.

It is getting across, but it is a very grim view of the standardization process 
itself. Are there many examples of MUST in RFCs that turned out to be ignored by 
implementations just because they were not part of the core semantics and hence 
not enforced by other peers ?

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 2 Oct 2018 15:13:25 +0000

Alexandre Ferrieux wrote:
>> The point I can't seem to be getting across is that irrespective of
>> what the WG decides to require, there is no penalty for individual
>> stacks at deployment time (or after) to do whatever they wish, since
>> the spin bit by design has no impact on protocol operation and interop.
>
>itself. Are there many examples of MUST in RFCs that turned out to be
>ignored by implementations just because they were not part of the core
>semantics and hence not enforced by other peers ?
>

One example is provided in RFC 6919 Section 1 [1].

[1] https://tools.ietf.org/html/rfc6919#section-1

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 2 Oct 2018 10:24:04 -0500

On Tue, Oct 02, 2018 at 05:05:12PM +0200, [email-masked] wrote:
> It is getting across, but it is a very grim view of the standardization
> process itself. Are there many examples of MUST in RFCs that turned out to
> be ignored by implementations just because they were not part of the core

I guess that this bit of https://tools.ietf.org/html/rfc4120 qualifies:

   Implementations of Kerberos and protocols based on Kerberos MUST NOT
   use insecure DNS queries to canonicalize the hostname components of
   the service principal names (i.e., they MUST NOT use insecure DNS
   queries to map one name to another to determine the host part of the
   principal name with which one is to communicate).

It's a local matter that the remote peer can't enforce, and "everybody"
is doing it, still.

-Ben

---

From: Lars Eggert <[email-masked]>
Date: Tue, 2 Oct 2018 19:41:54 +0300

Hi,

> This would also be the case if one of the top-N CDNs and one major browser vendor did so (again, probabilistic activation suffices). The larger the set of the server networks implementing, the more…

thanks, this is helpful!

Lars

---

From: Mike Bishop <[email-masked]>
Date: Tue, 2 Oct 2018 16:50:32 +0000

I don’t think it’s a grim view, I think it’s a pragmatic view.  Our role is to specify the things which are fundamental to the protocol and have to happen for peers to interoperate.  Mis-using normative language for other things is an abuse of the process -- that's kind of the point of RFC 6919.

RFC 2119 says:
In particular, [normative language] MUST only be used where it is actually required for interoperation or to limit behavior which has potential for causing harm (e.g., limiting retransmisssions)  For example, they must not be used to try to impose a particular method on implementors where the method is not required for interoperability.

Given that the spin bit is not “actually required for interoperation,” but is the very definition of “try to impose a particular method on implementors,” I’d say that RFC 2119 imperatives are unwarranted.

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of [email-masked]
Sent: Tuesday, October 2, 2018 8:05 AM
To: Lars Eggert <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: Spin bit decision

On 10/02/18 16:00, Lars Eggert wrote:

> the spin bit by design has no impact on protocol operation and interop.

It is getting across, but it is a very grim view of the standardization process itself. Are there many examples of MUST in RFCs that turned out to be ignored by implementations just because they were not part of the core semantics and hence not enforced by other peers ?

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.

Thank you.

---

From: Ted Hardie <[email-masked]>
Date: Tue, 2 Oct 2018 10:06:25 -0700

On Tue, Oct 2, 2018 at 9:51 AM Mike Bishop <[email-masked]> wrote:

> I don’t think it’s a grim view, I think it’s a pragmatic view.  Our role
> is to specify the things which are fundamental to the protocol and have to
> happen for peers to interoperate.  Mis-using normative language for other
> things is an abuse of the process -- that's kind of the point of RFC 6919.
>
>
>
> RFC 2119 says:
>
> In particular, [normative language] MUST only be used where it is actually
> required for interoperation or to limit behavior which has potential for
> causing harm (e.g., limiting retransmisssions)  For example, they must not
> be used to try to impose a particular method on implementors where the
> method is not required for interoperability.
>
>
>
> Given that the spin bit is not “actually required for interoperation,” but
> is the very definition of “try to impose a particular method on
> implementors,” I’d say that RFC 2119 imperatives are unwarranted.
>

That's not quite what we generally mean by "interoperation" here.  It's
quite common to have something say, in effect, "If you use X, you MUST do
Y.".  So, we could have a MUST in Section 2.3 of the draft, instead of the
current language:

   Each client and server MUST reset its spin value to zero when sending the
   first packet of a given connection with a new connection ID.  This
   reduces the risk that transient spin bit state can be used to link
   flows across connection migration or ID change.

That MUST doesn't say anything about when you use the spin bit, it simply
says that *if*  you use it, other implementations will expect this
behavior; if they get a new connection ID with a non-zero spin value, it
will be a protocol error.  That pattern of use is very common.

Ted

---

From: Ian Swett <[email-masked]>
Date: Tue, 2 Oct 2018 13:13:43 -0400

One piece of information related to the above discussion is whether
detecting who is and is not spinning is easy and reliable.

One of my concerns is unintentional mis-use of the signal, and if it can't
be detected reliably in a mixed environment, that means SHOULD is unwise
and argues for either not including it or making it a MUST.

On Tue, Oct 2, 2018 at 12:51 PM Mike Bishop <[email-masked]> wrote:

> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
> exploites ou copies sans autorisation. Si vous avez recu ce message par
> erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les
> pieces jointes. Les messages electroniques etant susceptibles d'alteration,
> This message and its attachments may contain confidential or privileged
> information that may be protected by law; they should not be distributed,
> used or copied without authorisation.
>

---

From: Mike Bishop <[email-masked]>
Date: Tue, 2 Oct 2018 17:50:44 +0000

I’d consider that to be “potential for causing harm.”  We’ve decided that correlation is a potential security risk, and failure to do that exposes you to the risk.

From: Ted Hardie <[email-masked]>
Sent: Tuesday, October 2, 2018 10:06 AM
To: Mike Bishop <[email-masked]>
Cc: [email-masked]; Lars Eggert <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: Spin bit decision

On Tue, Oct 2, 2018 at 9:51 AM Mike Bishop <[email-masked]<mailto:[email-masked]>> wrote:

RFC 2119 says:

That's not quite what we generally mean by "interoperation" here.  It's quite common to have something say, in effect, "If you use X, you MUST do Y.".  So, we could have a MUST in Section 2.3 of the draft, instead of the current language:

   flows across connection migration or ID change.
That MUST doesn't say anything about when you use the spin bit, it simply says that *if*  you use it, other implementations will expect this behavior; if they get a new connection ID with a non-zero spin value, it will be a protocol error.  That pattern of use is very common.

Ted

---

From: Marten Seemann <[email-masked]>
Date: Tue, 2 Oct 2018 11:07:30 -0700

> One of my concerns is unintentional mis-use of the signal, and if it
can't be detected reliably in a mixed environment, that means SHOULD is
unwise and argues for either not including it or making it a MUST.

The problem is, even if we decide to make it a MUST, a middlebox still
needs to distinguish between QUIC traffic and other UDP traffic. We're not
exposing any unambiguous signal to the path that a particular flow actually
is QUIC traffic, so middlebox vendors will have to come up with some kind
of heuristic to identify QUIC.
My fear is that if all implementations actually implement the spin bit, and
middleboxes learn to rely on that signal (and enforce compliance), we might
do great harm to other UDP based protocols (current and future) which
happen to use the same code points that middleboxes came to rely on.

Coming back to Alexandre's question, I think here is little an experiment
could do to assuage that fear. This is just another symptom of exposing
this signal in the wrong OSI layer.

On Tue, Oct 2, 2018 at 10:18 AM Ian Swett <ianswett=
[email-masked]> wrote:

>> In particular, [normative language] MUST only be used where it is
>> actually required for interoperation or to limit behavior which has
>> potential for causing harm (e.g., limiting retransmisssions)  For example,
>> they must not be used to try to impose a particular method on implementors
>> where the method is not required for interoperability.
>>
>>
>>
>> Given that the spin bit is not “actually required for interoperation,”
>> but is the very definition of “try to impose a particular method on

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Tue, 2 Oct 2018 22:17:22 +0200

hi Marten,

> On 2 Oct 2018, at 20:07, Marten Seemann <[email-masked]> wrote:
> 
> > One of my concerns is unintentional mis-use of the signal, and if it can't be detected reliably in a mixed environment, that means SHOULD is unwise and argues for either not including it or making it a MUST.
> 
> The problem is, even if we decide to make it a MUST, a middlebox still needs to distinguish between QUIC traffic and other UDP traffic. We're not exposing any unambiguous signal to the path that a particular flow actually is QUIC traffic, so middlebox vendors will have to come up with some kind of heuristic to identify QUIC.

Will have to, have already, will continue to. Neither wishful thinking nor stern lectures in RFCs nor the suitable application of strong cryptography will make the *perceived* requirement to distinguish QUIC from not-QUIC go away (without comment on whether this perception is accurate).

> My fear is that if all implementations actually implement the spin bit, and middleboxes learn to rely on that signal (and enforce compliance), we might do great harm to other UDP based protocols (current and future) which happen to use the same code points that middleboxes came to rely on.

Notwithstanding that tracking spin edges to determine whether something is QUIC or not would be a heuristic so heavyweight that widespread deployment of spin-classification would be unlikely, IMO the correct design in the face of this fear has nothing to do with the spin bit at all -- if we care about keeping middleboxes from breaking things accidentally, then we should give these middleboxes something to grab on to that *won't* cause collateral damage: QUIC's invariants should be designed in such a way that a flow as defined by 5-tuple, if not a packet without the context of such a 5-tuple, can be unambiguously and obviously classified as QUIC. We kind of got close to this during the New York interim during the first octet discussion with the "QUIC bit"; maybe we should make that explicit.

> Coming back to Alexandre's question, I think here is little an experiment could do to assuage that fear. This is just another symptom of exposing this signal in the wrong OSI layer.

(as an aside, as to whether transport-experienced end-to-end RTT measurability is a transport or network layer function, see also section 5.1 of draft-trammell-tsvwg-spin.)

Cheers,

Brian

> 
> On Tue, Oct 2, 2018 at 10:18 AM Ian Swett <ianswett=[email-masked]> wrote:
> One piece of information related to the above discussion is whether detecting who is and is not spinning is easy and reliable.
> 
> One of my concerns is unintentional mis-use of the signal, and if it can't be detected reliably in a mixed environment, that means SHOULD is unwise and argues for either not including it or making it a MUST.
>

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Tue, 2 Oct 2018 23:10:43 +0200

hi Ian, all,

Perhaps we can exploit the asymmetry inherent in QUIC (there are servers and clients), the spin bit (the server reflects, the client inverts), and the desire to participate or not (ISTM that most of the discussion we've had about opting out have focused on client opt-out, based either on the principle of the thing and/or a suspicion that the signal is useless) to resolve this discussion.

A proposal:

1. Servers MUST reflect the spin value. Clients MAY treat an unambiguously detected non-reflected bit as a protocol error.

2a. Clients SHOULD invert the spin value, unless [EDNOTE: someone other than me should write convincing reasons for non-implementation here], or the client has been configured not to spin, or to use some other client behavior.

-OR-

2b. Clients set the spin bit value as they see fit, provided that this behavior is consistent per connection ID. In order to provide RTT information to the path, clients MAY invert the spin value (per reference to a separate RFC that looks a lot like draft-tsvwg-quic=spin, to be published by QUIC, TSVWG, or IPPM).

3. Servers and clients MUST NOT expose spin state from one Connection ID to another Connection ID.

1 is intended to drive deployment through server defaults. Yes, one can argue that the protocol error here is artificial (and the proposal explicitly acknowledges that through the client MAY error out), but the client can always opt out by clamping to a random value per CID.

The 2a variant is essentially a 2119-compliant way of saying "we'd really like it if you implemented the spin bit", leaving the question of client defaults (opt in versus opt out) up to the client implementors.

The 2b variant is designed to allow for future or parallel client-side behaviors other than spinning to support different measurement goals. For example, using the adaptive square wave proposed for on-path loss measurement (by either Mikkel or Kazuho, I believe) at the client with server reflection would allow accumulation of loss on the upstream as well as the downstream path, as well as upstream (toward the server) RTT measurement. Obviously, the set of these deployed in the wild would have to be kept small in order to allow a measurement device to heuristically determine which signal was in use given a spinbit stream.

3 prevents spin state from leaking across CIDs, reducing the (already tiny but nonzero) linkability surface.

(Obviously, the above needs wordsmithing, but these are the broad lines.)

Cheers,

Brian

> On 2 Oct 2018, at 19:13, Ian Swett <ianswett=[email-masked]> wrote:
>

---

From: Marten Seemann <[email-masked]>
Date: Tue, 2 Oct 2018 14:23:10 -0700

> 1. Servers MUST reflect the spin value. Clients MAY treat an
unambiguously detected non-reflected bit as a protocol error.

This might works for classical client-server deployment, but in the p2p use
case, the terms "client" and "server" are a bit misleading, and we're just
using them for simplicity in the draft. Both "client" and "server" are
really just peers on the network, one of which happened to initiate the
connection and the other one accepted the connection.
If we acknowledge that there are convincing reasons to opt-out of exposing
the spin bit, both sides need to have the ability to do so.

On Tue, Oct 2, 2018 at 2:11 PM Brian Trammell (IETF) <[email-masked]>
wrote:

> hi Ian, all,
>
> Perhaps we can exploit the asymmetry inherent in QUIC (there are servers
> and clients), the spin bit (the server reflects, the client inverts), and
> the desire to participate or not (ISTM that most of the discussion we've
> had about opting out have focused on client opt-out, based either on the
> principle of the thing and/or a suspicion that the signal is useless) to
> resolve this discussion.
>
>
> A proposal:
>
> 1. Servers MUST reflect the spin value. Clients MAY treat an unambiguously
> detected non-reflected bit as a protocol error.
>
> 2a. Clients SHOULD invert the spin value, unless [EDNOTE: someone other
> than me should write convincing reasons for non-implementation here], or
> the client has been configured not to spin, or to use some other client
> behavior.
>
> -OR-
>
> 2b. Clients set the spin bit value as they see fit, provided that this
> behavior is consistent per connection ID. In order to provide RTT
> information to the path, clients MAY invert the spin value (per reference
> to a separate RFC that looks a lot like draft-tsvwg-quic=spin, to be
> published by QUIC, TSVWG, or IPPM).
>
> 1 is intended to drive deployment through server defaults. Yes, one can
> argue that the protocol error here is artificial (and the proposal
> explicitly acknowledges that through the client MAY error out), but the
> client can always opt out by clamping to a random value per CID.
>
> The 2a variant is essentially a 2119-compliant way of saying "we'd really
> like it if you implemented the spin bit", leaving the question of client
> defaults (opt in versus opt out) up to the client implementors.
>
> The 2b variant is designed to allow for future or parallel client-side
> behaviors other than spinning to support different measurement goals. For
> example, using the adaptive square wave proposed for on-path loss
> measurement (by either Mikkel or Kazuho, I believe) at the client with
> server reflection would allow accumulation of loss on the upstream as well
> as the downstream path, as well as upstream (toward the server) RTT
> measurement. Obviously, the set of these deployed in the wild would have to
> be kept small in order to allow a measurement device to heuristically
> determine which signal was in use given a spinbit stream.
>

---

From: <[email-masked]>
Date: Wed, 3 Oct 2018 00:12:48 +0200

On 10/02/18 18:50, Mike Bishop wrote:
> Given that the spin bit is not “actually required for interoperation,” but is 
> the very definition of “try to impose a particular method on implementors,” I’d 
> say that RFC 2119 imperatives are unwarranted.

My naive belief was that we were in the process of (1) deciding whether we'd 
like the spin bit to be widespread; and later, (2) how to get there given the 
fact that implementors are moderately only constrained by  RFCs.

But your statements above seem to indicate that (1)=NO.
Is this consensual already ? If yes, why are we discussing at all ?

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 3 Oct 2018 08:31:28 +0900

> 1. Servers MUST reflect the spin value. Clients MAY treat an unambiguously detected non-reflected bit as a protocol error.

I oppose to using MUST for the server, because some *servers* might
not be willing to expose the path RTT, for example when it is running
hidden behind a UDP proxy.

I also do not think that maintaining the spin-bit for each
"connection" works. My understanding is that it should be per-5-tuple
because multiple QUIC connections can be coalesced onto single UDP
port on both sides, and because it is impossible for an observer to
track the pair of CIDs identifying the same connection across CID
changes.

I have opened https://github.com/quicwg/base-drafts/issues/1828 to
track the issue.

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 3 Oct 2018 10:46:47 +0900

> One piece of information related to the above discussion is whether detecting who is and is not spinning is easy and reliable.

+1.

>From practical standpoint, it is very difficult to design a mechanism
that detects if the peer is not sending false information using the
spin bit. We need to make sure that a middlebox cannot disrupt a
connection by reordering packets, leading an endpoint to believe that
the peer is faking the spin bits. The design needs take care of
multithreaded (or clustered) endpoints sending packets.

I am not against adopting the spin bit proposal, but I strongly prefer
having the possibility to *not* use the spin bits for the endpoints
that want to hide the latency. For example for the endpoints could for
example be behind a NAT or a UDP proxy.

OTOH, I think it is fine to expose a signal that indicates if the spin
bit is turned on. We have an unused bit in the short header :-)

-- 
Kazuho Oku

---

From: <[email-masked]>
Date: Wed, 3 Oct 2018 09:31:10 +0200

On 10/03/18 01:31, Kazuho Oku wrote>
> I oppose to using MUST for the server, because some *servers* might not be
> willing to expose the path RTT, for example when it is running hidden behind
> a UDP proxy.

At last a concrete argument, thanks ;)
But why would exposing the RTT be harmful in a NAT/UDP-proxy scenario ?

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 3 Oct 2018 00:45:06 -0700

Great firewall censorship. VPN proxy to bypass streaming restrictions by
country. Hackers that proxy traffic via a legitimate cooperation. Internet
access providers prioritising local traffic. High latency onion routing
traffic.

Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu
ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.

This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and
delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Wed, 3 Oct 2018 09:58:01 +0200

hi Kazuho,

> On 3 Oct 2018, at 01:31, Kazuho Oku <[email-masked]> wrote:
> 

s/MUST/SHOULD, unless.../ is an acceptable change, as long as we have a nice tight box around the unless. The tradeoff is that every caveat we put on the server-side reduces server-side deployment, which reduces the chances that a cooperating server and a cooperating client meet up.

I'm also quite sympathetic to Marten's argument that "server" and "client" are more arbitrary in P2P situations. However, the asymmetry of the spinbit is pretty far down the list of asymmetries (stream handling, handshake, TLS certs, etc) that I'd want to see optionally addressed in a QUIC version i2 (is this the terminology we're using now?)

Backing off the MUST for now for such situations is IMO a good tradeoff, though, especially since we only need fractions of a percent of deployment to start seeing useful signal for baseline/anomaly measurement of large aggregates.

> track the pair of CIDs identifying the same connection across CID

Well, it should be per 5-tuple, but it should also change when CIDs change. Dumping state on any CID change is the easiest way I know to break linkability, and the most obvious way to do this when you have CID concurrency is to keep state per CID.

That might be too far into the implementation weeds for the spec though... the language needs to be clear (and clearer than in my initial proposal, which is one reason that's a list message and not a PR ;) )

> I have opened https://github.com/quicwg/base-drafts/issues/1828 to

Thanks, will follow discussion there as well.

Cheers,

Brian

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Wed, 3 Oct 2018 09:58:11 +0200

hi Mikkel,

I think we're having trouble with the many meanings of the word "proxy" here. It can mean at least three things in this context, each of which has different implications for RTT measurement:

(1) Web proxying. The web proxy terminates transport connections, and as such the spin bit would only expose the RTT on the segment between the proxy and its Internet-side peer. Since the whole point of the proxy from a privacy standpoint is to make traffic look like it's coming from the proxy, exposing this RTT is not an issue.

(2) Transport-terminating proxying. Here the proxying happens at the stream, as opposed to the request level. For spin bit measurement, this is equivalent to (1) above. (Note that this is widely deployed for TCP in the form of transparent back-to-back proxies, which won't work in QUIC since the security association protecting the transport is end-to-end.)

(3) Proxy-via-NAT. Here the proxy simply rewrites UDP addresses; in this case, since the transport association is not terminated, the spin bit will expose full end-to-end RTT. Indeed, in this case, endpoints may want to disable RTT measurement.

> On 3 Oct 2018, at 09:45, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> 
>> 
> Great firewall censorship.

Say more here? I don't see the relevance.

> VPN proxy to bypass streaming restrictions by country.

AFAIK most/all of these are of type (1) and (2), so not relevant to the discussion.

> Hackers that proxy traffic via a legitimate cooperation.

For web traffic, at least, this would probably be of type (1).

> Internet access providers prioritising local traffic.

This can and always will be done via other identifiers, so even if RTT is exposed here, it is of limited utility/threat (depending on which team you're rooting for).

> High latency onion routing traffic.

Necessarily always terminates transport. However, the fundamental design properties of circuit-based anonymity systems do mean that if you're doing something like running a VPN tunnel over ToR (for.. some reason?), then you should turn off the spin bit for traffic within that tunnel. Client opt-outs seem to be mostly sufficient for this, though.

Cheers,

Brian

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 3 Oct 2018 01:18:32 -0700

You are right that there are other obstacles to prevent detection that just
RTT.

You can imagine an alternative DNS service and router system that send
traffic in one direction and then forwards traffic into the general
backbone. So you have a person in Asia connect to a US university that has
router that forwards packets out into the general internet without
terminating anything as a sort of extended NAT box. I might be missing
something here, but that is my train of thoughts. The latency is not hiding
the distance between the two continents but it can reveal detectable
anomalies in traffic. More generally, you could apply machine learning to
RTT - or to any other observable parameter.

I’m not really against the spin bit and I agree that there are often better
ways to achieve such attacks, but dropping the spin bit is a really easy
first line of defence. Hence, for the cases I listed you cannot expect
cooperation on the spin bit even if the attack is unlikely.

On 3 October 2018 at 09.58.12, Brian Trammell (IETF) ([email-masked])
wrote:

hi Mikkel,

I think we're having trouble with the many meanings of the word "proxy"
here. It can mean at least three things in this context, each of which has
different implications for RTT measurement:

(1) Web proxying. The web proxy terminates transport connections, and as
such the spin bit would only expose the RTT on the segment between the
proxy and its Internet-side peer. Since the whole point of the proxy from a
privacy standpoint is to make traffic look like it's coming from the proxy,
exposing this RTT is not an issue.

(2) Transport-terminating proxying. Here the proxying happens at the
stream, as opposed to the request level. For spin bit measurement, this is
equivalent to (1) above. (Note that this is widely deployed for TCP in the
form of transparent back-to-back proxies, which won't work in QUIC since
the security association protecting the transport is end-to-end.)

(3) Proxy-via-NAT. Here the proxy simply rewrites UDP addresses; in this
case, since the transport association is not terminated, the spin bit will
expose full end-to-end RTT. Indeed, in this case, endpoints may want to
disable RTT measurement.

> VPN proxy to bypass streaming restrictions by country.

AFAIK most/all of these are of type (1) and (2), so not relevant to the
discussion.

> Internet access providers prioritising local traffic.

This can and always will be done via other identifiers, so even if RTT is
exposed here, it is of limited utility/threat (depending on which team
you're rooting for).

> High latency onion routing traffic.

Necessarily always terminates transport. However, the fundamental design
properties of circuit-based anonymity systems do mean that if you're doing
something like running a VPN tunnel over ToR (for.. some reason?), then you
should turn off the spin bit for traffic within that tunnel. Client
opt-outs seem to be mostly sufficient for this, though.

Cheers,

Brian

>> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
recu ce message par erreur, veuillez le signaler

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 3 Oct 2018 08:38:24 +0000

Nice summary Brian.

Thinking about tunneling QUIC in QUIC and proxy chains, we might end up with multiple layered QUIC security contexts, with the innermost layer being the end-to-end QUIC connection that is used for an actual application layer. Do you think this would benefit from multiple spin bits? Is that covered by the cases you already set out?

Lucas
________________________________________
From: QUIC [[email-masked]] on behalf of Brian Trammell (IETF) [[email-masked]]
Sent: 03 October 2018 08:58
To: Mikkel Fahnøe Jørgensen
Cc: [email-masked]; Mike Bishop; Ian Swett; Lars Eggert; IETF QUIC WG; Kazuho Oku
Subject: Re: a proposed way forward was Re: Spin bit decision

hi Mikkel,

Cheers,

Brian

---

From: <[email-masked]>
Date: Wed, 3 Oct 2018 12:04:45 +0200

On 10/03/18 09:58, Brian Trammell (IETF) wrote:
> Backing off the MUST for now for such situations is IMO a good tradeoff,
> though, especially since we only need fractions of a percent of deployment to
> start seeing useful signal for baseline/anomaly measurement of large
> aggregates.

If the consensus is that we must allow for such situations, then there are two 
possibilities:

  (a) weak spec language (MAY WISH TO or similar) => many implementations will 
simply drop it

  (b) negotiated option where the negotiation mechanism is mandatory

In the vein of (b), Christian suggested offline to introduce negotiation to 
allow for experimentation of the remaining two reserved bits. Then may be we can 
synthesize both ideas by the following proposal:

  - in the first few exchanges of the 5-tuple, use the three bits for option 
negotiation

  - then use them as defined by the selected option

Example encodings:

  000 : nothing
  001 : spin bit alone : S00
  010 : spin bit + VEC : SVV
  ... : other extensions

The negotiation mechanism allows both endpoints to force 000.
And since it is in the clear first byte, it allows on-path observers to identify 
the option without resorting to heuristics; this helps in the case of a small 
support ratio.

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 3 Oct 2018 22:32:06 +0900

2018年10月3日(水) 16:58 Brian Trammell (IETF) <[email-masked]>:
> s/MUST/SHOULD, unless.../ is an acceptable change, as long as we have a nice tight box around the unless.

My current thinking is that I would not oppose to having a simple
"SHOULD", which is IMO stronger than SHOULD with exceptions.

Regarding the "unless..." clause, I think all we can state is the
general rule; i.e. that an endpoint not willing to expose RTT needs to
opt-out from using spin bit. We could provide some examples of cases
where opting-out is desirable, but I do not think we could come up
with an exhaustive list of the cases.

> The tradeoff is that every caveat we put on the server-side reduces server-side deployment, which reduces the chances that a cooperating server and a cooperating client meet up.

I agree. Generally speaking, I think it is a good idea to specify a
tool that network operators can use for troubleshooting. And spin bit
looks simple enough to implement. Even for a person like me who does
not place Spin bit a high priority in his/her ToDo list, it is good to
have a mechanism that the person can run for help when facing trouble.

> Well, it should be per 5-tuple, but it should also change when CIDs change. Dumping state on any CID change is the easiest way I know to break linkability, and the most obvious way to do this when…

I am not sure if you need to dump the spin bit state when changing
CID, because it would be obvious to the observer that a flow (that
consists of multiple connections) continue running on the same
5-tuple.

As I have stated in my previous mail and on the GH issue, it is
practical for an observer to assume that QUIC connections sharing one
5-tuple are between two machines, even in case NAT or UDP proxy is
involved. That means that RTT does not change across CID changes, and
therefore to me it seems that resetting the spin bit state is
unnecessary.

Regarding breaking linkability when coalescing multiple QUIC
connections, my understanding is that the practical method is to
change CID for all the connections at once. Doing so would hide each
connection behind the anonymity set of N connections where N is the
number of connections being coalesced. I think that is the most we can
achieve.

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 3 Oct 2018 22:55:16 +0900

> (2) Transport-terminating proxying. Here the proxying happens at the stream, as opposed to the request level. For spin bit measurement, this is equivalent to (1) above. (Note that this is widely de…

FWIW, IIUC TCP proxies that forward HTTPS traffic exist in the market.
At the very least, we care of such proxies in the ESNI draft [1]. The
practical merit of using such proxy is to mitigate DDoS attacks in an
application-protocol-agnostic way, but I would not be surprised if
some of the users are also interested in hiding the true location of
the server.

As you correctly note, such proxy services do not leak the end-to-end
RTT, because TCP is terminated by the proxy, but for QUIC the
transport becomes end-to-end, and it would essentially leak the RTT
between the proxy and the hidden server.

In my view, this should be considered as a degradation in terms of
privacy preservation from TCP to QUIC. I am opposed to having such
degradation, even though it could be considered as a niche. Hence my
argument for SHOULD rather than MUST.

[1] https://tools.ietf.org/html/draft-ietf-tls-esni-01#section-3.1

-- 
Kazuho Oku

---

From: Mike Bishop <[email-masked]>
Date: Wed, 3 Oct 2018 18:41:34 +0000

Not at all -- whether we'd like it to be widespread behavior is a separate question from whether it's a requirement to successfully interoperate.

-----Original Message-----
From: [email-masked] <[email-masked]> 
Sent: Tuesday, October 2, 2018 3:13 PM
To: Mike Bishop <[email-masked]>; Lars Eggert <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: Re: Spin bit decision

> implementors,” I’d say that RFC 2119 imperatives are unwarranted.

My naive belief was that we were in the process of (1) deciding whether we'd like the spin bit to be widespread; and later, (2) how to get there given the fact that implementors are moderately only constrained by  RFCs.

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 4 Oct 2018 18:50:20 +0900

Hi,

Thank you for starting the discussion specific to how we could
possibly negotiate the use of spin bits.

Regarding the topic, my preference goes to using the version number
field of the long header for negotiating the presence of spin bits, or
any additional signal being exposed to the network.

For example, QUICv1 without spin bit could use version 0x101, and v1
with spin bit could use 0x102. In the future, we can assign a
different version number for QUICv1 with multiple spin + VEC (or
people interested in testing the feature can designate a private
version number even before that).

I see the following benefits to using the version field as a method to
negotiate the use of spin bits.

1. One of the concern regarding spin bits has been that they could
lead to ossification. Spin bits are not part of the Invariants, but if
the observation tools start looking at those bits without checking the
version number field, we will have the pressure to not change how the
bits are used. Using the version number to indicate the presence of
spin bits is the best approach to resolve such concern.

2. We have hoped to roll out multiple versions of QUIC in a small
interval so that the version number field does not get ossified. The
best solution is obviously to roll out two flavors of the protocol
that uses different version numbers at the same time. We will have a
real-world use of version negotiation from day one, because we are in
a condition where implementors have different opinions on if they
should support spin bit :-)

The blocker to this approach has been version negotiation requiring an
additional round-trip, but that is going to be resolved by #1755.

-- 
Kazuho Oku

---

From: <[email-masked]>
Date: Thu, 4 Oct 2018 12:46:32 +0200

Thanks Kazuho. Using version numbers directly was an obvious choice, but so far 
I was discouraged by strong language in the spec, saying 0x00000001 was the 
unavoidable point of convergence after all 0xFF0000XX experiments. So in essence 
you're proposing to weaken that a bit, right ?

If so, how do you envision segmenting the version number space ?

  - direct enumeration:

	0x00000001 = v1dry
	0x00000002 = v1 + spinbit
	0x00000003 = v1 + spinbit + VEC
	
  - low order bits for options:

	0x000001XX = v1 + option XX
	0x000002YY = v2 + option YY

Also, are we sure that this explosion won't weigh on the new VN scheme, inducing 
slower convergence due to longer lists on either side ?

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 5 Oct 2018 06:40:24 +0900

> you're proposing to weaken that a bit, right ?

I am not sure if we have reached consensus that QUICv1 should use
0x00000001 as the one and only number for the version number field.

My understanding is that it is simply written as such because I-D is a
draft of a RFC. A RFC needs to clarify the value of version number
field it uses, and I-D needs to have a paragraph that amends on how
the draft-versions uses the field.

>         0x000002YY = v2 + option YY

I do not think we can segment the version number space, because IMO it
essentially means making spin bits an invariant. My understanding is
that spin bits is something specific to v1.

QUICv1 might use different values on the version number field to
represent if spin bits is (or other signals are) available, but that
does not mean that v2 will expose the same signals or they would use
the version number field for indicating that.

If it is the case that the use of 0x1xx for V1 might give people the
impression that the lower bits are used to indicate the availability
of unencrypted signals across multiple versions, I think starting from
0x00000001 (i.e. the one you describe as "direct enumeration") would
be preferable.

> slower convergence due to longer lists on either side ?

I do not think that is an issue for v1, because what currently at
stake is if we should assign one or two extra version numbers (i.e.
one for spin bit, and possibly another for spin+vec as a separate
draft).

If folks agree to expose other signals that requires a "configuration"
to be exposed as well, then they can define a new version number that
introduces a new field to the long header format for exposing the
"configuration."

-- 
Kazuho Oku

---

From: Gabriel Montenegro <[email-masked]>
Date: Thu, 4 Oct 2018 21:53:14 +0000

We support explicit signals provided there are no additional privacy implications and the performance cost is negligible. We know this is true today for the spin bit. Whereas we cannot make any future product commitments, we are likely to include default support for the spin bit in our client and server QUIC implementation if it is part of the QUIC specification.

---

From: <[email-masked]>
Date: Thu, 4 Oct 2018 23:55:28 +0200

> the draft-versions uses the field.

OK, thanks for the clarification.

> 0x00000001 (i.e. the one you describe as "direct enumeration") would

Crystal clear, and agreed:)

> introduces a new field to the long header format for exposing the

OK. So, wrapping up, and abstracting away for now the bit layout of the version 
number by using symbolic names, we'd start (right after Bangkok) with something 
like:

	- "v1"     = v1 without extensions
	- "v1s"    = v1 with spin bit only
	- "v1svv"  = v1 with spin bit and VEC
	- "v1sqr"  = v1 with spin bit and loss bits Q and R
	- etc (?)

One question though: what part of the version negotiation exchange is in clear ?
IOW, would the selected version always be exposed to the path, or should 
spin-consumers rely on heuristics to guess the variant in use ?

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 5 Oct 2018 07:15:54 +0900

>         - "v1sqr"  = v1 with spin bit and loss bits Q and R

Yes.

We will have "v1" and "v1s", assuming that we would agree to ship the
one-bit "spin bit" approach as part of QUICv1. People will also have
the freedom for using more bits for the flavors (e.g. "v1svvv",
"v1sqr").

> spin-consumers rely on heuristics to guess the variant in use ?

My understanding is (assuming #1755 gets merged):
* the version number field of a long header would always be "v1" for
the first Initial sent from the client
* the value would be the negotiated version for exchanges that follows that
* there is no version number field in the short header

Therefore, for each path identified the 5-tuple, an observer would
remember the version number of the last long header packet that it has
seen (or it could remember the version number of any Handshake
packet), and use that to determine what is being exposed on the wire.
No heuristics is required.

OTOH, because version number field does not exist in the short header,
an observer cannot determine if any signal is exposed for a connection
after it migrates to a 5-tuple, or for a connection that has been
alive before starting the observation. Use of heuristics would be the
only way for such cases.

-- 
Kazuho Oku

---

From: Mike Bishop <[email-masked]>
Date: Thu, 4 Oct 2018 22:40:02 +0000

So long as this is the only thing being negotiated, it's not too bad.  If we want to negotiate other things, it quickly gets out of hand.  One could plausibly argue that the spin bit is special in that we actually want parties observing the transaction to understand which option has been selected.

It's important to remember, though, that this is information you get only at the beginning of a new connection.  If there's migration or NAT rebinding, it's not guaranteed (and in fact, not intended) than you can correlate it back to a particular previous handshake.  You'll still either need heuristics to identify whether the spin bit is actually in use, or you'll need to ignore any traffic for which you didn't see the handshake.

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of [email-masked]
Sent: Thursday, October 4, 2018 3:47 AM
To: Kazuho Oku <[email-masked]>
Cc: Brian Trammell <[email-masked]>; IETF QUIC WG <[email-masked]>; Christian Huitema <[email-masked]>
Subject: Re: Spin bit as a negotiated option

Thanks Kazuho. Using version numbers directly was an obvious choice, but so far I was discouraged by strong language in the spec, saying 0x00000001 was the unavoidable point of convergence after all 0xFF0000XX experiments. So in essence you're proposing to weaken that a bit, right ?

  - direct enumeration:

	0x00000001 = v1dry

Also, are we sure that this explosion won't weigh on the new VN scheme, inducing slower convergence due to longer lists on either side ?

On 10/04/18 11:50, Kazuho Oku wrote:
>> > Backing off the MUST for now for such situations is IMO a good 
>> > tradeoff, though, especially since we only need fractions of a 
>> > percent of deployment to start seeing useful signal for 
>> > baseline/anomaly measurement of large aggregates.
>>
>>   (a) weak spec language (MAY WISH TO or similar) => many 
>> implementations will simply drop it
>>
>> In the vein of (b), Christian suggested offline to introduce 
>> negotiation to allow for experimentation of the remaining two 
>> reserved bits. Then may be we can synthesize both ideas by the following proposal:
>>
>>   - in the first few exchanges of the 5-tuple, use the three bits for 
>> option negotiation
>>
>> And since it is in the clear first byte, it allows on-path observers 
>> to identify the option without resorting to heuristics; this helps in 
>> the case of a small support ratio.
>>
>>
>>
>>
>> _____________________________________________________________________
>> ____________________________________________________
>>
>> exploites ou copies sans autorisation. Si vous avez recu ce message 
>> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or 
>> privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.

---

From: Marcus Ihlar <[email-masked]>
Date: Fri, 5 Oct 2018 10:15:13 +0000

Hi,
I like this approach since it allows us to disregard non-spinning flows by simply observing the initial handshake. This saves us quite some work. 
On the heuristics part, we cannot get away from it, even if we see the spin version in the initial handshake as there´s no guarantee that an endpoint will set the bit properly anyway.
However, for the version where the intent to use spinbit is explicit it actually makes sense for an endpoint to treat non-participating behavior as a protocol error.

Marcus

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of [email-masked]
Sent: Thursday, October 4, 2018 3:47 AM
To: Kazuho Oku <[email-masked]>
Cc: Brian Trammell <[email-masked]>; IETF QUIC WG <[email-masked]>; Christian Huitema <[email-masked]>
Subject: Re: Spin bit as a negotiated option

  - direct enumeration:

	0x00000001 = v1dry

---

From: Marten Seemann <[email-masked]>
Date: Fri, 5 Oct 2018 12:33:11 +0200

The problem with using a specific version for implementations that use and
don’t use the spin bit is that middleboxes can effectively enforce usage of
the spin bit version by blackholing every packet advertising a version that
doesn’t use the spin bit. If only a few middleboxes do that, this would
force *all* implementations to avoid these versions.
Implementations that don’t want to expose their RTT would then have to
negotiate a version that is supposed to use the spin bit, but then grease
the corresponding bits in order to preserve their privacy. Of course, this
would only work if the peer doesn’t enforce compliance. Otherwise, we’ll
end up in a world where it’s basically impossible to opt-out of the
spin-bit.

On Fri, Oct 5, 2018 at 12:15 Marcus Ihlar <[email-masked]> wrote:

> Hi,
> I like this approach since it allows us to disregard non-spinning flows by
> simply observing the initial handshake. This saves us quite some work.
> On the heuristics part, we cannot get away from it, even if we see the
> spin version in the initial handshake as there´s no guarantee that an
> endpoint will set the bit properly anyway.
> However, for the version where the intent to use spinbit is explicit it
> actually makes sense for an endpoint to treat non-participating behavior as
> a protocol error.
>
> Marcus
>
> So long as this is the only thing being negotiated, it's not too bad.  If
> we want to negotiate other things, it quickly gets out of hand.  One could
> plausibly argue that the spin bit is special in that we actually want
> parties observing the transaction to understand which option has been
> selected.
>
> It's important to remember, though, that this is information you get only
> at the beginning of a new connection.  If there's migration or NAT
> rebinding, it's not guaranteed (and in fact, not intended) than you can
> correlate it back to a particular previous handshake.  You'll still either
> need heuristics to identify whether the spin bit is actually in use, or
> you'll need to ignore any traffic for which you didn't see the handshake.
>
> Thanks Kazuho. Using version numbers directly was an obvious choice, but
> so far I was discouraged by strong language in the spec, saying 0x00000001
> was the unavoidable point of convergence after all 0xFF0000XX experiments.
> So in essence you're proposing to weaken that a bit, right ?
>
> Also, are we sure that this explosion won't weigh on the new VN scheme,
> inducing slower convergence due to longer lists on either side ?
>
> >> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi
> que les pieces jointes. Les messages electroniques etant susceptibles
> d'alteration, Orange decline toute responsabilite si ce message a ete
> altere, deforme ou falsifie. Merci.
> >>
> >> privileged information that may be protected by law; they should not be
> distributed, used or copied without authorisation.

---

From: <[email-masked]>
Date: Fri, 5 Oct 2018 12:52:30 +0200

As Kazuho reminded, the aim is to fight ossification by having a wealth of 
different versions around from the beginning. Typically, each major server would 
choose its QUIC implementation and supported versions. So, an active middlebox 
that would block traffic on these grounds would instantly break access to a 
major application or content provider. No sane operator would tolerate that for 
more than an hour.

On 10/05/18 12:33, Marten Seemann wrote:
> doesn’t use the spin bit. If only a few middleboxes do that, this would force 
> *all* implementations to avoid these versions.
> negotiate a version that is supposed to use the spin bit, but then grease the 
> corresponding bits in order to preserve their privacy. Of course, this would 
> only work if the peer doesn’t enforce compliance. Otherwise, we’ll end up in a 
> world where it’s basically impossible to opt-out of the spin-bit.
>
> On Fri, Oct 5, 2018 at 12:15 Marcus Ihlar <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi,
>     actually makes sense for an endpoint to treat non-participating behavior
>     as a protocol error.
>
>     Marcus
>
>     From: QUIC <[email-masked] <mailto:[email-masked]>> On
>     Behalf Of Mike Bishop
>     To: [email-masked] <mailto:[email-masked]>;
>     Kazuho Oku <[email-masked] <mailto:[email-masked]>>
>     Cc: Brian Trammell <[email-masked] <mailto:[email-masked]>>; IETF QUIC
>     WG <[email-masked] <mailto:[email-masked]>>; Christian Huitema
>     <[email-masked] <mailto:[email-masked]>>
>     From: QUIC <[email-masked] <mailto:[email-masked]>> On
>     Behalf Of [email-masked] <mailto:[email-masked]>
>     To: Kazuho Oku <[email-masked] <mailto:[email-masked]>>
>     Cc: Brian Trammell <[email-masked] <mailto:[email-masked]>>; IETF QUIC
>     WG <[email-masked] <mailto:[email-masked]>>; Christian Huitema
>     <[email-masked] <mailto:[email-masked]>>
>     > 2018年10月3日(水) 19:06 <[email-masked]
>     <mailto:[email-masked]>>:
>     >>
>     pieces jointes. Les messages electroniques etant susceptibles

---

From: Marcus Ihlar <[email-masked]>
Date: Fri, 5 Oct 2018 10:56:05 +0000

I see your point, but I don´t believe it is a particularly realistic scenario.

It is in the interest of the network operators to ensure that their subscribers has as good and consistent experience.

Blackholing versions of QUIC has the potential of completely disrupting services and introduce inconsistent behavior.

The spinbit is intended as a tool to assist troubleshooting and quality monitoring. Explicitly introducing potentially poor network experience with the hopes that your troubleshooting / monitoring capabilities improves is a really bad idea, even for the most cynical network operator.

From: Marten Seemann <[email-masked]>
Sent: den 5 oktober 2018 12:33
To: Marcus Ihlar <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: Spin bit as a negotiated option

The problem with using a specific version for implementations that use and don’t use the spin bit is that middleboxes can effectively enforce usage of the spin bit version by blackholing every packet advertising a version that doesn’t use the spin bit. If only a few middleboxes do that, this would force *all* implementations to avoid these versions.

Implementations that don’t want to expose their RTT would then have to negotiate a version that is supposed to use the spin bit, but then grease the corresponding bits in order to preserve their privacy. Of course, this would only work if the peer doesn’t enforce compliance. Otherwise, we’ll end up in a world where it’s basically impossible to opt-out of the spin-bit.

On Fri, Oct 5, 2018 at 12:15 Marcus Ihlar <[email-masked]<mailto:[email-masked]>> wrote:

   Hi,

   Marcus

   -----Original Message-----
   From: QUIC <[email-masked]<mailto:[email-masked]>> On Behalf Of [email-masked]<mailto:[email-masked]>
   Sent: Thursday, October 4, 2018 3:47 AM
   To: Kazuho Oku <[email-masked]<mailto:[email-masked]>>
   Cc: Brian Trammell <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>; Christian Huitema <[email-masked]<mailto:[email-masked]>>
   Subject: Re: Spin bit as a negotiated option

     - direct enumeration:

           0x00000001 = v1dry

   On 10/04/18 11:50, Kazuho Oku wrote:
   > 2018年10月3日(水) 19:06 <[email-masked]<mailto:[email-masked]>>:
   >>

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Fri, 5 Oct 2018 14:22:45 +0200

Hi Kazuho, Marcus, all,

I am of two minds about this proposal.

I like it, because:

- it exercises the version negotiation machinery in a way beyond greasing, i.e. such that the negotiated information actually has an impact on the operation of the protocol. That is worth it for itself, spin bit or no.

-  it makes deploying and debugging the spin bit easier -- when negotiating the spin bit then not using it is a protocol error, then subtle errors in spin bit implementation don't require fully deployed heuristic measurement devices to detect.

- it is an explicit signal (both to the path and the other endpoint) about intent with respect to measurability. As Marcus notes, this makes it easier to reject flow state for non-spinning flows early, which eases measurement in the (likely) case that some smallish (1-10%) of flows in the Internet actually spin.

I'm less happy about the complexity implied:

- in any (non-VEC) implementation of a negotiated spin bit, the negotiation code will dwarf the code for the signal it's negotiating, which seems wrong.

- if we decide to support other client behaviors for other measurement goals in the future, it pushes the complexity of deciding what to measure to the endpoints, as opposed to placing that complexity at the midpoint.

I get that this seems like a good compromise for those who are queasy about the spin bit for whatever reason. But personally, I see this more as a good way to meaningfully exercise version control (IMO a hard requirement for V1) than as a good way to control the use of the spin bit.

Cheers,

Brian

> On 5 Oct 2018, at 12:15, Marcus Ihlar <[email-masked]> wrote:
> 
> Hi,

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Fri, 5 Oct 2018 14:27:12 +0200

> On 5 Oct 2018, at 12:56, Marcus Ihlar <[email-masked]> wrote:
> 

+ a whole lot to this. Treating middleboxes as a byzantine adversary is a useful thought experiment to determine the corner cases in a protocol, but I do not think we are well served by treating all arbitrary aims and capabilities on path as equally likely.

At the end of the day, every one of the middleboxes that we curse for their ossification was bought and sold because the operator deploying it believed it solved a problem they had, so analyzing the space of likely *intentional* behaviors reduces to an analysis of the business case for each of those behaviors.

I frankly don't see the business case for a spin-bit-blocking middlebox that wouldn't just block QUIC wholesale. But I'm not a middlebox vendor, so maybe I'm just missing something.

Now, there may be some *unintentional* breakage because some middlebox vendor only happens to analyze traffic that spins when trying to reverse-engineer QUIC, and decides that the non-spinning version is just grease and blocks it. But that seems to me to be more an argument to robustly exercise version negotiation from day one than anything else.

Cheers,

Brian

---

From: <[email-masked]>
Date: Fri, 5 Oct 2018 14:32:15 +0200

On 10/05/18 14:22, Brian Trammell (IETF) wrote:
> - in any (non-VEC) implementation of a negotiated spin bit, the negotiation
> code will dwarf the code for the signal it's negotiating, which seems wrong.

The version negotiation code is a one-shot investment for all extensions. Not a 
single extra line when the set moves from {v1,v1s} to {v1,v1s,v1svv,v1sqr?...}.

> - if we decide to support other client behaviors for other measurement goals
> in the future, it pushes the complexity of deciding what to measure to the
> endpoints, as opposed to placing that complexity at the midpoint.
The one thing that pushes the burden onto the endpoints is not negotiation but 
enforcement. I don't think enforcement is a good idea, for this very reason. 
Since these extensions are only for observers and may easily be opted out by 
either endpoint, it makes sense to honor them in a best-effort way, no more.

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 5 Oct 2018 22:43:19 +0900

+1.

I would also argue that having explicit negotiation using the version
number field helps both the endpoints and the middleboxes, because it
ensures us of having the freedom to evolve.

Having one bit (or several bits) tied forever to the behavior we
define in v1 would be a loss for everybody.

> either endpoint, it makes sense to honor them in a best-effort way, no more.

I would also argue that the endpoints must be given the freedom to
choose the signals they would expose, because there could be privacy
implications based on how the endpoints are deployed.

FWIW, I am not opposed to the idea of middlebox suggesting a signal it
wants to see, possibly by injecting a long header packet into the
flow.

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 5 Oct 2018 23:00:36 +0900

2018年10月5日(金) 21:27 Brian Trammell (IETF) <[email-masked]>:
> I frankly don't see the business case for a spin-bit-blocking middlebox that wouldn't just block QUIC wholesale.

In addition to what Alexandre and Marcus have pointed out, I would
like to second what Brian has stated here.

There are business-related reasons to prefer TCP over QUIC. There are
middlebox vendors selling TCP traffic-shaping middleboxes that claim
to reduce the consumed bandwidth (by reducing the number of
retransmits, utilizing the information about the condition of the
mobile network). There are (still!) networks that give preferential
treatment to non-TLS HTTP because they can deploy transparent,
compressing proxies.

Compared to those cases, spin bit has very weak reason for
preferential treatment (i.e. troubleshooting and quality monitoring).

Besides, the endpoints would also have the choice to expose fake
signal once we start seeing networks blocking QUIC without the spin
bit. Actually, that is the status-quo, in which we do not have any
negotiation. To put it the other way, the endpoints lose nothing by
using the version number field for exposing the availability of the
signal.

> But I'm not a middlebox vendor, so maybe I'm just missing something.
>

-- 
Kazuho Oku

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Fri, 5 Oct 2018 16:00:55 +0200

> On 5 Oct 2018, at 15:43, Kazuho Oku <[email-masked]> wrote:
> 

Okay, I buy that argument. That probably tips me into the "let's do this with VN" camp.

Cheers,

Brian

---

From: Marcus Ihlar <[email-masked]>
Date: Fri, 5 Oct 2018 15:12:19 +0000

I'm happy with endpoints having the possibility to grease the bit even for the negotiated version.

The fewer assumptions middleboxes make about expected behaviour the better.

For a spinbox there's basically only three assumptions that can be made then: non-spin versions are uninteresting, spin-version(s) might be interesting, flows with unknown version might be interesting.

________________________________
Från: QUIC <[email-masked]> för Kazuho Oku <[email-masked]>
Skickat: den 5 oktober 2018 16:00
Till: Brian Trammell
Kopia: IETF QUIC WG; Lili Peaudchien; Mike Bishop; Marten Seemann; Christian Huitema; Marcus Ihlar
Ämne: Re: Spin bit as a negotiated option

> > Thanks Kazuho. Using version numbers directly was an obvious choice, but so far I was discouraged by strong language in the spec, saying 0x00000001 was the unavoidable point of convergence after all 0xFF0000XX experiments.. So in essence you're proposing to weaken that a bit, right ?
> >
> > Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes.. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
> >

--
Kazuho Oku

---

From: <[email-masked]>
Date: Sun, 14 Oct 2018 19:05:13 +0200

Hello,

FWIW, PicoQuic now implements several "spin bit" variants, to be selected by 
version number as discussed here. Endpoints can thus decide to allow for on-path 
RTT measurement, loss location, both, or none.

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 17 Oct 2018 16:59:11 +0900

> RTT measurement, loss location, both, or none.

Fantastic. It's reassuring to know that it works, and that you are
running multiple variants.

-- 
Kazuho Oku