Subject: Is "Version Greasing" a new benfit or a new obstacle?

From: G Fairhurst <[email-masked]>
Date: Wed, 10 Apr 2019 09:48:29 +0100

Obscuring the version of a protocol seems like a major design design 
decision for wider use cases. So, I'm trying to understand the 
motivation for version greasing.

(1) I know there were instances where some early versions of QUIC were 
blocked due to an uninitentional matching of the header. Is there 
evidence of intentional attempts to block updates to protocols?

(2) Thinking about operating a network that cares about user support and 
protection from unwanted traffic, I would expect that there would be 
cases where traffic pattern anomolies are found and the appropriate 
thing would be to try to determine if a new protocol had been deployed 
and monitor it, if not, then the next most obvious thing could be to 
block all unexpected traffic, that seems like a decision to hide the 
version could increase ossification for new versions in these cases.

(3) Similarly, if a threat is known to impact only a specific (older) 
version, it would seem to motivate a drop of that traffic that seeks to 
use that version, while still permitting other traffic. Forcing version 
detection to use pattern matching/ML will lead to less predictable 
outcomes, or blocking based on address, etc.

(4) This obfusticates the most basic piece of reporting information used 
for support. It hides the extent of deployment of the current protocol 
version and prevlance of old implementations.

(5) On the support, if a problem only emerges when a particular version 
is used with a particular address, then this helps pinpoint the issues. 
Matching client versions to servers is much more of an issue if the user 
community uses a wide range of servers (less so, I expect for major 
providers: google, facebook, etc, etc), but significant when there is a 
use of a diverse set of external sites and sites with their own load 
balancers, etc and a need to manage interactions with L2 services.

I am intersted in knowing if this is likely to benefit or be a new obstacle?

Gorry

---

From: Roberto Peon <[email-masked]>
Date: Wed, 10 Apr 2019 16:51:39 +0000

You're kinda between a rock-and-a-hard-place either way:

  - We've seen how much fun ossification is in TCP and HTTP. If the thing is observable, it will be ossified seems to be the lesson. A lot of the reason why QUIC was started in the first place was because of the inability to improve TCP due to this ossification.
  - OTOH, there is the fear of unknown/unobservable which might cause operators to block things, whether predictably or not.

My opinion is that it is better to start with preventing ossification, and then if that results in too large a percentage of operators blocking things, to re-evaluate. 

My guesses:
IP+port tuples and traffic patterns are still observable (for better and worse), which implies operators will still have significant tools for managing traffic. I believe that these are acted on/matched (ML or not) regardless of any other data presented. In other words, I have a doubt that stating the version in an observable way will prevent the use of such tools.

Most problems I've seen associated with implementations rather than protocol versions (though when the latter happens it is pretty severe). If you believe this assertion, then acting on protocol version is less interesting than attempting to act based on implementation fingerprints.
-=R

On 4/10/19, 1:48 AM, "QUIC on behalf of G Fairhurst" <[email-masked] on behalf of [email-masked]> wrote:

    

    

    Gorry

---

From: "Border, John" <[email-masked]>
Date: Wed, 10 Apr 2019 16:58:49 +0000

I understand why people want to come down on the side of preventing ossification.  But, things have changed.  There are a lot of more negative consequences now when people unnecessarily block things.  I think operators would put a lot of pressure on vendors to not do it now and to fix it if they did "by accident".  Of course, I am only one operator.  It would be nice to hear from others...

John

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Roberto Peon
Sent: Wednesday, April 10, 2019 12:52 PM
To: G Fairhurst <[email-masked]>; [email-masked]
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

WARNING: The sender of this email could not be validated and may not match the person in the "From" field.

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

My guesses:

    Gorry

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 10 Apr 2019 10:25:23 -0700

China blocks as a rule.
Russia is running an experiment to block the rest of the internet.
USA blocks net neutrality.
EU blocks cookies.
GB blocks itself.

So blocking is not limited to what an operator considers best for business.

On 10 April 2019 at 18.59.15, Border, John ([email-masked]) wrote:

I understand why people want to come down on the side of preventing
ossification. But, things have changed. There are a lot of more negative
consequences now when people unnecessarily block things. I think operators
would put a lot of pressure on vendors to not do it now and to fix it if
they did "by accident". Of course, I am only one operator. It would be nice
to hear from others...

John

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Roberto Peon
Sent: Wednesday, April 10, 2019 12:52 PM
To: G Fairhurst <[email-masked]>; [email-masked]
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

WARNING: The sender of this email could not be validated and may not match
the person in the "From" field.

CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you recognize the sender and know
the content is safe.

- We've seen how much fun ossification is in TCP and HTTP. If the thing is
observable, it will be ossified seems to be the lesson. A lot of the reason
why QUIC was started in the first place was because of the inability to
improve TCP due to this ossification.
- OTOH, there is the fear of unknown/unobservable which might cause
operators to block things, whether predictably or not.

My opinion is that it is better to start with preventing ossification, and
then if that results in too large a percentage of operators blocking
things, to re-evaluate.

My guesses:
IP+port tuples and traffic patterns are still observable (for better and
worse), which implies operators will still have significant tools for
managing traffic. I believe that these are acted on/matched (ML or not)
regardless of any other data presented. In other words, I have a doubt that
stating the version in an observable way will prevent the use of such
tools.

Most problems I've seen associated with implementations rather than
protocol versions (though when the latter happens it is pretty severe). If
you believe this assertion, then acting on protocol version is less
interesting than attempting to act based on implementation fingerprints.
-=R

On 4/10/19, 1:48 AM, "QUIC on behalf of G Fairhurst" <[email-masked]
on behalf of [email-masked]> wrote:

I am intersted in knowing if this is likely to benefit or be a new
obstacle?

Gorry

---

From: "Border, John" <[email-masked]>
Date: Wed, 10 Apr 2019 17:32:55 +0000

That is not ossification.  But, it is a valid point.

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Sent: Wednesday, April 10, 2019 1:25 PM
To: G Fairhurst <[email-masked]>; Border, John <[email-masked]>; Roberto Peon <[email-masked]>; [email-masked]
Subject: RE: Is "Version Greasing" a new benfit or a new obstacle?

China blocks as a rule.

On 10 April 2019 at 18.59.15, Border, John ([email-masked]<mailto:[email-masked]>) wrote:

John

-----Original Message-----
From: QUIC <[email-masked]<mailto:[email-masked]>> On Behalf Of Roberto Peon
Sent: Wednesday, April 10, 2019 12:52 PM
To: G Fairhurst <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

My guesses:

On 4/10/19, 1:48 AM, "QUIC on behalf of G Fairhurst" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

Gorry

---

From: "Gorry (erg)" <[email-masked]>
Date: Wed, 10 Apr 2019 19:13:52 +0100

Thanks Mikkel, I do understand that various actors intentionally drop - but are you saying these actors would specifically choose to block a new version of QUIC ...  I do not understand that assertion.

Gorry

> On 10 Apr 2019, at 18:25, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> 
> China blocks as a rule.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 10 Apr 2019 12:56:50 -0700

Let’s say  China makes a deal with Google to allow their search engine (a
not entirely unreasonably proposition).
The Great Firewall of China (GFC) is reconfigured to accept traffic from a
specific IP range for QUIC protocol v1.

Google later upgrades to v2. GFC is updated. Later, out of ignorance, some
other department within said company decides that its browsers should all
use forced version aliases as a rule. GFC breaks. (Assuming Googles servers
are not located within the demilitarised zone and users use Chrome).

Alternative 1: Google up front decides to deploy forced version aliasing.
China rejects the deal because they do not want to support random traffic
through GFC.

Alternative 2: Google up front decides to deploy forced version aliasing,
and China does not reject the deal, out of ignorance. Later some department
in China GFC oversight realises the deal and forces Google to remove forced
version aliasing, or shut down its service.

Alternative 3: Google up front decides to deploy forced version aliasing
and China says, of course, we trust everything from Googles IP range.

Alternative 4: Google realises that forced version aliasing does not work
in the general case, and makes a special case for users within a certain
geographical area.

Disclaimer: I’m not affiliated with Google and this might not at all be how
things would work, but it does highlight some of the challenges.

Of course, to be fair, China is not the only government entity that might
have a vested interest. Let's take a fictional example of US a real estate
investor with heavy ties to the eastern european block who somehow finds
himself elected as president needing to find leverage on trade negotiations
without effectively hurting personal finances. It turns out that blocking
certain digital services from said eastern european block is the perfect
tool in the trade negotiations. Advisers point out that those services run
heavily encrypted on the dark web with perpetual circulation of IP ranges
so, ignoring any legal concerns, it would not be practically possible.
Eventually someone figures out that these services all usually are at the
forefront of technology and currently use QUIC v3.2 and no-one else has
deployed that version yet, so it would suffice to ask NSA to tap into the
backbone and disrupt specific packets. Of course, this ends up taking down
the Bavarian local government election process in Germany where they are
the first to use a new digital election system. Not that the is an issue,
since trade negotiations are also running hot in that area, so that is just
an accidental bonus.

Or an endless number of other developments over the next few decades if the
past few years is anything to go by.

On 10 April 2019 at 20.13.59, Gorry (erg) ([email-masked]) wrote:

Thanks Mikkel, I do understand that various actors intentionally drop - but
are you saying these actors would specifically choose to block a new
version of QUIC ...  I do not understand that assertion.

Gorry

On 10 Apr 2019, at 18:25, Mikkel Fahnøe Jørgensen <[email-masked]>
wrote:

China blocks as a rule.

John

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Roberto Peon
Sent: Wednesday, April 10, 2019 12:52 PM
To: G Fairhurst <[email-masked]>; [email-masked]
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

My guesses:
stating the version in an observable way will prevent the use of such tools.

Gorry

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 10 Apr 2019 21:45:13 -0400

It's always possible to come up with a scenario where the network helps here, but the cost of help is the risk of harm.

I would cast this question differently: who has the right(?) to decide when a new version of the protocol is permitted?  The motivation behind this is that it is considerably more likely if only endpoints are involved in that process.

The architectural consequences of protecting a "clean" version negotiation process are what makes me positive about this change.  It is always possible to invent a new version negotiation process that looks very much like QUIC v1, but enables the use of v2.  That is what we ended up doing for the TLS 1.2 to 1.3 transition.  For instance, you could learn that you can negotiate QUICv2 by changing the case of certain characters in the server_name extension.  Or you could use the ordering of transport parameters to signal the support for an alternative version.  If you are willing to tolerate a risk of downgrade, then you can even negotiate different versions in a way that can't be detected without access to session keys.

I don't want to have to repeat the ugly hacks we had to add to TLS.  If we can deploy aliasing, I have a little hope that we can use the versionin gmechanism we have designed.

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 11 Apr 2019 08:03:12 +0000

Hi Mikkel,

I guess it is much more likely that any of those actors would just block QUIC all together (as there is actually a fallback to TCP).

Mirja

On 10.04.19, 21:57, "QUIC on behalf of Mikkel Fahnøe Jørgensen" <[email-masked] on behalf of [email-masked]> wrote:

    Let’s say  China makes a deal with Google to allow their search engine (a not entirely unreasonably proposition).
    The Great Firewall of China (GFC) is reconfigured to accept traffic from a specific IP range for QUIC protocol v1.

    Google later upgrades to v2. GFC is updated. Later, out of ignorance, some other department within said company decides that its browsers should all use forced version aliases as a rule. GFC breaks. (Assuming Googles servers are not located within the demilitarised
     zone and users use Chrome).

    Alternative 1: Google up front decides to deploy forced version aliasing. China rejects the deal because they do not want to support random traffic through GFC.

    Alternative 2: Google up front decides to deploy forced version aliasing, and China does not reject the deal, out of ignorance. Later some department in China GFC oversight realises the deal and forces Google to remove forced version aliasing, or shut down
     its service.

    Alternative 3: Google up front decides to deploy forced version aliasing and China says, of course, we trust everything from Googles IP range.

    Alternative 4: Google realises that forced version aliasing does not work in the general case, and makes a special case for users within a certain geographical area.

    Disclaimer: I’m not affiliated with Google and this might not at all be how things would work, but it does highlight some of the challenges.

    Of course, to be fair, China is not the only government entity that might have a vested interest. Let's take a fictional example of US a real estate investor with heavy ties to the eastern european block who somehow finds himself elected as president needing
     to find leverage on trade negotiations without effectively hurting personal finances. It turns out that blocking certain digital services from said eastern european block is the perfect tool in the trade negotiations. Advisers point out that those services
     run heavily encrypted on the dark web with perpetual circulation of IP ranges so, ignoring any legal concerns, it would not be practically possible. Eventually someone figures out that these services all usually are at the forefront of technology and currently
     use QUIC v3.2 and no-one else has deployed that version yet, so it would suffice to ask NSA to tap into the backbone and disrupt specific packets. Of course, this ends up taking down the Bavarian local government election process in Germany where they are
     the first to use a new digital election system. Not that the is an issue, since trade negotiations are also running hot in that area, so that is just an accidental bonus.

    Or an endless number of other developments over the next few decades if the past few years is anything to go by.

    

    Gorry

    
    China blocks as a rule.

    

    I understand why people want to come down on the side of preventing ossification. But, things have changed. There are a lot of more negative consequences now when people unnecessarily block things. I think operators would put a lot of pressure on vendors to
     not do it now and to fix it if they did "by accident". Of course, I am only one operator. It would be nice to hear from others...

    John

    -----Original Message-----
    From: QUIC <[email-masked]> On Behalf Of Roberto Peon
    Sent: Wednesday, April 10, 2019 12:52 PM
    To: G Fairhurst <[email-masked]>;
    [email-masked]
    Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

    

    My guesses:
    IP+port tuples and traffic patterns are still observable (for better and worse), which implies operators will still have significant tools for managing traffic. I believe that these are acted on/matched (ML or not) regardless of any other data presented. In
     other words, I have a doubt that stating the version in an observable way will prevent the use of such tools.
    
    Most problems I've seen associated with implementations rather than protocol versions (though when the latter happens it is pretty severe). If you believe this assertion, then acting on protocol version is less interesting than attempting to act based on implementation
     fingerprints.
    -=R

    On 4/10/19, 1:48 AM, "QUIC on behalf of G Fairhurst" <[email-masked] on behalf of
    [email-masked]> wrote:

    

    

    Gorry

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 11 Apr 2019 08:11:12 +0000

Hi Martin,

I'm still very undecided what is the better choice now: paying the additional complexity cost straight away and for everybody (basically out of fear based on past experience but without taking into account that the circumstances may have changed as Gorry mentioned) or having to add one of the those, from a design point of view, ugly hacks potentially later (which still may be less complex). It might be silly to "not" learn from the past but it might also be a chance. Maybe I'm a bit naïve here but I don't want to give up yet. But I also uncertain because we all can predict the future. However, I think the important plan is to have a usable path for version updates (even thought it might be a hack).

Mirja

On 11.04.19, 03:45, "QUIC on behalf of Martin Thomson" <[email-masked] on behalf of [email-masked]> wrote:

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 11 Apr 2019 01:20:29 -0700

In a not so distant future, a fallback to TCP would not be a viable
alternative for a number of services. Think WebRTC.
If QUIC and other encrypted protocols become dominant, just blocking
everything would literally block everything.
Of course, http(s) as we know it, it will continue to live on, but many
mobile apps would not.

On 11 April 2019 at 10.03.14, Mirja Kuehlewind (
[email-masked]) wrote:

Hi Mikkel,

I guess it is much more likely that any of those actors would just block
QUIC all together (as there is actually a fallback to TCP).

Mirja

On 10.04.19, 21:57, "QUIC on behalf of Mikkel Fahnøe Jørgensen" <
[email-masked] on behalf of [email-masked]> wrote:

are not located within the demilitarised

Alternative 2: Google up front decides to deploy forced version aliasing,
and China does not reject the deal, out of ignorance. Later some department
in China GFC oversight realises the deal and forces Google to remove forced
version aliasing, or shut down
its service.

Of course, to be fair, China is not the only government entity that might
have a vested interest. Let's take a fictional example of US a real estate
investor with heavy ties to the eastern european block who somehow finds
himself elected as president needing
to find leverage on trade negotiations without effectively hurting personal
finances. It turns out that blocking certain digital services from said
eastern european block is the perfect tool in the trade negotiations.
Advisers point out that those services
run heavily encrypted on the dark web with perpetual circulation of IP
ranges so, ignoring any legal concerns, it would not be practically
possible. Eventually someone figures out that these services all usually
are at the forefront of technology and currently
use QUIC v3.2 and no-one else has deployed that version yet, so it would
suffice to ask NSA to tap into the backbone and disrupt specific packets.
Of course, this ends up taking down the Bavarian local government election
process in Germany where they are
the first to use a new digital election system. Not that the is an issue,
since trade negotiations are also running hot in that area, so that is just
an accidental bonus.

Gorry

China blocks as a rule.

I understand why people want to come down on the side of preventing
ossification. But, things have changed. There are a lot of more negative
consequences now when people unnecessarily block things. I think operators
would put a lot of pressure on vendors to
not do it now and to fix it if they did "by accident". Of course, I am only
one operator. It would be nice to hear from others...

John

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Roberto Peon
Sent: Wednesday, April 10, 2019 12:52 PM
To: G Fairhurst <[email-masked]>;
[email-masked]

My guesses:
IP+port tuples and traffic patterns are still observable (for better and
worse), which implies operators will still have significant tools for
managing traffic. I believe that these are acted on/matched (ML or not)
regardless of any other data presented. In
other words, I have a doubt that stating the version in an observable way
will prevent the use of such tools.

Most problems I've seen associated with implementations rather than
protocol versions (though when the latter happens it is pretty severe). If
you believe this assertion, then acting on protocol version is less
interesting than attempting to act based on implementation
fingerprints.
-=R

on behalf of

Gorry

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 12 Apr 2019 15:53:24 +0000

Version greasing will create problems for DDoS protection. The DDoS protection middlebox will have to treat all traffic with unknown version numbers as plain old QUIC and rate limit it. Any solution that requires per connection state to track version numbers doesn’t scale. Especially in the cloud scenario customers bring their own web servers so this makes any opt-out solution not good enough. So I do think this is an obstacle for many deployments.

From: QUIC <[email-masked]> On Behalf Of Mikkel Fahnøe Jørgensen
Sent: Thursday, April 11, 2019 1:20 AM
To: Gorry (erg) <[email-masked]>; Mirja Kuehlewind <[email-masked]>
Cc: [email-masked]; Border, John <[email-masked]>; Roberto Peon <[email-masked]>
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

In a not so distant future, a fallback to TCP would not be a viable alternative for a number of services. Think WebRTC.
If QUIC and other encrypted protocols become dominant, just blocking everything would literally block everything.
Of course, http(s) as we know it, it will continue to live on, but many mobile apps would not.

On 11 April 2019 at 10.03.14, Mirja Kuehlewind ([email-masked]<mailto:[email-masked]>) wrote:
Hi Mikkel,

Mirja

On 10.04.19, 21:57, "QUIC on behalf of Mikkel Fahnøe Jørgensen" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

to find leverage on trade negotiations without effectively hurting personal finances. It turns out that blocking certain digital services from said eastern european block is the perfect tool in the trade negotiations.. Advisers point out that those services

On 10 April 2019 at 20.13.59, Gorry (erg) ([email-masked]<mailto:[email-masked]>) wrote:

Gorry

On 10 Apr 2019, at 18:25, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:

China blocks as a rule.

John

-----Original Message-----
From: QUIC <[email-masked]<mailto:[email-masked]>> On Behalf Of Roberto Peon
Sent: Wednesday, April 10, 2019 12:52 PM
To: G Fairhurst <[email-masked]<mailto:[email-masked]>>;
[email-masked]<mailto:[email-masked]>
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

My guesses:

Most problems I've seen associated with implementations rather than protocol versions (though when the latter happens it is pretty severe).. If you believe this assertion, then acting on protocol version is less interesting than attempting to act based on implementation
fingerprints.
-=R

On 4/10/19, 1:48 AM, "QUIC on behalf of G Fairhurst" <[email-masked]<mailto:[email-masked]> on behalf of
[email-masked]<mailto:[email-masked]>> wrote:

version could increase ossification for new versions in these cases..

Gorry

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 12 Apr 2019 15:55:30 +0000

Sorry for a typo. Plain old UDP not QUIC.

From: Praveen Balasubramanian
Sent: Friday, April 12, 2019 8:53 AM
To: Mikkel Fahnøe Jørgensen <[email-masked]>; Gorry (erg) <[email-masked]>; Mirja Kuehlewind <[email-masked]>
Cc: [email-masked]; Border, John <[email-masked]>; Roberto Peon <[email-masked]>
Subject: RE: Is "Version Greasing" a new benfit or a new obstacle?

From: QUIC <[email-masked]<mailto:[email-masked]>> On Behalf Of Mikkel Fahnøe Jørgensen
Sent: Thursday, April 11, 2019 1:20 AM
To: Gorry (erg) <[email-masked]<mailto:[email-masked]>>; Mirja Kuehlewind <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; Border, John <[email-masked]<mailto:[email-masked]>>; Roberto Peon <[email-masked]<mailto:[email-masked]>>
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

Mirja

Gorry

China blocks as a rule.

John

-----Original Message-----
From: QUIC <[email-masked]<mailto:[email-masked]>> On Behalf Of Roberto Peon
Sent: Wednesday, April 10, 2019 12:52 PM
To: G Fairhurst <[email-masked]<mailto:[email-masked]>>;

My guesses:

Gorry

---

From: Ian Swett <[email-masked]>
Date: Fri, 12 Apr 2019 11:57:23 -0400

Possibly I'm missing something, but wouldn't version greasing be an
optional extension?  So if you wanted to do so you could, but those with
load balancing that couldn't handle it would not?

On Fri, Apr 12, 2019 at 11:55 AM Praveen Balasubramanian <pravb=
[email-masked]> wrote:

> *From:* Praveen Balasubramanian
> *Sent:* Friday, April 12, 2019 8:53 AM
> *To:* Mikkel Fahnøe Jørgensen <[email-masked]>; Gorry (erg) <
> [email-masked]>; Mirja Kuehlewind <[email-masked]>
> *Cc:* [email-masked]; Border, John <[email-masked]>; Roberto Peon <
> [email-masked]>
> *Subject:* RE: Is "Version Greasing" a new benfit or a new obstacle?
>
>
>
> Version greasing will create problems for DDoS protection. The DDoS
> protection middlebox will have to treat all traffic with unknown version
> numbers as plain old QUIC and rate limit it. Any solution that requires per
> connection state to track version numbers doesn’t scale. Especially in the
> cloud scenario customers bring their own web servers so this makes any
> opt-out solution not good enough. So I do think this is an obstacle for
> many deployments.
>
>
>
> *From:* QUIC <[email-masked]> *On Behalf Of *Mikkel Fahnøe
> Jørgensen
> *Sent:* Thursday, April 11, 2019 1:20 AM
> *To:* Gorry (erg) <[email-masked]>; Mirja Kuehlewind <
> [email-masked]>
> *Cc:* [email-masked]; Border, John <[email-masked]>; Roberto Peon <
> [email-masked]>
> *Subject:* Re: Is "Version Greasing" a new benfit or a new obstacle?
>
>
>
> Disclaimer: I’m not affiliated with Google and this might not at all be
> how things would work, but it does highlight some of the challenges.
>
>
> to find leverage on trade negotiations without effectively hurting
> personal finances. It turns out that blocking certain digital services from
> said eastern european block is the perfect tool in the trade negotiations..
> Thanks Mikkel, I do understand that various actors intentionally drop -
> but are you saying these actors would specifically choose to block a new
> not do it now and to fix it if they did "by accident". Of course, I am
> only one operator. It would be nice to hear from others...
>
>
>
> John
>
>
> protocol versions (though when the latter happens it is pretty severe).. If

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 12 Apr 2019 16:08:07 +0000

It being optional helps in cases where the service being deployed and the DoS protection service are under same administrative domain. So depending on how DoS protection is done the extension could be turned on or off. The problem is with the IaaS scenario if the extension is turned on by default in some major web server. Now DDoS protection cannot be offered as a cloud service without harming said traffic because it will be rate limited. And rate limiting is strictly worse than blocking QUIC because performance will suffer.

We will likely need some text in manageability draft which says that turning on this extension may break network functions and cause performance issues so services / implementations should only turn this on after these considerations.

From: QUIC <[email-masked]> On Behalf Of Ian Swett
Sent: Friday, April 12, 2019 8:57 AM
To: Praveen Balasubramanian <pravb=[email-masked]>
Cc: Gorry (erg) <[email-masked]>; Roberto Peon <[email-masked]>; Mirja Kuehlewind <[email-masked]>; Mikkel Fahnøe Jørgensen <[email-masked]>; [email-masked]; Border, John <[email-masked]>
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

Possibly I'm missing something, but wouldn't version greasing be an optional extension?  So if you wanted to do so you could, but those with load balancing that couldn't handle it would not?

On Fri, Apr 12, 2019 at 11:55 AM Praveen Balasubramanian <pravb=[email-masked]<mailto:[email-masked]>> wrote:
Sorry for a typo. Plain old UDP not QUIC.

To: Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>>; Gorry (erg) <[email-masked]<mailto:[email-masked]>>; Mirja Kuehlewind <[email-masked]<mailto:[email-masked]>>

Mirja

On 10 April 2019 at 20.13.59, Gorry (erg) ([email-masked]<mailto:[email-masked]..ac.uk>) wrote:

Gorry

China blocks as a rule.

John

-----Original Message-----
From: QUIC <[email-masked]<mailto:[email-masked]>> On Behalf Of Roberto Peon
Sent: Wednesday, April 10, 2019 12:52 PM
To: G Fairhurst <[email-masked]<mailto:[email-masked]>>;

My guesses:

On 4/10/19, 1:48 AM, "QUIC on behalf of G Fairhurst" <[email-masked]<mailto:[email-masked]> on behalf of
[email-masked]..uk<mailto:[email-masked]>> wrote:

Gorry

---

From: Christian Huitema <[email-masked]>
Date: Fri, 12 Apr 2019 09:13:31 -0700

On 4/12/2019 8:57 AM, Ian Swett wrote:
> *...*
> protection middlebox will have to treat all traffic with unknown
> version numbers as plain old QUIC and rate limit it. Any solution that
> requires per connection state to track version numbers doesn’t scale.
> Especially in the cloud scenario customers bring their own web servers
> so this makes any opt-out solution not good enough. So I do think this
> is an obstacle for many deployments.
>
Don't we have an example of middle-box ossification just there?

Praveen's example points to the perils of the "shortest path" approach
to middle-box design. Taking the shortest path means finding something
that works now, and once you have found it deploy it without worry about
consequences -- which very often include ossification. If the DDOS
protection box looks for "QUIC versions that it knows" and just rate
limits the other types of packets, then the box will by design prevent
deployment of versions that it does not know.

I assume that big organizations like Azure will update their protections
once a new version is out, but the experience shows that many other
organizations don't. And even when they do, the focus on published
versions will block experimenting with new versions, not to mention
interfering with other features like connection migration. I take
Praveen's example as an argument why we should in fact standardize
version greasing, precisely to avoid this "shortest path middle-box
design".

In the case of DDOS protection, I will point out that DDOS protection
also needs to work with short header packets, which do not carry any
version number. In order to avoid DDOS by short header packets, the
boxes have to be able to distinguish between packets that belong to
authorized connections and packets that don't. And if you do that, then
you don't really need to look at the version numbers, you can simply
rely on the invariants.

-- Christian Huitema

---

From: Roberto Peon <[email-masked]>
Date: Fri, 12 Apr 2019 16:15:51 +0000

For any large deployment, I suspect they’ll need LB.

For most kinds of LB-ing, I’d expect the LB system to either directly or indirectly be involved in choosing/vending connection IDs.

If it is involved in vending/choosing connection IDs, it can encode version information into the CID.

-=R

From: Christian Huitema <[email-masked]>
Date: Friday, April 12, 2019 at 9:13 AM
To: Ian Swett <ianswett=[email-masked]>, Praveen Balasubramanian <pravb=[email-masked]>
Cc: "Gorry (erg)" <[email-masked]>, Roberto Peon <[email-masked]>, Mirja Kuehlewind <[email-masked]>, Mikkel Fahnøe Jørgensen <[email-masked]>, "[email-masked]" <[email-masked]>, "Border, John" <[email-masked]>
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

...

Praveen's example points to the perils of the "shortest path" approach to middle-box design. Taking the shortest path means finding something that works now, and once you have found it deploy it without worry about consequences -- which very often include ossification. If the DDOS protection box looks for "QUIC versions that it knows" and just rate limits the other types of packets, then the box will by design prevent deployment of versions that it does not know.

I assume that big organizations like Azure will update their protections once a new version is out, but the experience shows that many other organizations don't. And even when they do, the focus on published versions will block experimenting with new versions, not to mention interfering with other features like connection migration. I take Praveen's example as an argument why we should in fact standardize version greasing, precisely to avoid this "shortest path middle-box design".

In the case of DDOS protection, I will point out that DDOS protection also needs to work with short header packets, which do not carry any version number. In order to avoid DDOS by short header packets, the boxes have to be able to distinguish between packets that belong to authorized connections and packets that don't. And if you do that, then you don't really need to look at the version numbers, you can simply rely on the invariants.

-- Christian Huitema

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 12 Apr 2019 16:28:15 +0000

Re. Encoding in CID. Then it needs to be standardized and if it is standardized what’s the point in greasing? Let’s not ignore the smaller services who use the cloud i.e. people deploying say NGINX web servers in AWS, Azure or GCP. Not every QUIC deployment is a large first party service which controls all aspects how LB and DDoS are done. In public clouds LB and DDoS protection are network functions (services) and are middleboxes under a completely different administrative domain.

Christian, this is about source address validation by the middlebox. This is how it works for TCP (think SYN cookies) and we need to have parity for QUIC. When under attack, short header packets will be dropped for addresses that have not been validated, but this is the same for non-SYN packets for TCP. I don’t understand how invariants can be used here. If we don’t have parity for DoS protection in network then QUIC traffic will be rate limited which is going to be really bad for performance. FWIW there is already DoS traffic that looks like QUIC.

From: Roberto Peon <[email-masked]>
Sent: Friday, April 12, 2019 9:16 AM
To: Christian Huitema <[email-masked]>; Ian Swett <[email-masked]>; Praveen Balasubramanian <[email-masked]>
Cc: Gorry (erg) <[email-masked]>; Mirja Kuehlewind <[email-masked]>; Mikkel Fahnøe Jørgensen <[email-masked]>; [email-masked]; Border, John <[email-masked]>
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

-=R

From: Christian Huitema <[email-masked]<mailto:[email-masked]>>
Date: Friday, April 12, 2019 at 9:13 AM
To: Ian Swett <ianswett=[email-masked]<mailto:ianswett=[email-masked]>>, Praveen Balasubramanian <pravb=[email-masked]<mailto:pravb=[email-masked]>>
Cc: "Gorry (erg)" <[email-masked]<mailto:[email-masked]>>, Roberto Peon <[email-masked]<mailto:[email-masked]>>, Mirja Kuehlewind <[email-masked]<mailto:[email-masked]>>, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "Border, John" <[email-masked]<mailto:[email-masked]>>
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 12 Apr 2019 09:34:56 -0700

You could add some Magic(™) to Kubernetes ingress controllers and
Kubernetes services of kind Load Balancer.
These already do coordination with mapping internal IP’s to external IP’s
(or ought to do that, since it doesn’t always work, but I guess things will
mature).

For example, you have the name of a Kubernetes secret that is used to
configure the load balancer.

Of course, if you have high performance dedicated hardware that runs for
many customers concurrently, it might be an issue, but surely the ordinary
load balancers that take configurations can observe and alert dedicated
hardware of suspicious 5-tuples.

Outside of Kubernetes, various cloud providers have REST API’s for setting
TLS certificates etc, so I take it, a shared secret could also slip through
here.

I get it, that infrastructure folks don’t want to rewrite their load
balancers, but it is not like it is impossible by any stretch.

Mikkel

On 12 April 2019 at 18.08.09, Praveen Balasubramanian ([email-masked])
wrote:

It being optional helps in cases where the service being deployed and the
DoS protection service are under same administrative domain. So depending
on how DoS protection is done the extension could be turned on or off. The
problem is with the IaaS scenario if the extension is turned on by default
in some major web server. Now DDoS protection cannot be offered as a cloud
service without harming said traffic because it will be rate limited. And
rate limiting is strictly worse than blocking QUIC because performance will
suffer.

We will likely need some text in manageability draft which says that
turning on this extension may break network functions and cause performance
issues so services / implementations should only turn this on after these
considerations.

*From:* QUIC <[email-masked]> *On Behalf Of * Ian Swett
*Sent:* Friday, April 12, 2019 8:57 AM
*To:* Praveen Balasubramanian <pravb=[email-masked]>
*Cc:* Gorry (erg) <[email-masked]>; Roberto Peon <[email-masked]>;
Mirja Kuehlewind <[email-masked]>; Mikkel Fahnøe Jørgensen <
[email-masked]>; [email-masked]; Border, John <[email-masked]>
*Subject:* Re: Is "Version Greasing" a new benfit or a new obstacle?

*From:* QUIC <[email-masked]> *On Behalf Of *Mikkel Fahnøe Jørgensen

Hi Mikkel,

Mirja

eastern european block is the perfect tool in the trade negotiations..

On 10 April 2019 at 20.13.59, Gorry (erg) ([email-masked]
<[email-masked]..ac.uk>) wrote:

Gorry

China blocks as a rule.

John

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Roberto Peon
Sent: Wednesday, April 10, 2019 12:52 PM
To: G Fairhurst <[email-masked]>;
[email-masked]

My guesses:

On 4/10/19, 1:48 AM, "QUIC on behalf of G Fairhurst" <[email-masked]
on behalf of
[email-masked]..uk <[email-masked]>> wrote:

Gorry

---

From: Roberto Peon <[email-masked]>
Date: Fri, 12 Apr 2019 16:41:48 +0000

I’m not sure that I agree there—there is no need to have it standardized so long as the LB is involved in picking it. Each LB system can choose whatever it wants. Thus, every LB *deployment* could use a different scheme with different grease. Every LB deployment could change the scheme it uses pseudo-randomly or according to the lunar calendar, etc.

Because there is a dynamic (and controlling) component (namely the part of the LB system which vends and then later routes CIDs), there is no need for anything else.

-=R

From: Praveen Balasubramanian <[email-masked]>
Date: Friday, April 12, 2019 at 9:28 AM
To: Roberto Peon <[email-masked]>, Christian Huitema <[email-masked]>, Ian Swett <[email-masked]>
Cc: "Gorry (erg)" <[email-masked]>, Mirja Kuehlewind <[email-masked]>, Mikkel Fahnøe Jørgensen <[email-masked]>, "[email-masked]" <[email-masked]>, "Border, John" <[email-masked]>
Subject: RE: Is "Version Greasing" a new benfit or a new obstacle?

-=R

-- Christian Huitema

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 12 Apr 2019 09:50:08 -0700

On 12 April 2019 at 18.42.45, Roberto Peon ([email-masked]) wrote:

 pseudo-randomly or according to the lunar calendar, etc.

We had that problem at Nokia a long time ago. Mobile phones had problems
adjusting the calendar in certain regions because local time was dependent
on people standing on hills looking at the moon. And sometimes it was
cloudy.

---

From: Dmitri Tikhonov <[email-masked]>
Date: Fri, 12 Apr 2019 13:58:06 -0400

> on people standing on hills looking at the moon. And sometimes it was

I'd love to hear the full story!  I googled "nokia phone adjust time
look at moon," [1]  but nothing came up.

  - Dmitri.

1. https://www.google.com/search?q=nokia+phone+adjust+time+look+at+moon

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 12 Apr 2019 11:29:33 -0700

"Each month of the Islamic calendar commences on the birth of the new lunar
cycle. Traditionally this is based on actual observation of the
crescent (*hilal
<https://en.wiktionary.org/wiki/hilal#Noun>*) marking the end of the
previous lunar cycle and hence the previous month, thereby beginning the
new month. Consequently, each month can have 29 or 30 days depending on the
visibility of the moon, astronomical positioning of the earth and weather
conditions."

https://en.wikipedia.org/wiki/Islamic_calendar

I believe some sort of standardisation took place to accommodate electronic
calendars. I’m not familiar with the details, but I did have some coworkers
working on time algorithms in the S30 series (from the 3310 team) who had
been struggling with that problem.

Mikkel

On 12 April 2019 at 19.58.13, Dmitri Tikhonov ([email-masked])
wrote:

- Dmitri.

---

From: G Fairhurst <[email-masked]>
Date: Sat, 13 Apr 2019 12:03:48 +0100

Bringing this thread back - to me it seems there has been lots of 
discussion about how "version greasing" can be added in various ways, 
and very little on what makes this necessary and worth the impact on the 
network.

I still don't do not see that the side effects are worse when the 
protocol requires the version to be unobfusticated (as in most other 
IETF transport protocols).

Gorry

On 12/04/2019, 17:41, Roberto Peon wrote:
>
> I’m not sure that I agree there—there is no need to have it 
> standardized so long as the LB is involved in picking it. Each LB 
> system can choose whatever it wants. Thus, every LB **deployment** 
> could use a different scheme with different grease. Every LB 
> deployment could change the scheme it uses pseudo-randomly or 
> according to the lunar calendar, etc.
>
> Because there is a dynamic (and controlling) component (namely the 
> part of the LB system which vends and then later routes CIDs), there 
> is no need for anything else.
>
> -=R
>
> *From: *Praveen Balasubramanian <[email-masked]>
> *Date: *Friday, April 12, 2019 at 9:28 AM
> *To: *Roberto Peon <[email-masked]>, Christian Huitema 
> <[email-masked]>, Ian Swett <[email-masked]>
> *Cc: *"Gorry (erg)" <[email-masked]>, Mirja Kuehlewind 
> <[email-masked]>, Mikkel Fahnøe Jørgensen 
> <[email-masked]>, "[email-masked]" <[email-masked]>, "Border, John" 
> <[email-masked]>
> *Subject: *RE: Is "Version Greasing" a new benfit or a new obstacle?
>
> Re. Encoding in CID. Then it needs to be standardized and if it is 
> standardized what’s the point in greasing? Let’s not ignore the 
> smaller services who use the cloud i.e. people deploying say NGINX web 
> servers in AWS, Azure or GCP. Not every QUIC deployment is a large 
> first party service which controls all aspects how LB and DDoS are 
> done. In public clouds LB and DDoS protection are network functions 
> (services) and are middleboxes under a completely different 
> administrative domain.
>
> Christian, this is about source address validation by the middlebox. 
> This is how it works for TCP (think SYN cookies) and we need to have 
> parity for QUIC. When under attack, short header packets will be 
> dropped for addresses that have not been validated, but this is the 
> same for non-SYN packets for TCP. I don’t understand how invariants 
> can be used here. If we don’t have parity for DoS protection in 
> network then QUIC traffic will be rate limited which is going to be 
> really bad for performance. FWIW there is already DoS traffic that 
> looks like QUIC.
>
> *From:* Roberto Peon <[email-masked]>
> *Sent:* Friday, April 12, 2019 9:16 AM
> *To:* Christian Huitema <[email-masked]>; Ian Swett 
> <[email-masked]>; Praveen Balasubramanian <[email-masked]>
> *Cc:* Gorry (erg) <[email-masked]>; Mirja Kuehlewind 
> <[email-masked]>; Mikkel Fahnøe Jørgensen 
> <[email-masked]>; [email-masked]; Border, John <[email-masked]>
> For most kinds of LB-ing, I’d expect the LB system to either directly 
> or indirectly be involved in choosing/vending connection IDs.
>
> *From: *Christian Huitema <[email-masked] 
> <mailto:[email-masked]>>
> *Date: *Friday, April 12, 2019 at 9:13 AM
> *To: *Ian Swett <ianswett=[email-masked] 
> <mailto:ianswett=[email-masked]>>, Praveen 
> Balasubramanian <pravb=[email-masked] 
> <mailto:pravb=[email-masked]>>
> *Cc: *"Gorry (erg)" <[email-masked] 
> <mailto:[email-masked]>>, Roberto Peon <[email-masked] 
> <mailto:[email-masked]>>, Mirja Kuehlewind 
> <[email-masked] 
> <mailto:[email-masked]>>, Mikkel Fahnøe Jørgensen 
> <[email-masked] <mailto:[email-masked]>>, "[email-masked] 
> <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>, 
> "Border, John" <[email-masked] <mailto:[email-masked]>>
> *Subject: *Re: Is "Version Greasing" a new benfit or a new obstacle?
>
>     Version greasing will create problems for DDoS protection. The
>     DDoS protection middlebox will have to treat all traffic with
>     unknown version numbers as plain old QUIC and rate limit it. Any
>     solution that requires per connection state to track version
>     numbers doesn’t scale. Especially in the cloud scenario customers
>     bring their own web servers so this makes any opt-out solution not
>     good enough. So I do think this is an obstacle for many deployments.
>
> that works now, and once you have found it deploy it without worry 
> about consequences -- which very often include ossification. If the 
> DDOS protection box looks for "QUIC versions that it knows" and just 
> rate limits the other types of packets, then the box will by design 
> prevent deployment of versions that it does not know.
>
> I assume that big organizations like Azure will update their 
> protections once a new version is out, but the experience shows that 
> many other organizations don't. And even when they do, the focus on 
> published versions will block experimenting with new versions, not to 
> mention interfering with other features like connection migration. I 
> take Praveen's example as an argument why we should in fact 
> standardize version greasing, precisely to avoid this "shortest path 
> middle-box design".
>
> authorized connections and packets that don't. And if you do that, 
> then you don't really need to look at the version numbers, you can 
> simply rely on the invariants.
>
> -- Christian Huitema
>

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Mon, 15 Apr 2019 11:19:48 +0200

hi Gorry, all,

> On 13 Apr 2019, at 13:03, G Fairhurst <[email-masked]> wrote:
> 
> Bringing this thread back - to me it seems there has been lots of discussion about how "version greasing" can be added in various ways, and very little on what makes this necessary and worth the impact on the network.
> 
> I still don't do not see that the side effects are worse when the protocol requires the version to be unobfusticated (as in most other IETF transport protocols).

+1. I continue to be confused about the amount of effort we're apparently willing to put into version negotiation protection of a protocol that is *explicitly* designed to be blockable outright (by virtue of the TCP fallback requirement). I haven't seen anything in this thread, or any of these discussions preceding it on this subject, that indicates that the actual risk of harm with respect to version blocking is discernibly worse (or even discernibly different) than the risk of harm with respect to blocking of the entire protocol regardless of version. 

I can see two reasons a version negotiation process could get blocked in the network:

(1) As an explicit choice by a traffic inspection function that it must have a specific version (or versions) of a protocol because it inspects information only available in that version of the wire image.

Note that QUIC is different from all other situations from which we could derive experience, since there simply isn't that much wire image for an inspection function to grab on to, and I can't imagine a future in which a future version would omit information it'd be worth blocking the version over (as opposed to all of QUIC, forcing TCP). Indeed, note that we made the latency spin bit probabilistic, as opposed to negotiated, in part to remove a potential incentive to ossify version negotiation.

It's possible that a device might want to block non-V1 versions because the handshake protection keys are hardcoded into the device. I'm not sure I see how dynamic aliasing could be reliably made to address this problem, though publishing v1 with N aliases (i.e. static aliasing) with different keys would at least ensure that devices with only one key slot don't work.

(2) As an overcorrection by a traffic inspection function that mistakes thinks that a particular sequence of version negotiation packets is a protocol identifier.

As I see it, the current version negotiation proposals only make this worse, not better, by creating a semantically indistinguishable set of aliases for the same operation. This problem could be nicely sidestepped by punting version negotiation into the future.

Cheers,

Brian

>> I’m not sure that I agree there—there is no need to have it standardized so long as the LB is involved in picking it. Each LB system can choose whatever it wants. Thus, every LB **deployment** could use a different scheme with different grease. Every LB deployment could change the scheme it uses pseudo-randomly or according to the lunar calendar, etc.
>> 
>> *To: *Roberto Peon <[email-masked]>, Christian Huitema <[email-masked]>, Ian Swett <[email-masked]>
>> *Cc: *"Gorry (erg)" <[email-masked]>, Mirja Kuehlewind <[email-masked]>, Mikkel Fahnøe Jørgensen <[email-masked]>, "[email-masked]" <[email-masked]>, "Border, John" <[email-masked]>
>> *To:* Christian Huitema <[email-masked]>; Ian Swett <[email-masked]>; Praveen Balasubramanian <[email-masked]>
>> *Cc:* Gorry (erg) <[email-masked]>; Mirja Kuehlewind <[email-masked]>; Mikkel Fahnøe Jørgensen <[email-masked]>; [email-masked]; Border, John <[email-masked]>
>> *From: *Christian Huitema <[email-masked] <mailto:[email-masked]><mailto:[email-masked] <mailto:[email-masked]>>>
>> *To: *Ian Swett <ianswett=[email-masked] <mailto:ianswett=[email-masked]><mailto:ianswett=[email-masked] <mailto:ianswett=[email-masked]>>>, Praveen Balasubramanian <pravb=[email-masked] <mailto:pravb=[email-masked]><mailto:pravb=[email-masked] <mailto:pravb=[email-masked]>>>
>> *Cc: *"Gorry (erg)" <[email-masked] <mailto:[email-masked]><mailto:[email-masked] <mailto:[email-masked]>>>, Roberto Peon <[email-masked] <mailto:[email-masked]><mailto:[email-masked] <mailto:[email-masked]>>>, Mirja Kuehlewind <[email-masked] <mailto:[email-masked]><mailto:[email-masked] <mailto:[email-masked]>>>, Mikkel Fahnøe Jørgensen <[email-masked] <mailto:[email-masked]><mailto:[email-masked] <mailto:[email-masked]>>>, "[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>" <[email-masked] <mailto:[email-masked]><mailto:[email-masked] <mailto:[email-masked]>>>, "Border, John" <[email-masked] <mailto:[email-masked]><mailto:[email-masked] <mailto:[email-masked]>>>

---

From: Brian Trammell (IETF) <[email-masked]>
Date: Mon, 15 Apr 2019 11:20:10 +0200

hi Mikkel, 

perhaps a bit late to this party, I'm not sure I understand the process you envision by which the TCP fallback will go away. 

Once it's in, it'll get used in a high enough proportion of cases that the availability risk of turning it off, balanced against the fact that it already works (as it's pretty much mandatory for deploying in today's 95%-UDP-transparent Internet), will go away very slowly indeed.

Cheers,

Brian

> On 11 Apr 2019, at 10:20, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> 
> On 11 April 2019 at 10.03.14, Mirja Kuehlewind ([email-masked]) wrote:
> 
>> Hi Mikkel, 
>>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 15 Apr 2019 02:28:47 -0700

QUIC is not going to be used only for HTTP/3 and I don’t see how TCP
fallback will work for the general case.

A lot of games, VR applications, streaming services etc. are going to build
semi-proprietary QUIC versions, or at least use QUIC with customised
application protocols where TCP fallback is simply pointless because it
doesn’t support the framing needed and if it were possible, why would adapt
an application layer protocol to work on TCP if you don’t have to - of
course you have to if it gets blocked - hence try to avoid that.

I don’t have any numbers on transparency, but given the reach of voice call
applications, you can probably safely assume that UDP will be supported
through most networks. The problem again is if only some UDP is allowed
through traffic inspection or at least port numbering.

It can be a good thing that a QUIC version is detectable if the alternative
is that anything unknown gets blocked. That is the big unknown. Will it
help to be is opaque as possible, or not? Some of the scenarios I posted
earlier show that no matter what you do, something will get in the way.

Mikkel

On 15 April 2019 at 11.20.25, Brian Trammell ([email-masked]) wrote:

hi Mikkel,

perhaps a bit late to this party, I'm not sure I understand the process you
envision by which the TCP fallback will go away.

Once it's in, it'll get used in a high enough proportion of cases that the
availability risk of turning it off, balanced against the fact that it
already works (as it's pretty much mandatory for deploying in today's
95%-UDP-transparent Internet), will go away very slowly indeed.

Cheers,

Brian

mobile apps would not.
>> The Great Firewall of China (GFC) is reconfigured to accept traffic from
a specific IP range for QUIC protocol v1.
>>
>>
>>
>> Google later upgrades to v2. GFC is updated. Later, out of ignorance,
some other department within said company decides that its browsers should
all use forced version aliases as a rule. GFC breaks. (Assuming Googles
servers are not located within the demilitarised
>> Alternative 1: Google up front decides to deploy forced version
aliasing. China rejects the deal because they do not want to support random
traffic through GFC.
>>
>>
>> Alternative 2: Google up front decides to deploy forced version
aliasing, and China does not reject the deal, out of ignorance. Later some
department in China GFC oversight realises the deal and forces Google to
remove forced version aliasing, or shut down
>>
>>
>> Alternative 4: Google realises that forced version aliasing does not
work in the general case, and makes a special case for users within a
certain geographical area.
>>
>>
>> Of course, to be fair, China is not the only government entity that
might have a vested interest. Let's take a fictional example of US a real
estate investor with heavy ties to the eastern european block who somehow
finds himself elected as president needing
>> the first to use a new digital election system. Not that the is an
issue, since trade negotiations are also running hot in that area, so that
is just an accidental bonus.
>> - We've seen how much fun ossification is in TCP and HTTP. If the thing
is observable, it will be ossified seems to be the lesson. A lot of the
reason why QUIC was started in the first place was because of the inability
to improve TCP due to this ossification.
>>
>> My opinion is that it is better to start with preventing ossification,
and then if that results in too large a percentage of operators blocking
things, to re-evaluate.
obstacle?

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Mon, 15 Apr 2019 13:14:58 +0200

> On 15 Apr 2019, at 11:28, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> 
> QUIC is not going to be used only for HTTP/3 and I don’t see how TCP fallback will work for the general case.
> 
> A lot of games, VR applications, streaming services etc. are going to build semi-proprietary QUIC versions, or at least use QUIC with customised application protocols where TCP fallback is simply pointless because it doesn’t support the framing needed and if it were possible, why would adapt an application layer protocol to work on TCP if you don’t have to - of course you have to if it gets blocked - hence try to avoid that.

So you see the deployment strategy for these applications to be "zero availability on non-UDP-accessible networks"? That seems bold to me unless you're talking about QUIC being used as a replacement for "application-provided transport layer over UDP", which is certainly a big part of the market but not all of it.

You could always tunnel UDP over TCP of course, but from a transport layer perspective (and a transport-visible wire image perspective), that's a TCP fallback.

> I don’t have any numbers on transparency, but given the reach of voice call applications, you can probably safely assume that UDP will be supported through most networks.

When we looked at this, admittedly a few years ago, we saw UDP blocking on par with the QUIC failure rates reported by Google in 2015 - O(3-6%) of paths. That's one nine of availability pathwise, which is... not great.

> The problem again is if only some UDP is allowed through traffic inspection or at least port numbering.
> 
> It can be a good thing that a QUIC version is detectable if the alternative is that anything unknown gets blocked. That is the big unknown.

As I said in my other message this morning, I don't think version negotiation greasing is useful at this time because there is no reasonable model of a wire-visible version difference where an on-path device might want to let through one greased-version and not another greased-version, because the wire-image profiles and semantic properties of those two versions are by definition of greasing identical. 

At this point, given connection-availability as the metric to optimize for, it seems like reducing features that might be mistaken for QUIC-distinguishers in the wire image would be useful, and version greasing seems like one such feature.

> Will it help to be is opaque as possible, or not? Some of the scenarios I posted earlier show that no matter what you do, something will get in the way.

This all comes down to the threat model, and the model of incentives for on-path devices.

I personally think the correct design target is to put trivial QUIC-discriminability into the wire image, since there will be a lot of pressure to distinguish QUIC from not-QUIC traffic regardless of what we do, and if we don't put grippy bits in the wire image, the distinguishing functions will grip on to the slippy bits, with difficult to predict outcomes. I recognize I am in the minority (of people who speak up) on this issue, though.

Cheers,

Brian

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 15 Apr 2019 04:31:09 -0700

Any kind of game or low latency control system such as 5G avionics cannot
use TCP as fallback since the buffering would be a non-starter.

I would expect UDP to be more commonly supported both as games with their
own protocols, WebRTC, and QUIC all become more prominent.
Also, it would be helpful to understand where UDP is not supported. For
example, it might be that UDP is blocked on some corporate firewalls
because conferencing is only supported on the internal network, or via VPN.
In these cases QUIC would probably still work perfectly fine for the
intended use cases, assuming the VPN solution does not introduce buffering
in packet tunnelling.

Adding TCP as a fallback isn’t merely tunneling data. It requires a new
QUIC emulation interface and it typically requires a separate host with
separate TLS certificate management. TCP typically runs in kernel where
QUIC runs in userspace and pulling in a userspace TCP stack is also work.
And then of course the latency considerations. So many deployments would
happily sacrifice the 5% of users that are not running UDP as they done
before with users running IE6.

>
> A lot of games, VR applications, streaming services etc. are going to
build semi-proprietary QUIC versions, or at least use QUIC with customised

So you see the deployment strategy for these applications to be "zero
availability on non-UDP-accessible networks"? That seems bold to me unless
you're talking about QUIC being used as a replacement for
"application-provided transport layer over UDP", which is certainly a big
part of the market but not all of it.

You could always tunnel UDP over TCP of course, but from a transport layer
perspective (and a transport-visible wire image perspective), that's a TCP
fallback.

> I don’t have any numbers on transparency, but given the reach of voice
call applications, you can probably safely assume that UDP will be
supported through most networks.

When we looked at this, admittedly a few years ago, we saw UDP blocking on
par with the QUIC failure rates reported by Google in 2015 - O(3-6%) of
paths. That's one nine of availability pathwise, which is... not great.

> It can be a good thing that a QUIC version is detectable if the
alternative is that anything unknown gets blocked. That is the big unknown.

As I said in my other message this morning, I don't think version
negotiation greasing is useful at this time because there is no reasonable
model of a wire-visible version difference where an on-path device might
want to let through one greased-version and not another greased-version,
because the wire-image profiles and semantic properties of those two
versions are by definition of greasing identical.

At this point, given connection-availability as the metric to optimize for,
it seems like reducing features that might be mistaken for
QUIC-distinguishers in the wire image would be useful, and version greasing
seems like one such feature.

> Will it help to be is opaque as possible, or not? Some of the scenarios I
posted earlier show that no matter what you do, something will get in the
way.

This all comes down to the threat model, and the model of incentives for
on-path devices.

I personally think the correct design target is to put trivial
QUIC-discriminability into the wire image, since there will be a lot of
pressure to distinguish QUIC from not-QUIC traffic regardless of what we
do, and if we don't put grippy bits in the wire image, the distinguishing
functions will grip on to the slippy bits, with difficult to predict
outcomes. I recognize I am in the minority (of people who speak up) on this
issue, though.

Cheers,

Brian

>> perhaps a bit late to this party, I'm not sure I understand the process
you envision by which the TCP fallback will go away.
>>
>> Once it's in, it'll get used in a high enough proportion of cases that
the availability risk of turning it off, balanced against the fact that it
already works (as it's pretty much mandatory for deploying in today's
95%-UDP-transparent Internet), will go away very slowly indeed.
[email-masked]) wrote:
>> >
>> >> Hi Mikkel,
>> >>
>> >> I guess it is much more likely that any of those actors would just
block QUIC all together (as there is actually a fallback to TCP).
[email-masked] on behalf of [email-masked]> wrote:
>> >>
>> >>
>> >> Let’s say China makes a deal with Google to allow their search engine
(a not entirely unreasonably proposition).
>> >> The Great Firewall of China (GFC) is reconfigured to accept traffic
from a specific IP range for QUIC protocol v1.
>> >> its service.
>> >>
>> >>
>> >> Alternative 3: Google up front decides to deploy forced version
aliasing and China says, of course, we trust everything from Googles IP
range.
>> >>
>> >>
>> >> Disclaimer: I’m not affiliated with Google and this might not at all
be how things would work, but it does highlight some of the challenges.
>> >> run heavily encrypted on the dark web with perpetual circulation of
IP ranges so, ignoring any legal concerns, it would not be practically
possible. Eventually someone figures out that these services all usually
are at the forefront of technology and currently
>> >> use QUIC v3.2 and no-one else has deployed that version yet, so it
would suffice to ask NSA to tap into the backbone and disrupt specific
packets. Of course, this ends up taking down the Bavarian local government
election process in Germany where they are
>> >>
>> >>
>> >> Or an endless number of other developments over the next few decades
if the past few years is anything to go by.
>> >> Thanks Mikkel, I do understand that various actors intentionally drop
- but are you saying these actors would specifically choose to block a new
version of QUIC ... I do not understand that assertion.
>> >> not do it now and to fix it if they did "by accident". Of course, I
am only one operator. It would be nice to hear from others...
>> >> CAUTION: This email originated from outside of the organization. Do
not click links or open attachments unless you recognize the sender and
know the content is safe.
>> >> - We've seen how much fun ossification is in TCP and HTTP. If the
thing is observable, it will be ossified seems to be the lesson. A lot of
the reason why QUIC was started in the first place was because of the
inability to improve TCP due to this ossification.
>> >>
>> >> My opinion is that it is better to start with preventing
ossification, and then if that results in too large a percentage of
operators blocking things, to re-evaluate.
>> >>
>> >> My guesses:
>> >> IP+port tuples and traffic patterns are still observable (for better
and worse), which implies operators will still have significant tools for
managing traffic. I believe that these are acted on/matched (ML or not)
regardless of any other data presented. In
obstacle?

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 15 Apr 2019 04:32:29 -0700

To be clear, I’m not saying that TCP fallback goes away from HTTP/3 - this
is not likely. But in this case infrastructure is available in abundance
already.

On 15 April 2019 at 13.31.09, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Cheers,

Brian

obstacle?

---

From: Ted Hardie <[email-masked]>
Date: Mon, 15 Apr 2019 08:28:55 -0700

Howdy,

> part of the market but not all of it.
There are classes of applications for which it is not particularly bold,
since they don't function well at all with TCP.

There are also classes where the fallback would be to a different stack.
DNS over QUIC might fall back to DNS over TLS if UDP is blocked, for
example.  QUIC doesn't need a fallback there because the DNS stack has one.

Ted

> At this point, given connection-availability as the metric to optimize
> for, it seems like reducing features that might be mistaken for
> > Will it help to be is opaque as possible, or not? Some of the scenarios
> I posted earlier show that no matter what you do, something will get in the
> way.
>
> >> >> Let’s say China makes a deal with Google to allow their search
> engine (a not entirely unreasonably proposition).
> >> >> Google later upgrades to v2. GFC is updated. Later, out of
> ignorance, some other department within said company decides that its
> browsers should all use forced version aliases as a rule. GFC breaks.
> (Assuming Googles servers are not located within the demilitarised
> >> >> Thanks Mikkel, I do understand that various actors intentionally
> drop - but are you saying these actors would specifically choose to block a
> new version of QUIC ... I do not understand that assertion.
> >> >>
> >> >>
> >> >> Gorry
> >> >>
> >> >> other words, I have a doubt that stating the version in an
> observable way will prevent the use of such tools.
> >> >>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Mon, 15 Apr 2019 15:35:03 +0000

Re. (1) I don’t think any inspection function will block versions of QUIC it doesn’t understand, although I understand that nothing prevents it from doing so desired. These versions should just be treated like general UDP traffic, so that we are no worse off from the way things are today in the network. 

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Brian Trammell (IETF)
Sent: Monday, April 15, 2019 2:20 AM
To: G Fairhurst <[email-masked]>
Cc: Praveen Balasubramanian <[email-masked]>; Ian Swett <[email-masked]>; Roberto Peon <[email-masked]>; Christian Huitema <[email-masked]>; Mirja Kuehlewind <[email-masked]>; Mikkel Fahnøe Jørgensen <[email-masked]>; [email-masked]; Border, John <[email-masked]>
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

hi Gorry, all,

Cheers,

Brian

>> *To: *Ian Swett <ianswett=[email-masked] <mailto:ianswett=[email-masked]><mailto:ianswett=[email-masked] <mailto:ianswett=[email-masked]>>>, Praveen Balasubramanian <pravb=[email-masked] <mailto:pravb=[email-masked]><mailto:pravb=[email-masked] <mailto:pravb=[email-masked]>>>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 15 Apr 2019 08:44:44 -0700

Sure, these examples are chosen exactly because they don’t work well with
TLS. If you argue that TLS fallback doesn’t always work, it is no good
coming up with examples where they do work, especially when it is not
contested that there are cases where fallback is valid.

Since QUIC is QUIC, any cloaking applies to all application classes. You
can’t argue cloaking is not needed because HTTP and DNS does not need it
due to fallback. Besides, it is an unproven assumption that cloaking is not
worthwhile for cases where fallback is required. If both peers accept QUIC
and the network generally does not block random appearing UDP cloaking
could work, although it does not guarantee privacy. Part of the concerns
are that when recognised as QUIC, only certain QUIC versions are permitted.
That could happen due to broken subsystems where packets go to bucket A or
B, where A is QUIC and B is everything else.

FWIW I’m not convinced that this version negotiation cloaking is worthwhile
or even beneficial, I’m just pointing out that whatever you do, it has
consequences.

On the specific example of DNS falling back to TLS. That might not work if
QUIC-DNS introduces a higher level of privacy. A fallback would be a
privacy leak. Of course, that would then break things on another level, so
it would be an opt-in, but if chosen, fallback should be disabled.

Mikkel

On 15 April 2019 at 17.29.22, Ted Hardie ([email-masked]) wrote:

Howdy,

Ted

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 15 Apr 2019 08:56:54 -0700

s/TLS/TCP/ first line

19 at 17.44.44, Mikkel Fahnøe Jørgensen ([email-masked]) wrote:

Mikkel

Howdy,

Ted

---

From: "Deval, Manasi" <[email-masked]>
Date: Mon, 15 Apr 2019 18:47:52 +0000

The current connection setup happens from client to server. Version negotiation draft states that:
the client
   maintains a list of QUIC versions it supports, ordered by preference.
   Its Initial packet is sent using the version that the server is most
   likely to support

The initiator of the greased version value is crucial for this definition. Is this agreed upon already? If the greased version value was chosen by a client, this can get inconsistent very quickly, so greased values must universally correspond to particular QUIC operational version.

For the middleboxes doing loadbalancing and firewall, they need to identify and classify incoming packets based on the IP, UDP and QUIC header fields for effective operation. We are already seeing that the length of connection Id cannot be deciphered in short header packets. Would the version number negotiation open up more issues like this.

Manasi

From: QUIC [mailto:[email-masked]] On Behalf Of Mikkel Fahnøe Jørgensen
Sent: Monday, April 15, 2019 8:57 AM
To: Ted Hardie <[email-masked]>; Brian Trammell (IETF) <[email-masked]>
Cc: Gorry Fairhurst <[email-masked]>; [email-masked]; Border, John <[email-masked]>; Roberto Peon <[email-masked]>; Mirja Kuehlewind <[email-masked]>
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

s/TLS/TCP/ first line

19 at 17.44.44, Mikkel Fahnøe Jørgensen ([email-masked]<mailto:[email-masked]>) wrote:
Sure, these examples are chosen exactly because they don’t work well with TLS. If you argue that TLS fallback doesn’t always work, it is no good coming up with examples where they do work, especially when it is not contested that there are cases where fallback is valid.

Since QUIC is QUIC, any cloaking applies to all application classes. You can’t argue cloaking is not needed because HTTP and DNS does not need it due to fallback. Besides, it is an unproven assumption that cloaking is not worthwhile for cases where fallback is required. If both peers accept QUIC and the network generally does not block random appearing UDP cloaking could work, although it does not guarantee privacy. Part of the concerns are that when recognised as QUIC, only certain QUIC versions are permitted. That could happen due to broken subsystems where packets go to bucket A or B, where A is QUIC and B is everything else.

FWIW I’m not convinced that this version negotiation cloaking is worthwhile or even beneficial, I’m just pointing out that whatever you do, it has consequences.

On the specific example of DNS falling back to TLS. That might not work if QUIC-DNS introduces a higher level of privacy. A fallback would be a privacy leak. Of course, that would then break things on another level, so it would be an opt-in, but if chosen, fallback should be disabled.

Mikkel

On 15 April 2019 at 17.29.22, Ted Hardie ([email-masked]<mailto:[email-masked]>) wrote:
Howdy,

On Mon, Apr 15, 2019 at 4:15 AM Brian Trammell (IETF) <[email-masked]<mailto:[email-masked]>> wrote:

> On 15 Apr 2019, at 11:28, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:
>

There are classes of applications for which it is not particularly bold, since they don't function well at all with TCP.

There are also classes where the fallback would be to a different stack.  DNS over QUIC might fall back to DNS over TLS if UDP is blocked, for example.  QUIC doesn't need a fallback there because the DNS stack has one.

Ted

Cheers,

Brian

> On 15 April 2019 at 11.20.25, Brian Trammell ([email-masked]<mailto:[email-masked]>) wrote:
>
>> hi Mikkel,
>>
>> > On 11 Apr 2019, at 10:20, Mikkel Fahnøe Jørgensen <[email-masked]<mailto:[email-masked]>> wrote:
>> >

---

From: Martin Duke <[email-masked]>
Date: Fri, 19 Apr 2019 11:32:48 -0700

> outcomes. I recognize I am in the minority (of people who speak up) on this
Can Brian (or anyone else) comment on what the threat model was that caused
middleboxes to drop TLS 1.3 when it used the actual version field? It seems
especially perverse for non-surveillance devices to drop future versions
that are, almost by definition, more secure. At this moment, my strong
support for greasing is driven my fear of this behavior that seems
nonsensical to me.

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 19 Apr 2019 19:58:59 +0000

>Can Brian (or anyone else) comment on what the threat model was that caused middleboxes to drop TLS 1.3 when it used the actual version field?

“safety”

Less flippantly, they didn’t understand the new version so they could not be sure it was okay to pass through, they didn’t know how to do content inspection, TLS offload, etc.

---

From: Martin Duke <[email-masked]>
Date: Fri, 19 Apr 2019 15:48:52 -0700

Right. This is the exact use case that version greasing is supposed to
avoid; i.e. a nontrivial amount of QUIC traffic greases, so middleboxes
that do this kind of thing impose unacceptable performance penalties on
users and don't make their way into the network.

On Fri, Apr 19, 2019 at 12:59 PM Salz, Rich <[email-masked]> wrote:

> *>*Can Brian (or anyone else) comment on what the threat model was that
> caused middleboxes to drop TLS 1.3 when it used the actual version field?
>
>
>
> “safety”
>
>
>
> Less flippantly, they didn’t understand the new version so they could not
> be sure it was okay to pass through, they didn’t know how to do content
> inspection, TLS offload, etc.
>
>
>

---

From: Gorry Fairhurst <[email-masked]>
Date: Sat, 20 Apr 2019 03:00:07 +0100

So... to check I understand that argument, it seems that some people 
expecs to see deployment of middleboxes that intercept QUIC - is that right?

- if the middlebox doesn't exist, because it can't usefully add value, 
the version number does not need to be encrypted.
- but if there are incentives to deploy middleboxes to do something 
(what?) then I could seee there could be a concern that these boxes 
could be implemented in a way that the implementers make them dependent 
on the version of QUIC.

Gorry

The first step is the part I don't understand, with TLS/TCP I understand 
the use-case existed.

On 19/04/2019, 23:48, Martin Duke wrote:
> avoid; i.e. a nontrivial amount of QUIC traffic greases, so 
> middleboxes that do this kind of thing impose unacceptable performance 
> penalties on users and don't make their way into the network.
>
> On Fri, Apr 19, 2019 at 12:59 PM Salz, Rich <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     *>*Can Brian (or anyone else) comment on what the threat model was
>     that caused middleboxes to drop TLS 1.3 when it used the actual
>     version field?
>
>     “safety”
>
>     Less flippantly, they didn’t understand the new version so they
>     could not be sure it was okay to pass through, they didn’t know
>     how to do content inspection, TLS offload, etc.
>

---

From: Watson Ladd <[email-masked]>
Date: Sat, 20 Apr 2019 00:35:22 -0700

> could be implemented in a way that the implementers make them dependent

The existence of the middlebox is sometimes a testament to the prowess
of the  sales team rather then the engineering team

-- 
"Man is born free, but everywhere he is in chains".
--Rousseau.