Subject: on HTTP/QUIC and HTTPBis

From: Patrick McManus <[email-masked]>
Date: Fri, 10 Mar 2017 14:38:36 -0500

On Thu, Mar 9, 2017 at 9:59 PM, Adrien de Croy <[email-masked]> wrote:

>
> for the benefit of those of us who maybe aren't so familiar with QUIC, are
> there any good resources you can refer us to which deal with the bigger
> picture such as why even have QUIC?
>
>
Thanks Adrien.

[minus quicwg, who doesn't need to hear a summary of their own work :) -
and changing the subject line to intentionally break threading]

I'm going to include an *opinion* here on the value of the quic work from
the perspective of the http ecosystem. I think its useful to have something
like this that isn't flavored quite as strongly by the Google experience as
most of the material out there is. (though we can only thank them for their
work and experience - its simply that diversity is valuable).  An important
element of the quic-wg charter is that they work closely with httpbis -
let's consider this part of that cooperation.

tl;dr; There are motivators both large and small - but for me the big 3
are: better perf under high packet loss, lower startup latency, better
streaming. This is largely the transport work leftover from http/2 that
couldn't be done in the TCP/TLS context this wg chose for h2.

A little bit of history: h2 made two really fundamental design choices. One
was to preserve HTTP/1 semantics (i.e. focus on transport issues), and the
other was to constrain that work by what could be done in a traditional
TLS/TCP context. This latter choice wasn't a forgone conclusion - there was
lots of side chatter during the pre-h2 bakeoff phase about whether anyone
would propose something more along the lines of sctp/udp/[d]tls (where RTC
Datachannels was headed contemporaneously). In the end folks felt that
experience and running code were important properties for h2 - building it
on tcp/tls lowered the risk of boiling the ocean, improved the latency of
getting something deployed, and captured meaningful low hanging fruit. For
the issues that it couldn't address there could always be another protocol
and we took seriously the need to create an effective means of versioning
to facilitate that. Rough consensus for a tcp/tls scope was obvious when
all the proposals turned out to be tcp/tls based :)

I think our operational experience has proved that to be a good decision.
h2 is a big win for a lot of cases, it has good uptake, its non-standard
predecessor has receded, and I don't think we could have done too much
better within a TCP/TLS context. Additionally, it can serve as a bootstrap
for 'another protocol' and the means for negotiating that via Alt-Svc and
ALPN which were both defined by the same effort if not the same rfc. That
'other protocol' turns out to be quic and its a good sign for the vibrancy
of http/2 that the quic charter explicitly seeks to map that h2 into its
ecosystem. QUIC seems to be taking on exactly the items that were foreseen
by the scoping of h2.

The chief performance frustration with h2 happens under higher than normal
packet loss. The in-order property of TCP spans different http transactions
in h2 - so a single packet loss in one message prevents subsequent
independent transactions from being delivered until the loss is repaired
(i.e. 2 images in 2 different packets with 1 packet loss can delay them
both.l). At the tail of high packet loss this makes h2 performance regress
from h1. Our community has seen some good data on this recently from Akamai
and Fastly. QUIC fixes this - it multiplexes multiple streams onto one
connection in a way very familiar to those used by 7540, but gives each
stream its own ordering context as if they were independent tcp streams so
they can be delivered independently.

Much of the work of the quic/h2 mapping is disentangling the parts of h2
that utilize the in-order property of the connection. .particularly the
relationship between control data and streams. hpack obviously has some
issues, and the extension-ack as a checkpoint is a problem too. Also
lifecycle counting issues (how do you count the number of open streams if
you can open them without permission but they can reordered around closes?)
etc.. Mike Bishop is doing a lot of the heavy lifting on this (and will
present for you all when we meet in Chicago - so come with opinions and
send pull requests to the quic wg! or at least read the document :)!) We
owe Mike our thanks and support for lots of careful work.

The other major h2 performance issue addressed by QUIC is startup latency.
Essentially it gives you common secure 0RTT connections for previously
visited hosts. You can just fire off N requests to a known host without
blocking on handshaking at the TCP or TLS layer while maintaining a very
small amount of closed state for the host (just a little TLS state, no H2
state at all). Congestion control is the only practical limit. Responses
come back in 1RTT. This is also resilient to things like NAT rebinding. A
world in which you deployed TCP Fast Open with TLS 1.3 Early Data would get
you some of the way to the same goal, but you would still be limited to
just a single packet in that first round trip and the h2 data would have to
share room in that packet with the TCP SYN and the TLS Client Hello - that
1 ethernet sized packet gets crowded real fast.

Speaking of improved latency, quic addresses a common misalignment between
typical streamed http data and TLS. QUIC framing is cognizant of UDP
boundaries and the crypto operations are done on a per UDP basis. The
implications here are important - when doing TLS in a TCP context the
atomic crypto operation is done on a TLS record (of relatively arbitrary
size - but 4k, 8k, 16k are common). Due to the packet agnostic byte-stream
interface of TCP, unless you worked amazingly super duper hard at it these
records commonly span packet boundaries - but the delivery time of the
application level data (e.g. progressive images) was dependent on the
arrival of the last packet in the record. The application would incur ~a
round trip of latency rendering partially received data whenever a
congestion window limit was hit even without loss or reordering.. in the
presence of loss or reordering it can get really bad. Ilya has done some
great writing on this topic in the past - but implementations still
commonly do it poorly. QUIC automatically optimally aligns the crypto with
the UDP boundary (and has some PMTU magic that will generally ensure UDP is
not fragmented).

So those are the big 3: more stable under high packet loss, lower startup
latency, better streaming. They're really big :) But there's more -

Blending the application and the transport has some advantages - I think
the chief one here is that the application is going to have a much better
idea of how to manage its buffers as it will have closer access to ack
feedback loops. We've seen this several times in H2 implementations where
code doesn't know how to gauge a BDP worth of data and ends up creating
huge amounts of buffering which impact its own ability to multiplex,
cancel, and honor priority (which are afterall the major features of h2).
The result can be horrible responsiveness. We will have a presenter in
Chicago talk about some related academic measurements around this topic as
part of our effort to share feedback on the working group's past outputs.
This is a subtle thing as I think you can do approximately-the-right-thing
in both h2 and quic, but the quic architecture should make it easier to get
quality implementations.

I also expect that a lot of the weird corner cases that result in data loss
in h1/h2 around TCP RST will be eliminated. (RSTs are often routinely
generated by application errors and sometimes result in silent kernel
discard of data that has already been received and acked but not yet
delivered to the application - this combined with large TLS records can
sometimes result in the loss of a lot of data. Its a pretty weak part of
TCP.)

Further, TCP Tuning for HTTP (1 or 2) is operationally complicated. One of
the complications is the administrative separation between transport and
application. How do you set IW? Are you doing TLP? RACK? Is Fast Open an
option for you? QUIC, because it can be bundled with an application in
userspace, provides a deployment option for getting best of breed transport
algorithms deployed to hosts that may be running legacy TCP stacks. You can
expect a more consistent "best practices" experience across platforms and a
pivot for iterating on new designs and ideas for evolution. Speaking of
evolution, quic is almost entirely encrypted and basically fully
authenticated. This intentionally exposes very few bits to the network with
the intention of preventing ossification around patterns introduced in v1
the way things. Ossification has been a huge problem for TCP and http and
we'd like to do better.

I'll wrap it up by noting that blending transport and crypto has some
advantages beyond packet alignment too - perhaps my favorite is that it
seems likely quic will be resistant to RST injections and the modification
of basic congestion control by things like peps. (you may or may not
consider than an advantage - I'm aware.) Though it will be vulnerable to
downgrade attacks to TCP in the same way h2 is vulnerable to downgrade
attacks to h1 :(

I think the question for our working group is how we want to engage with
all this change - as part of that you can expect regular updates about the
quic/http mapping to be part of our process for a while and Mike is doing a
great job of making clear choices like how quic interacts with the legacy
registries more visible in this audience. I'd like to make it possible for
those with only an interest in the HTTP portions of this (to whatever
extent that is possible) to be able to share their experience without
joining the full QUIC-WG firehose - let Mark and I know if you have any
thoughts on how to further facilitate that beyond what we're doing.

So far our experience with QUIC (as a proxy) has been to simply block it,
> since it provides a bypass mechanism for proxy control.
>
>
Obviously this hasn't been a proxy centric review, but there is no
fundamental reason a client can't speak QUIC to a proxy in the same way
some of them support h2 to the proxy now. These are not changes in
semantics. I would hope to see it for any client with forward proxy support
but it might not be the first thing rolled out.

-Patrick

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 10 Mar 2017 20:22:26 +0000

--------
In message <[email-masked]>
, Patrick McManus writes:

Thanks for an excellent summary of QUIC.

I am amazed by how unashamed you applaud movement, nay, rush towards
"mixing application and transport" and "mixing application and
crypto" considering that our entire 50 year experience in computing
has consistently shown that to be The Worst Idea Ever, no matter
how many times people try it.

Given the current state of affairs, I find that course even more
baffeling in light of IETFs snobby "We don't do APIs" attitude,
which is not going to make it any easier for the people who will
be writing the applicryptotransications.

>all this change - [...]

May I suggest: "Recoil in Horror" ?

Poul-Henning

PS: Have you considered sending your (excellent!) summary of QUIC
to RISKS Digest preemptively ?  I for one would love to hear PGN's
reaction...

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 10 Mar 2017 21:39:35 +0100

Hi Patrick,

> let's consider this part of that cooperation.
(...)

Thank you very much for this detailed overview. It's more or less what
I was already aware of, but much more clear :-)

> algorithms deployed to hosts that may be running legacy TCP stacks.
(...)

In fact, as one developer having encountered many difficulties when
trying to implement H2 on an existing proxy, especially during the
early days, I think that being forced to implement a new transport layer
will give many of us a good excuse for breaking a lot of existing code,
resulting in more flexibility and choice for the new implementations.
This will definitely come with a cost but starting from scratch is often
easier than bending existing stuff enough so that it more or less adapts.

As a WG participant I suffered from not being able to experiment with
H2 in the product I cared about during the design discussions. There
was so much an offset between running implementations and those
scratching their heads trying to identify what to break to approach
the goal that the design was mostly driven by existing implementations
and those written from scratch. Here we may have a chance to involve
more people since a lot of work will have to be implemented from scratch
in most products, granting a bit more of "equity" in front of the design
constraints.

I personnaly am not yet convinced about the performance a program like
haproxy could achieve by dealing with packets in userland, there are
pros and cons. However I'm seeing a benefit. We're seeing more and more
applications relying on frameworks like DPDK though these ones require
you to add a full TCP stack which generally requires totally different
semantics. That's why often they're just switches and routers. Mapping
UDP in such a packet-oriented framework can be much easier and I predict
we'll see much more adoption there.

Best regards,
Willy

---

From: Alex Rousskov <[email-masked]>
Date: Fri, 10 Mar 2017 14:34:14 -0700

On 03/10/2017 12:38 PM, Patrick McManus wrote:
> there is no fundamental reason a client can't speak QUIC to a proxy in
> the same way some of them support h2 to the proxy now.

As the recent "how to block h2 without MitM" thread illustrates,
"speaking in the same way" is mostly useless when it comes to proxies.
If QUIC folks want to support proxies properly, they have to design the
protocol to allow for trusted proxies. Besides politics, there are
serious technical problems that neither HTTP/1 nor HTTP/2 solve in that
space.

Alex.

---

From: Mike Bishop <[email-masked]>
Date: Sat, 11 Mar 2017 00:16:33 +0000

While Patrick approaches it from a standpoint of how we can squeeze out extra performance by intertwining the layers, I see them as different stand-alone system binaries owned by different teams, and as things we will almost certainly want to modularize for different uses.  And there's value in diversity of viewpoints, since Patrick and I are in very different implementation situations despite both sitting on the browser side.

That's a big reason why the original monolithic design of QUIC has been separated into a transport draft which has pluggable crypto and flow control algorithms, drafts defining instances of those algorithms, and an HTTP mapping on top of that transport.  I'm trying to make the separation between the drafts as clean as possible, because I know that in my system they will remain separate components that want to have an interface that's as minimal as possible.  TLS concerns should remain at the TLS layer, and all the transport should know is what bytes to relay and when the keys have been generated.  HTTP should not *have to* know or care about flow control windows, ACKs, etc.

Patrick's also correct that, if the components' interfaces aren't minimal, you might be able to get some interesting efficiencies by pulling out extra information.  But I'd argue the RFCs should not assume that -- those are optimizations an implementation can make locally, at its own risk, which should not influence wire compatibility.  If you'd like to step over into the QUIC side of things and argue for clean separation, we'd love to have your input.

-----Original Message-----
From: Poul-Henning Kamp [mailto:[email-masked]] 
Sent: Friday, March 10, 2017 12:22 PM
To: Patrick McManus <[email-masked]>
Cc: Adrien de Croy <[email-masked]>; Mike Bishop <[email-masked]>; HTTP working group mailing list <[email-masked]>
Subject: Re: on HTTP/QUIC and HTTPBis

--------

I am amazed by how unashamed you applaud movement, nay, rush towards "mixing application and transport" and "mixing application and crypto" considering that our entire 50 year experience in computing has consistently shown that to be The Worst Idea Ever, no matter how many times people try it.

Given the current state of affairs, I find that course even more baffeling in light of IETFs snobby "We don't do APIs" attitude, which is not going to make it any easier for the people who will be writing the applicryptotransications.

>I think the question for our working group is how we want to engage 
>with all this change - [...]

Poul-Henning

PS: Have you considered sending your (excellent!) summary of QUIC to RISKS Digest preemptively ?  I for one would love to hear PGN's reaction...

---

From: Mike Bishop <[email-masked]>
Date: Sat, 11 Mar 2017 00:23:51 +0000

Which are orthogonal to QUIC, except that proxies will be even less able to be transparent -- and I consider that an advantage.

If you define something like an INSPECTED_CONNECT method that works over HTTP/1.1 or HTTP/2, it will work over HTTP/QUIC.  If you try to MitM, you'll have all the same problems that exist with TLS MitM today.  I'm sympathetic to the lack of configured proxies' abilities to, with user/client awareness, monitor the contents of transactions which are otherwise encrypted and return meaningful errors.  I'm less sympathetic to middleboxes that act without client awareness or configuration.

-----Original Message-----
From: Alex Rousskov [mailto:[email-masked]] 
Sent: Friday, March 10, 2017 1:34 PM
To: Patrick McManus <[email-masked]>
Cc: HTTP working group mailing list <[email-masked]>
Subject: Re: on HTTP/QUIC and HTTPBis

> the same way some of them support h2 to the proxy now.

As the recent "how to block h2 without MitM" thread illustrates, "speaking in the same way" is mostly useless when it comes to proxies.
If QUIC folks want to support proxies properly, they have to design the protocol to allow for trusted proxies. Besides politics, there are serious technical problems that neither HTTP/1 nor HTTP/2 solve in that space.

Alex.

---

From: Patrick McManus <[email-masked]>
Date: Fri, 10 Mar 2017 20:45:04 -0500

Hey Mike, I really don't think we're that far apart except maybe in your
use of the term layers. quic is being effectively componentized but not
into layers that create full abstractions in the way a traditional stack
works. So while transport and crypto are definitely blended (you cannot
implement the transport without doing both transport'y things and crypto
things and then transporty things again) I wouldn't call them layers. I'm
trying out the term blended functions :) ietf quic has effectively been
able to use the services of TLS 1.3 without modification - and thank
goodness as we really don't need another security protocol - but the
layering of cleartext->ciphertext->packets no longer exists (retransmits
being an easy example). But these components definitely have logical
interfaces and can be expressed in code in many many different ways which I
hope makes them suitable to a great deal of diverse environments. This
seems pretty close to your pluggable component vision and I support that as
at least a model for describing what's going on.

That's all really an interesting side-discussion distracting from the
original question (roughly: what feature motivates this thing?) I'm
curious, do you have different answers?

On Fri, Mar 10, 2017 at 7:16 PM, Mike Bishop <[email-masked]>
wrote:

> While Patrick approaches it from a standpoint of how we can squeeze out
> extra performance by intertwining the layers, I see them as different
> stand-alone system binaries owned by different teams, and as things we will
> almost certainly want to modularize for different uses.  And there's value
> in diversity of viewpoints, since Patrick and I are in very different
> implementation situations despite both sitting on the browser side.
>
> That's a big reason why the original monolithic design of QUIC has been
> separated into a transport draft which has pluggable crypto and flow
> control algorithms, drafts defining instances of those algorithms, and an
> HTTP mapping on top of that transport.  I'm trying to make the separation
> between the drafts as clean as possible, because I know that in my system
> they will remain separate components that want to have an interface that's
> as minimal as possible.  TLS concerns should remain at the TLS layer, and
> all the transport should know is what bytes to relay and when the keys have
> been generated.  HTTP should not *have to* know or care about flow control
> windows, ACKs, etc.
>
> Patrick's also correct that, if the components' interfaces aren't minimal,
> you might be able to get some interesting efficiencies by pulling out extra
> information.  But I'd argue the RFCs should not assume that -- those are
> optimizations an implementation can make locally, at its own risk, which
> should not influence wire compatibility.  If you'd like to step over into
> the QUIC side of things and argue for clean separation, we'd love to have
> your input.
>
> Cc: Adrien de Croy <[email-masked]>; Mike Bishop <
> [email-masked]>; HTTP working group mailing list <
> [email-masked]>
> "mixing application and transport" and "mixing application and crypto"
> considering that our entire 50 year experience in computing has
> consistently shown that to be The Worst Idea Ever, no matter how many times
> people try it.
>
> Given the current state of affairs, I find that course even more baffeling
> in light of IETFs snobby "We don't do APIs" attitude, which is not going to
> make it any easier for the people who will be writing the
> applicryptotransications.
>
> PS: Have you considered sending your (excellent!) summary of QUIC to RISKS
> Digest preemptively ?  I for one would love to hear PGN's reaction...
>
> --

---

From: Patrick McManus <[email-masked]>
Date: Fri, 10 Mar 2017 21:00:43 -0500

Hey Willy -

> we'll see much more adoption there.

I agree that the CPU/scheduling aspect of userspace is the one we least
understand and has the biggest range of possible outcomes particularly for
different platforms. It will be interesting times.  But as you mention,
there are a number of examples that suggest this is a real opportunity but
I don't think of it as a motivator.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 11 Mar 2017 07:07:29 +0000

--------
In message <[email-masked].
outlook.com>, Mike Bishop writes:

> I'm sympathetic to the lack of configured proxies' abilities to,
> with user/client awareness, monitor the contents of transactions
> which are otherwise encrypted and return meaningful errors.  I'm
> less sympathetic to middleboxes that act without client awareness
> or configuration.

If so, wouldn't it make sense to proactively make it easier to
roll out the former than the latter, rather than make it equally
hard ?

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 11 Mar 2017 07:39:54 +0000

--------
In message <[email-masked].
outlook.com>, Mike Bishop writes:

>If you'd like to step over into the
>QUIC side of things and argue for clean separation, we'd love to have your
>input.

My input is this:

1. The more complex you make HTTP, whatever you name it, the more
   broken shit is going to be deployed, putting everybodys privacy and
   personal security at risk from attacks by global players, criminal
   and "legal", which we as society have no way to hold accountable.

2. The so-called "advances in privacy" brought about by IETFs "SSL
   all the things" agenda means that it is becomming close to
   impossible to hold big transnational companies like FaceBook,
   Google, Apple and Microsoft accountable for their industrial
   scale violations of privacy.  "Sunlight is the best disinfectant"
   but IETF is busy pulling down all the blinds.

3. Whether IETFs membership realizes it or not, every new application
   of cryptography is an argument in a political debate about privacy
   as a basic human right, and the proponents of such a human right
   are loosing that debate, because we leave the other side no other
   options than breaking all crypto open by law.

4. If you, as I do, belive in a fundamental human right to privacy,
   that argument should be raised loud and clear in the political
   debate, no matter how silly and dirty that debate is.  Advancing
   your agenda by abusing an accidentally powerfull position without
   asking the other 99.99% of the population how they feel about
   it didn't work for USSR and it wont work for IETF.

And yes, feel free to dismiss this with the usual "We don't do
politics in IETF", but before you do that, remember that you don't
get to define if what you do is "political" or not:  It becomes
political the moment somebody opposes what you do, for political
reasons.

---

From: Stefan Eissing <[email-masked]>
Date: Sat, 11 Mar 2017 09:28:43 +0100

Mike and Patrick,

very well written and very interesting to read your views on this matter. Thank you!

I look forward to see how these protocol/components continue to evolve and very curious at implementation prototypes outside google and how this userland packet handling will be tackled. Eventually, I expect some reusable components to emerge so that we will have some software abstractions that everyone can build upon, again.

Patrick's mention of buffer management I can very much relate to as it is one of the headaches for h2 implementations I try to tackle myself. I look forward to find out if the snake will eat its tail when we go from 6 tcp flows (h1) to 100 flows in 1 tcp (h2) to 100(0?) udp flows (hq).

When I wrote earlier to "abandon all hope" to get h2 and hq perfectly aligned, I meant that more positive than it sounds (excuse my german). What I wanted to say is that treating them as the same in regards to frame definitions/registry is not a good idea and breaking with this is good.

-stefan

> Am 11.03.2017 um 01:16 schrieb Mike Bishop <[email-masked]>:
>

---

From: "Adrien de Croy" <[email-masked]>
Date: Sun, 12 Mar 2017 21:28:02 +0000

Hi Patrick

thanks for taking the time to write all that up.

I can see the strong desire to reduce latency / RTTs in setup phase pre 
application data, but it makes me worry about a few things.  SYN flood 
attacks were rendered ineffective by SYN cookies, which meant the server 
didn't need to maintain state about a connection until after the ACK to 
the SYN ACK was received, thereby precluding spoofing source IP.

However if the initial packet has crypto setup as well, the server will 
need to record and retain client state from the initial packet which 
will open it up to a SYN flood style of resource consumption attack?

I also wonder why we need to keep redesigning various mux layers on top 
of IP.  IP is already multiplexing all the TCP connections.  We decided 
for whatever reason that too many TCP connections is bad, and so we went 
to sub-streams (h2) and other mechanisms to provide TCP style behaviour 
without somehow re-introducing the same things that made us decide too 
many TCP connections is bad.  I just wonder whether it won't come to 
pass that we decide too many streams on QUIC is bad, and we need another 
different mux layer.

the obvious question is can't we just fix TCP so that lots of 
connections isn't so bad?

We keep adding more layers and overhead as well, maybe it's time to make 
the internet do bigger packets :)

Adrien

------ Original Message ------
From: "Patrick McManus" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "Mike Bishop" <[email-masked]>; "HTTP working group 
mailing list" <[email-masked]>
Sent: 11/03/2017 8:38:36 AM
Subject: on HTTP/QUIC and HTTPBis

>>for the benefit of those of us who maybe aren't so familiar with QUIC, 
>>are there any good resources you can refer us to which deal with the 
>>bigger picture such as why even have QUIC?
>>
>
>Thanks Adrien.
>
>[minus quicwg, who doesn't need to hear a summary of their own work :) 
>- and changing the subject line to intentionally break threading]
>
>I'm going to include an opinion here on the value of the quic work from 
>the perspective of the http ecosystem. I think its useful to have 
>something like this that isn't flavored quite as strongly by the Google 
>experience as most of the material out there is. (though we can only 
>thank them for their work and experience - its simply that diversity is 
>valuable).  An important element of the quic-wg charter is that they 
>work closely with httpbis - let's consider this part of that 
>cooperation.
>
>A little bit of history: h2 made two really fundamental design choices. 
>One was to preserve HTTP/1 semantics (i.e. focus on transport issues), 
>and the other was to constrain that work by what could be done in a 
>traditional TLS/TCP context. This latter choice wasn't a forgone 
>conclusion - there was lots of side chatter during the pre-h2 bakeoff 
>phase about whether anyone would propose something more along the lines 
>of sctp/udp/[d]tls (where RTC Datachannels was headed 
>contemporaneously). In the end folks felt that experience and running 
>code were important properties for h2 - building it on tcp/tls lowered 
>the risk of boiling the ocean, improved the latency of getting 
>something deployed, and captured meaningful low hanging fruit. For the 
>issues that it couldn't address there could always be another protocol 
>and we took seriously the need to create an effective means of 
>versioning to facilitate that. Rough consensus for a tcp/tls scope was 
>obvious when all the proposals turned out to be tcp/tls based :)
>
>I think our operational experience has proved that to be a good 
>decision. h2 is a big win for a lot of cases, it has good uptake, its 
>non-standard predecessor has receded, and I don't think we could have 
>done too much better within a TCP/TLS context. Additionally, it can 
>serve as a bootstrap for 'another protocol' and the means for 
>negotiating that via Alt-Svc and ALPN which were both defined by the 
>same effort if not the same rfc. That 'other protocol' turns out to be 
>quic and its a good sign for the vibrancy of http/2 that the quic 
>charter explicitly seeks to map that h2 into its ecosystem. QUIC seems 
>to be taking on exactly the items that were foreseen by the scoping of 
>h2.
>
>The chief performance frustration with h2 happens under higher than 
>normal packet loss. The in-order property of TCP spans different http 
>transactions in h2 - so a single packet loss in one message prevents 
>subsequent independent transactions from being delivered until the loss 
>is repaired (i.e. 2 images in 2 different packets with 1 packet loss 
>can delay them both.l). At the tail of high packet loss this makes h2 
>performance regress from h1. Our community has seen some good data on 
>this recently from Akamai and Fastly. QUIC fixes this - it multiplexes 
>multiple streams onto one connection in a way very familiar to those 
>used by 7540, but gives each stream its own ordering context as if they 
>were independent tcp streams so they can be delivered independently.
>
>Much of the work of the quic/h2 mapping is disentangling the parts of 
>h2 that utilize the in-order property of the connection. .particularly 
>the relationship between control data and streams. hpack obviously has 
>some issues, and the extension-ack as a checkpoint is a problem too. 
>Also lifecycle counting issues (how do you count the number of open 
>streams if you can open them without permission but they can reordered 
>around closes?) etc.. Mike Bishop is doing a lot of the heavy lifting 
>on this (and will present for you all when we meet in Chicago - so come 
>with opinions and send pull requests to the quic wg! or at least read 
>the document :)!) We owe Mike our thanks and support for lots of 
>careful work.
>
>The other major h2 performance issue addressed by QUIC is startup 
>latency. Essentially it gives you common secure 0RTT connections for 
>previously visited hosts. You can just fire off N requests to a known 
>host without blocking on handshaking at the TCP or TLS layer while 
>maintaining a very small amount of closed state for the host (just a 
>little TLS state, no H2 state at all). Congestion control is the only 
>practical limit. Responses come back in 1RTT. This is also resilient to 
>things like NAT rebinding. A world in which you deployed TCP Fast Open 
>with TLS 1.3 Early Data would get you some of the way to the same goal, 
>but you would still be limited to just a single packet in that first 
>round trip and the h2 data would have to share room in that packet with 
>the TCP SYN and the TLS Client Hello - that 1 ethernet sized packet 
>gets crowded real fast.
>
>Speaking of improved latency, quic addresses a common misalignment 
>between typical streamed http data and TLS. QUIC framing is cognizant 
>of UDP boundaries and the crypto operations are done on a per UDP 
>basis. The implications here are important - when doing TLS in a TCP 
>context the atomic crypto operation is done on a TLS record (of 
>relatively arbitrary size - but 4k, 8k, 16k are common). Due to the 
>packet agnostic byte-stream interface of TCP, unless you worked 
>amazingly super duper hard at it these records commonly span packet 
>boundaries - but the delivery time of the application level data (e.g. 
>progressive images) was dependent on the arrival of the last packet in 
>the record. The application would incur ~a round trip of latency 
>rendering partially received data whenever a congestion window limit 
>was hit even without loss or reordering.. in the presence of loss or 
>reordering it can get really bad. Ilya has done some great writing on 
>this topic in the past - but implementations still commonly do it 
>poorly. QUIC automatically optimally aligns the crypto with the UDP 
>boundary (and has some PMTU magic that will generally ensure UDP is not 
>fragmented).
>
>So those are the big 3: more stable under high packet loss, lower 
>startup latency, better streaming. They're really big :) But there's 
>more -
>
>Blending the application and the transport has some advantages - I 
>think the chief one here is that the application is going to have a 
>much better idea of how to manage its buffers as it will have closer 
>access to ack feedback loops. We've seen this several times in H2 
>implementations where code doesn't know how to gauge a BDP worth of 
>data and ends up creating huge amounts of buffering which impact its 
>own ability to multiplex, cancel, and honor priority (which are 
>afterall the major features of h2). The result can be horrible 
>responsiveness. We will have a presenter in Chicago talk about some 
>related academic measurements around this topic as part of our effort 
>to share feedback on the working group's past outputs. This is a subtle 
>thing as I think you can do approximately-the-right-thing in both h2 
>and quic, but the quic architecture should make it easier to get 
>quality implementations.
>
>I also expect that a lot of the weird corner cases that result in data 
>loss in h1/h2 around TCP RST will be eliminated. (RSTs are often 
>routinely generated by application errors and sometimes result in 
>silent kernel discard of data that has already been received and acked 
>but not yet delivered to the application - this combined with large TLS 
>records can sometimes result in the loss of a lot of data. Its a pretty 
>weak part of TCP.)
>
>Further, TCP Tuning for HTTP (1 or 2) is operationally complicated. One 
>of the complications is the administrative separation between transport 
>and application. How do you set IW? Are you doing TLP? RACK? Is Fast 
>Open an option for you? QUIC, because it can be bundled with an 
>application in userspace, provides a deployment option for getting best 
>of breed transport algorithms deployed to hosts that may be running 
>legacy TCP stacks. You can expect a more consistent "best practices" 
>experience across platforms and a pivot for iterating on new designs 
>and ideas for evolution. Speaking of evolution, quic is almost entirely 
>encrypted and basically fully authenticated. This intentionally exposes 
>very few bits to the network with the intention of preventing 
>ossification around patterns introduced in v1 the way things. 
>Ossification has been a huge problem for TCP and http and we'd like to 
>do better.
>
>seems likely quic will be resistant to RST injections and the 
>modification of basic congestion control by things like peps. (you may 
>or may not consider than an advantage - I'm aware.) Though it will be 
>vulnerable to downgrade attacks to TCP in the same way h2 is vulnerable 
>to downgrade attacks to h1 :(
>
>with all this change - as part of that you can expect regular updates 
>about the quic/http mapping to be part of our process for a while and 
>Mike is doing a great job of making clear choices like how quic 
>interacts with the legacy registries more visible in this audience. I'd 
>like to make it possible for those with only an interest in the HTTP 
>portions of this (to whatever extent that is possible) to be able to 
>share their experience without joining the full QUIC-WG firehose - let 
>Mark and I know if you have any thoughts on how to further facilitate 
>that beyond what we're doing.
>
>>So far our experience with QUIC (as a proxy) has been to simply block 
>>it, since it provides a bypass mechanism for proxy control.
>>
>
>semantics. I would hope to see it for any client with forward proxy 
>support but it might not be the first thing rolled out.
>
>-Patrick
>

---

From: "Adrien de Croy" <[email-masked]>
Date: Sun, 12 Mar 2017 21:32:51 +0000

Yes, I think if we take the starting point that

* a company that is worried about cyber-slacking of employees
* a school worried about students surfing porn
* a mom and dad worried about their kids spending too much time watching 
minecraft videos on youtube

are going to block sites _no_matter_what_ we do, then we should make it 
possible to do this cleanly, and reliably without having to resort to 
nasty tricks.

Because if we don't allow people to block at protocol level, they will 
block at transport level, which will result in over-blocking, and very 
poor user experience.  It is not our place to say people should not 
block, and deny people the facility to do it properly, we are here and 
involved in protocol work for the people.  This includes people who need 
for completely legitimate reasons to block things.

Adrien

------ Original Message ------
From: "Poul-Henning Kamp" <[email-masked]>
To: "Mike Bishop" <[email-masked]>
Cc: "Alex Rousskov" <[email-masked]>; "Patrick 
McManus" <[email-masked]>; "HTTP working group mailing list" 
<[email-masked]>
Sent: 11/03/2017 8:07:29 PM
Subject: Re: on HTTP/QUIC and HTTPBis

>--------
>In message 
><[email-masked].

---

From: Mike Bishop <[email-masked]>
Date: Mon, 13 Mar 2017 17:39:33 +0000

All that is true.  However, it's not an issue of QUIC.  Yes, having transport-level encryption would make sense to make those things easier to do "the right way" -- and part of what QUIC provides is making it harder to do "the wrong way."  "The right way" is an HTTP-level concern (how clients and proxies interact), not a transport-level concern (clients talk to proxies and servers in the same way).

-----Original Message-----
From: Adrien de Croy [mailto:[email-masked]] 
Sent: Sunday, March 12, 2017 2:33 PM
To: Poul-Henning Kamp <[email-masked]>; Mike Bishop <[email-masked]>
Cc: Alex Rousskov <[email-masked]>; Patrick McManus <[email-masked]>; HTTP working group mailing list <[email-masked]>
Subject: Re: on HTTP/QUIC and HTTPBis

* a company that is worried about cyber-slacking of employees
* a school worried about students surfing porn
* a mom and dad worried about their kids spending too much time watching minecraft videos on youtube

are going to block sites _no_matter_what_ we do, then we should make it possible to do this cleanly, and reliably without having to resort to nasty tricks.

Because if we don't allow people to block at protocol level, they will block at transport level, which will result in over-blocking, and very poor user experience.  It is not our place to say people should not block, and deny people the facility to do it properly, we are here and involved in protocol work for the people.  This includes people who need for completely legitimate reasons to block things.

Adrien

------ Original Message ------
From: "Poul-Henning Kamp" <[email-masked]>
To: "Mike Bishop" <[email-masked]>
Cc: "Alex Rousskov" <[email-masked]>; "Patrick McManus" <[email-masked]>; "HTTP working group mailing list" 
<[email-masked]>

>> less sympathetic to middleboxes that act without client awareness  or 
>> configuration.
>
>If so, wouldn't it make sense to proactively make it easier to roll out 
>the former than the latter, rather than make it equally hard ?
>
>--

---

From: Mike Bishop <[email-masked]>
Date: Mon, 13 Mar 2017 22:52:47 +0000

For me, the biggest answer is agility.  The reality is that we’ve heavily invested in our TCP stacks, and QUIC going over UDP is likely to require time to catch up, so I’m unconvinced of an instant performance benefit.  I’d be thrilled if we even get parity with TFO+TLS 1.3+HTTP/2 at first.  But there’s been good work happening in TCP for years that isn’t reliably deployable on the Internet – how long has TFO been an RFC?  And yet, it spent a long time off by default in each OS.  In Windows, we aren’t turning it on by default until we’ve added mitigations for the middleboxes that don’t play well when they see a different TCP behavior than they expected.  They exist, and they’re numerous.

HTTP has faced the same problem.  We (Microsoft) have HTTP features that we recommend not be deployed over the Internet, because we know of middleboxes that panic or behave badly.  More recently, several new standards have been, formally or informally, restricted to HTTPS-only for the same reason.  We’re freer to innovate inside the HTTPS tunnel, because there’s better control of which devices can see those bits.

By having encryption and integrity protection at the transport level, the transport gets the same freedom to innovate in the future.  And HTTP will reap the benefit of those future improvements over time.  It’s an investment, but one I hope will eventually pay dividends.

From: Patrick McManus [mailto:[email-masked]]
Sent: Friday, March 10, 2017 5:45 PM
To: Mike Bishop <[email-masked]>
Cc: Poul-Henning Kamp <[email-masked]>; Patrick McManus <[email-masked]>; Adrien de Croy <[email-masked]>; HTTP working group mailing list <[email-masked]>
Subject: Re: on HTTP/QUIC and HTTPBis

Hey Mike, I really don't think we're that far apart except maybe in your use of the term layers. quic is being effectively componentized but not into layers that create full abstractions in the way a traditional stack works. So while transport and crypto are definitely blended (you cannot implement the transport without doing both transport'y things and crypto things and then transporty things again) I wouldn't call them layers. I'm trying out the term blended functions :) ietf quic has effectively been able to use the services of TLS 1.3 without modification - and thank goodness as we really don't need another security protocol - but the layering of cleartext->ciphertext->packets no longer exists (retransmits being an easy example). But these components definitely have logical interfaces and can be expressed in code in many many different ways which I hope makes them suitable to a great deal of diverse environments. This seems pretty close to your pluggable component vision and I support that as at least a model for describing what's going on.
That's all really an interesting side-discussion distracting from the original question (roughly: what feature motivates this thing?) I'm curious, do you have different answers?

On Fri, Mar 10, 2017 at 7:16 PM, Mike Bishop <[email-masked]<mailto:[email-masked]>> wrote:
While Patrick approaches it from a standpoint of how we can squeeze out extra performance by intertwining the layers, I see them as different stand-alone system binaries owned by different teams, and as things we will almost certainly want to modularize for different uses.  And there's value in diversity of viewpoints, since Patrick and I are in very different implementation situations despite both sitting on the browser side.

-----Original Message-----
From: Poul-Henning Kamp [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Friday, March 10, 2017 12:22 PM
To: Patrick McManus <[email-masked]<mailto:[email-masked]>>
Cc: Adrien de Croy <[email-masked]<mailto:[email-masked]>>; Mike Bishop <[email-masked]<mailto:[email-masked]>>; HTTP working group mailing list <[email-masked]<mailto:[email-masked]>>
Subject: Re: on HTTP/QUIC and HTTPBis

--------
In message <[email-masked]<mailto:[email-masked]>>
, Patrick McManus writes:

Poul-Henning

--
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]<mailto:[email-masked]>         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

---

From: Stefan Eissing <[email-masked]>
Date: Tue, 14 Mar 2017 09:25:47 +0100

+1 to your views on this. 

> Am 13.03.2017 um 23:52 schrieb Mike Bishop <[email-masked]>:
> 
> I’d be thrilled if we even get parity with TFO+TLS 1.3+HTTP/2 at first. 

I will work hard to keep that an ever improving target by itself. From working group pov this h2 thing is done. From implementation side, there is lots to do and optimize.

I do follow the quic efforts to have a good picture of where our implementation needs to go in the future. I see that basic h2 concepts still apply. I am not worried by changing frame numbers etc. 

Things like limiting SETTINGS to the start of a session make me a bit nervous. (maybe i got that wrong) If extension mechanism from h2 would disappear in hq, it could make things unnecessarily difficult.

-stefan

---

From: Patrick McManus <[email-masked]>
Date: Tue, 14 Mar 2017 11:48:01 -0400

hi Adrien - I wanted to address your good comment about source address
validation.

On Sun, Mar 12, 2017 at 5:28 PM, Adrien de Croy <[email-masked]> wrote:

> didn't need to maintain state about a connection until after the ACK to the
> SYN ACK was received, thereby precluding spoofing source IP.
>
> need to record and retain client state from the initial packet which will
> open it up to a SYN flood style of resource consumption attack?
>
>
QUIC worries a lot about DoS and amplification attacks including what you
are describing (and more!). These are certainly practical problems.

A server worried about growing amounts of outstanding handshake state can
choose to insert a round trip in order to accomplish source address
validation. Basically proof of receipt is done with a TLS extension and a
Client Hello can be stretched out for another RTT to include this "source
address validation token" via the TLS 1.3 Hello Retry Request mechanism.
0-RTT connections would always include the validation token, but if this
mitigation were being applied it would turn 1-RTT connects back into the
TCP/TLS style 2-RTT.

https://quicwg.github.io/base-drafts/draft-ietf-quic-
transport.html#client-address-validation-procedure
https://quicwg.github.io/base-drafts/draft-ietf-quic-tls.
html#source-address-validation
https://tlswg.github.io/tls13-spec/#rfc.section.4.1.4

The token can be anything the server likes (the client just has to echo it)
as long as it is unguessable. There are obvious ways to do that which don't
require state per conn.

the obvious question is can't we just fix TCP so that lots of connections
> isn't so bad?

>From time to time QUIC is described as fixing TCP and being a general
successor. Eye of the beholder I guess.
https://twitter.com/ietfmemes/status/757866970939330560

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 15 Mar 2017 07:36:09 +0100

Hi Patrick,

> as long as it is unguessable. There are obvious ways to do that which don't

In fact that's an area where I think we have an opportunity to fix this
TCP flood problem. The current remediations for TCP are limited. Against
SYN floods we have SYN cookies which are not perfect since they're limited
by the client's capabilities and are not *that* secure (but often enough
to limit the impact of massive attacks).

We also face the risk that a sequence number is guessed to establish a
connection from spoofing. That's rare and difficult but much easier once
the server starts to emit SYN cookies. And such an abused TCP connection
can be used to flood the server with many pipelined HTTP requests. Given
that most servers OSes have large outgoing TCP windows, the server can
push a lot of responses before the outgoing buffer is full and the
processing stalls. This can be used to abuse CPU/resources and/or as an
amplification attack (but good luck for making use of this).

Here by redesigning the transport layer we do have opportunities for
improving the situation. The risk is to overlook some robust points of
TCP that are considered as granted and which protect it. But if TCP is
mostly imitated with larger tokens instead of just SYN cookies, there
is an opportunity to keep its robustness by design and improve it by
limiting the risk of guessing some sensitive numbers.

The other risk I'm seeing is having the server perform too expensive
validations on incoming datagrams. Sometimes using multiple algorithms
at once can considerably help. Eg a token is made by the concatenation
of XXH32(udp endpoints) and its SHA256. You only validate the SHA256
part when the XXH32 is fine (it's much cheaper). This way you don't
have to perform expensive operations to deal with massive attacks.

Cheers,
Willy