Subject: [QUIC] Requirements

From: Michael Tuexen <[email-masked]>
Date: Tue, 24 May 2016 20:27:50 +0200

Dear all,

in the last years Google has done a wonderful job of experimenting with transport
protocol features to improve the overall performance of HTTP-based traffic.

In the Bar BOF at the IETF 93 in Prague they presented some of their results
showing that several features are necessary to get the performance benefits:
* 0 RTT connections setup
* Cross layer optimisations with TLS
* Improved error recovery
* Multiple streams/flows to avoid head of line problems

There are also some requirements for the deployment:
* It MUST be implementable as part of an application and usable by a user
  without any specific privileges
* It MUST pass through a large part of middle boxes.

Obviously, these benefits can be realised by using QUIC.

But here is the question:

Are there any plans to analyse the requirements in a protocol independent
way and try to figure out if these requirements can also be met by evolving
one or more of the existing transport protocols (in combination with the appropriate
(D)TLS)? And then making a decision whether it is appropriate to
evolve existing protocols in addition to or instead of developing a new one...

Best regards
Michael

---

From: Randall Stewart <[email-masked]>
Date: Tue, 24 May 2016 14:46:28 -0400

Michael:

I could not agree more. If this is going to be a WG to explore new
transport and/or transport features it should go through those steps
to see if an existing transport can’t be extended or used.

I think even TCP could be extend to cover most of this except maybe FEC (that
might be hard). I believe ekr had some proposals for both TCP and SCTP for
“fast start” which I believe was 0 RTT connection setup. Past things like
Minion have done multi-streaming in TCP.. and I think the list goes on :-)

It would be well worth exploring existing transports and their “mold-ability” to
do what is required.

Of course if the intent is just to document Google’s Quic then such requirements
workt is not needed and I would suppose neither is a WG..

R

On May 24, 2016, at 2:27 PM, Michael Tuexen <[email-masked]> wrote:

> _______________________________________________
> QUIC mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/quic

--------
Randall Stewart
[email-masked]
803-317-4952

---

From: Ted Hardie <[email-masked]>
Date: Wed, 25 May 2016 14:29:14 -0700

> evolve existing protocols in addition to or instead of developing a new
Being a yes-and kind of guy, I have to say I don't  see why evolving
existing protocols and developing news should be seen as in opposition.
The two can proceed in parallel and learn from each other (as QUIC has
certainly learned a lot on the congestion control front).

To put this another way, I think it is good for the Internet if new
protocols may be deployed; ossification is bad.  We can learn from the
deployments in a variety of ways.

Just my thinking,

regards,

Ted

---

From: Salvatore Loreto <[email-masked]>
Date: Thu, 26 May 2016 07:50:45 +0000

I agree with Ted
I do not see moving existing protocols over UDP and the development of new protocols natively designed to run over UDP like two things in opposition
Actually I see as an opportunity to learn more and do a better job on both the sides

/Sal

From: QUIC [mailto:[email-masked]] On Behalf Of Ted Hardie
Sent: den 25 maj 2016 23:29
To: Michael Tuexen <[email-masked]>
Cc: [email-masked]
Subject: Re: [QUIC] Requirements

On Tue, May 24, 2016 at 11:27 AM, Michael Tuexen <[email-masked]<mailto:[email-masked]>> wrote:

Being a yes-and kind of guy, I have to say I don't  see why evolving existing protocols and developing news should be seen as in opposition.  The two can proceed in parallel and learn from each other (as QUIC has certainly learned a lot on the congestion control front).
To put this another way, I think it is good for the Internet if new protocols may be deployed; ossification is bad.  We can learn from the deployments in a variety of ways.
Just my thinking,
regards,
Ted

---

From: Jana Iyengar <[email-masked]>
Date: Fri, 27 May 2016 19:30:18 -0700

Hi Michael,

> evolve existing protocols in addition to or instead of developing a new

I agree with you that the requirements of QUIC should be requirements for
new protocols and for evolution of existing ones. The lessons of QUIC are
expected to inform other transport work -- QUIC has provided inputs to TCP
and TLS about the importance of avoiding handshake RTTs, and more recently,
RACK (draft-cheng-tcpm-rack-00) is similar in spirit to QUIC's loss
recovery.

Subsequent protocol work must learn QUIC's lessons, just as QUIC learned
from its predecessors. While there's inevitably functional overlap between
QUIC and any of {TCP, SCTP, MPTCP} (which have overlaps of their own), much
of the work of QUIC has been in refining and tightly integrating mechanisms
that we have learned from previous transports into a modern transport --
protection of data and headers, low latency-overhead, congestion-control
agility, and strong loss-recovery, to name a few.

So, I agree with Ted and Sal's sentiments. Fixing up our existing designs
for existing deployments is definitely very valuable and hopefully
inevitable, but it does not take away from the fact that QUIC is a clean
protocol design that incorporates, as first principles, lessons learned
from our collective past. Going forward, I expect there to be debates about
which protocol is the best fit for an application, and I encourage that --
having real choice in transport protocols at design time and at run time is
a wonderful thing, and it's long overdue.

- jana

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 28 May 2016 10:17:28 +0200

> On 28 May 2016, at 04:30, Jana Iyengar <[email-masked]> wrote:
> 
> Hi Michael, 
> 
> I agree with you that the requirements of QUIC should be requirements for new protocols and for evolution of existing ones. The lessons of QUIC are expected to inform other transport work -- QUIC has provided inputs to TCP and TLS about the importance of avoiding handshake RTTs, and more recently, RACK (draft-cheng-tcpm-rack-00) is similar in spirit to QUIC's loss recovery. 
> 
> Subsequent protocol work must learn QUIC's lessons, just as QUIC learned from its predecessors. While there's inevitably functional overlap between QUIC and any of {TCP, SCTP, MPTCP} (which have overlaps of their own), much of the work of QUIC has been in refining and tightly integrating mechanisms that we have learned from previous transports into a modern transport -- protection of data and headers, low latency-overhead, congestion-control agility, and strong loss-recovery, to name a few. 
Hi Jana,

so these are some of the requirements, right? I'm wondering if it makes sense to provide such a list
of findings in a protocol independent way. What is important and what is not needs knowledge of the
results of the experiments Google where running.
Without such a description, it is hard for other groups to figure out if it is important to consider
integrating a specific feature of QUIC.

Best regards
Michael
> 
> So, I agree with Ted and Sal's sentiments. Fixing up our existing designs for existing deployments is definitely very valuable and hopefully inevitable, but it does not take away from the fact that QUIC is a clean protocol design that incorporates, as first principles, lessons learned from our collective past. Going forward, I expect there to be debates about which protocol is the best fit for an application, and I encourage that -- having real choice in transport protocols at design time and at run time is a wonderful thing, and it's long overdue.
> 
> - jana

---

From: Jim Roskind <[email-masked]>
Date: Sat, 28 May 2016 09:15:37 -0700

re: "...results of experiments [done by] Google."

Publication of experiments done in/around QUIC should be a treasure trove
for both QUIC and numerous other protocols.

Although not all-encompasing, I presented several of Google's experimental
results to IETF-88 back in 2013.  Be sure to take a look at:

https://www.ietf.org/proceedings/88/slides/slides-88-tsvarea-10.pdf

Included in that IETF presentation (which I think was video recorded) were
slides showing:

a) NAT Eviction probabilities vs time, for UDP bindings (slide 10)
b) Round-trip packet-loss probability for trains of paced vs unpaced
packets of various sizes (slides 22 and 23)
c) Generic 0 vs 1 vs 2 FEC utility for paced and unpaced trains (slide 24)

All such experiments (and many more) need to be repeated and published over
time, as the deployment of middle boxes, links (mobile variants vs
non-mobile), routers, contemporaneous traffic patterns, etc. continue to
evolve.

Item (a) showed the viability of a UDP based multi-packet protocol, with
(then current) caveats that pauses in the stream beyond 30 seconds increase
eviction rates. This drove QUIC to support handling of that progressive
increase in eviction. There is some hope that upcoming IETF standardization
in this area will further improve the NAT-eviction landscape, but that will
certainly depend on vendors and deployment (market penetration) rates.

Item (b) showed the tremendous impact (value) of pacing, and also showed
the remarkable vulnerability (i.e., increased loss probability) of the 2nd
packet in a cold-connection. That surprising (previously unpublished?)
2nd-packet loss statistic was recently observed and verified to me verbally
by a second vendor for a related UDP protocol just a month ago (May 2016)
at a public presentation I gave.

Item (c) better showed the nature of the packet loss correlation (and
showed that loss should *not* be modeled by an independent identically
distributed process.). It clearly showed the lack of marginal value of
multiple-packet error recovery (i.e., 2-packet-loss FEC) for trains as long
as 21 packets (which is a plausible congestion window for a lossy
connection). The data also justified the significant utility of
1-packet-loss FEC for such short windows (i.e., congestion window under 21)
with very latency sensitive transmissions (such as stream creation for
small latency sensitive requests, including perchance advertisements).

We deduced in 2011 that about 5-7% of all chrome clients had UDP blocked
(from reaching Google) for even small packets, which is what drove the
decision to have a TCP fallback plan in QUIC.  I believe that experimental
deployment results for QUIC continued to support that 5-7% statistic,
though I'm very hopeful that system administrators will change this
blocking over time (for the better!).  I also look forward to ongoing
publication of this statistic by Google.

Another Google published result is the fact that UDP payload sizes over
1350 bytes tend to be blocked more often, which was indeed the driving
factor behind QUIC (initially) adopting a static payload size of 1350
bytes.

I expect that experiments done by multiple vendors will be published.  I
know that Firefox adopted Chrome's metrics gathering approach, and
hopefully if/when an experimental QUIC implementation appears there, then
additional results will continue to be published.

On Sat, May 28, 2016 at 1:17 AM, Michael Tuexen <
[email-masked]> wrote:

> > I agree with you that the requirements of QUIC should be requirements
> for new protocols and for evolution of existing ones. The lessons of QUIC
> are expected to inform other transport work -- QUIC has provided inputs to
> TCP and TLS about the importance of avoiding handshake RTTs, and more
> recently, RACK (draft-cheng-tcpm-rack-00) is similar in spirit to QUIC's
> loss recovery.
> >
> > So, I agree with Ted and Sal's sentiments. Fixing up our existing
> designs for existing deployments is definitely very valuable and hopefully

---

From: Jana Iyengar <[email-masked]>
Date: Sat, 28 May 2016 20:05:34 -0700

Hi Michael,

> integrating a specific feature of QUIC.

I don't think of them as necessarily requirements, but more as lessons
learned. For instance, monotonically-increasing sequence numbers make loss
detection and recovery much more efficient, but that's a fundamental design
decision that can't be retrofitted easily. Of course, there are others,
such as the elimination of handshake latency, but that's not a new lesson
for the transport community -- TCP Fast Open was underway in 2009.

That said, point taken. Though I'm not sure exactly what form or shape such
a "lessons" document would take, besides continuing presenting data in
TSVAREA and/or MAPRG. I'm happy to continue chatting about such a doc, but
this forum may not be the right place for it... especially since I don't
think that these lessons are only being learned in/from QUIC deployment.

- jana

Best regards

---

From: Michael Tuexen <[email-masked]>
Date: Sun, 29 May 2016 17:28:08 +0200

> On 28 May 2016, at 18:15, Jim Roskind <[email-masked]> wrote:
> 
> Publication of experiments done in/around QUIC should be a treasure trove for both QUIC and numerous other protocols.
> 
> Although not all-encompasing, I presented several of Google's experimental results to IETF-88 back in 2013.  Be sure to take a look at:
> 
> Included in that IETF presentation (which I think was video recorded) were slides showing:
> 
> b) Round-trip packet-loss probability for trains of paced vs unpaced packets of various sizes (slides 22 and 23)
> All such experiments (and many more) need to be repeated and published over time, as the deployment of middle boxes, links (mobile variants vs non-mobile), routers, contemporaneous traffic patterns, etc. continue to evolve.
> 
> Item (a) showed the viability of a UDP based multi-packet protocol, with (then current) caveats that pauses in the stream beyond 30 seconds increase eviction rates. This drove QUIC to support handling of that progressive increase in eviction. There is some hope that upcoming IETF standardization in this area will further improve the NAT-eviction landscape, but that will certainly depend on vendors and deployment (market penetration) rates.
> 
> Item (b) showed the tremendous impact (value) of pacing, and also showed the remarkable vulnerability (i.e., increased loss probability) of the 2nd packet in a cold-connection. That surprising (previously unpublished?) 2nd-packet loss statistic was recently observed and verified to me verbally by a second vendor for a related UDP protocol just a month ago (May 2016) at a public presentation I gave. 
> 
> Item (c) better showed the nature of the packet loss correlation (and showed that loss should *not* be modeled by an independent identically distributed process.). It clearly showed the lack of marginal value of multiple-packet error recovery (i.e., 2-packet-loss FEC) for trains as long as 21 packets (which is a plausible congestion window for a lossy connection). The data also justified the significant utility of 1-packet-loss FEC for such short windows (i.e., congestion window under 21) with very latency sensitive transmissions (such as stream creation for small latency sensitive requests, including perchance advertisements). 
> 
> We deduced in 2011 that about 5-7% of all chrome clients had UDP blocked (from reaching Google) for even small packets, which is what drove the decision to have a TCP fallback plan in QUIC.  I believe that experimental deployment results for QUIC continued to support that 5-7% statistic, though I'm very hopeful that system administrators will change this blocking over time (for the better!).  I also look forward to ongoing publication of this statistic by Google.
> 
> Another Google published result is the fact that UDP payload sizes over 1350 bytes tend to be blocked more often, which was indeed the driving factor behind QUIC (initially) adopting a static payload size of 1350 bytes. 
> 
> I expect that experiments done by multiple vendors will be published.  I know that Firefox adopted Chrome's metrics gathering approach, and hopefully if/when an experimental QUIC implementation appears there, then additional results will continue to be published.
Hi Jim,

summarising the above means:
1. Legacy NAT traversal (doable via UDP encapsulation)
2. Pacing
3. FEC
4. Path MTU limit
combined with what Jana mentioned:
5. Fast connection setup
6. Improved loss recovery
7. CC agility
8. Encryption of transport layer information

Did I miss anything?

Best regards
Michael
> 
> 
> 
> 
> On Sat, May 28, 2016 at 1:17 AM, Michael Tuexen <[email-masked]> wrote:

---

From: Randall Stewart <[email-masked]>
Date: Mon, 30 May 2016 07:48:24 -0400

Michael:

See below..

>> I expect that experiments done by multiple vendors will be published.  I know that Firefox adopted Chrome's metrics gathering approach, and hopefully if/when an experimental QUIC implementation ap…

Thinking about this list in the broad sense..

> 1. Legacy NAT traversal (doable via UDP encapsulation)

I think with TCP this is not an issue, SCTP would have to be
like Quic and be wrapped in UDP I think that is already occurring
within rc-web’s data channel :-)

> 2. Pacing

I think pacing can be done on any transport and in general is
a *good* thing. The one thing about pacing is that it tends to 
really hurt you in TSO/LRO (at least for TCP). I am currently working
on trying to come up with a method for pacing that does not loose
all the benefits of TSO but still gives us the desired “paced” results.

> 3. FEC

This one I think for TCP would be a bit hard.. SCTP could in theory
do it.. I would be quite curious to know how much benefit is really
seen by this. I thought Ian mentioned to us at a past IETF that it
really helped in certain scenarios but how often does that occur
and what scenarios are they?  If google could share how often
they have this on and how much it helps in those cases that
would be interesting….

> 4. Path MTU limit

I have noticed, in my TCP work, that PMTU discovery does become
painful at times.. and just having a fixed size makes life a lot easier.
Of course with IPSEC and IPv6 and other fun things 1350 bytes may even be
too big :-)

Then again I have heard arguments for keeping Time-Stamps off by default
due to the wasted size in the packet. Which makes me wonder, if we are
so concerned about the time-stamp size should we be concerned about
the extra bytes that we loose in a 1500 byte MTU? 

Its a shame too since time-stamps really make TCP (with PRR and RACK) much
more robust.. We only see 15% or so of connections that have SACK not 
doing Time-Stamps so its not too bad (I think thats due to the fact that most of
our clients are linux based which like FreeBSD and Mac turn time-stamp on
by default). Sack on the other hand I see in about 99.9% of all clients which
is way cool :-)

> combined with what Jana mentioned:

This I think can be done in both TCP and SCTP.. I would hope that
TLS 1.3 will gain that for all protocols that use it :-)

> 6. Improved loss recovery

I have actually been implementing RACK and PRR for TCP and am
beginning to see the same benefits in TCP that we have heard
about with Quic..  so these are very very useful things thats for sure :-)

> 7. CC agility

I can’t speak for windows or MacOS .. but I think that both Linux and FreeBSD
have had this for many years now.. I know when I look in the cc for BSD there
are at least eight congestion control algorithms that are available and I am sure
Linux probably has more :)

> 8. Encryption of transport layer information

That does eliminate snooping middle boxes.. but some environments (such as
mobile) might not like that :-) 

I doubt you could easily do this with TCP or SCTP.. since middle boxes expect
packet formates to look a particular way. Anything new of course does not
have this issue.. possibly wrapping a protocol in UDP would gain you that though :-)

> Did I miss anything?

Is multi-streaming missing from the list? I think it was mentioned in some
of the slides. We currently don’t see any benefit in this, but we are not
loading multiple images for a web-page.. instead streaming a movie..

I would imagine our client teams during search for a movie might like this
but I would imagine they (like many web based things) open multiple connections.

One question I wonder (and this is for both Quic and SCTP) is the ability to do
LRO/TSO. I have found that one of our Caches which typically outputs about
36Gbps drops its speed in 1/2 if I ifconfig off TSO/LRO. When you pull the
things into user space and encrypt it you loose the ability to do these functions.
I don’t think SCTP has ever been able to do TSO/LRO as well..  Thats one big
impact to performance that needs to be thought about..

I would also like to see any new protocol implementable in the kernel as well. Currently
Quic uses a floating point number for time-stamps and that would be pretty heavy weight
for the kernel.. having a compatible integer equivalent would be something I think that is needed.

Best wishes

R

--------
Randall Stewart
[email-masked]
803-317-4952

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 30 May 2016 15:18:58 +0200

> On 30 May 2016, at 13:48, Randall Stewart <[email-masked]> wrote:
> 
> Michael:
> 
> See below..
> 
Ahh, yes. I missed that.
> 36Gbps drops its speed in 1/2 if I ifconfig off TSO/LRO. When you pull the
Is this also true when you do non-hardware assisted TLS? If not, then this
features becomes less important when you consider a combined transport/security
layer as an entity...

Best regards
Michael

---

From: Randall Stewart <[email-masked]>
Date: Mon, 30 May 2016 09:22:41 -0400

On May 30, 2016, at 9:18 AM, Michael Tuexen <[email-masked]> wrote:

> layer as an entity…

We actually do TLS encryption inside the kernel ahead of TSO. It is true
that we generally only send 16k at a time to TSO (the TLS record length) but this
does not seem to cause us to suffer too much.

How this all works can be found at:

https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf

and

https://people.freebsd.org/~rrs/asiabsd_tls_improved.pdf

So though encryption is happening in the kernel we still can use TSO.. and do :-)

We have one particular machine that we get 80Gbps of encrypted traffic out of (our
newest flash cache :-D.. thanks to a whole *lot* of work from the NF OCA team :D)

R

--------
Randall Stewart
[email-masked]
803-317-4952

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 30 May 2016 22:56:13 -0700

Michael, Randall,

In general, I'd agree with some of these, but I don't want to get into the
details here since I am not sure what the point of this exercise is at the
moment.

I think we are in agreement that doing a clean-slate design allows us to
build a transport with lessons we've learned so far as first principles.
This is the opportunity we as a community have right now. I think many of
these can be retrofit into SCTP (or into TCP), but some may have to be
coerced, and some others may either be impossible or require significant
surgery. We would only know by trying, and I encourage that work.

Perhaps a list of features that can be added to TCP can be discussed on
TCPM, and similarly to SCTP on TSVWG. I'd be happy to chime in on those
threads.

- jana

On Mon, May 30, 2016 at 6:22 AM, Randall Stewart <[email-masked]> wrote:

> All such experiments (and many more) need to be repeated and published
> over time, as the deployment of middle boxes, links (mobile variants vs
> factor behind QUIC (initially) adopting a static payload size of 1350 bytes.
>
>

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 31 May 2016 11:38:24 +0200

> On 31 May 2016, at 07:56, Jana Iyengar <[email-masked]> wrote:
> 
> Michael, Randall,
> 
> In general, I'd agree with some of these, but I don't want to get into the details here since I am not sure what the point of this exercise is at the moment.
I'm trying to figure out if one could evolve TCP or SCTP in combination with (D)TLS to get a similar
performing protocol.
> 
> I think we are in agreement that doing a clean-slate design allows us to build a transport with lessons we've learned so far as first principles. This is the opportunity we as a community have right now. I think many of these can be retrofit into SCTP (or into TCP), but some may have to be coerced, and some others may either be impossible or require significant surgery. We would only know by trying, and I encourage that work.
And this is what I'm trying to figure out. How hard is it to evolve either TCP or SCTP. From a deployment point of
view you want to support userland stacks, which requires UDP encapsulation. But that is defined for SCTP and even
could be done for TCP. The other points on the list are protocol features which would need evolving the protocols.
> 
> Perhaps a list of features that can be added to TCP can be discussed on TCPM, and similarly to SCTP on TSVWG. I'd be happy to chime in on those threads.
I'm currently trying to figure out how such a list might look like.

Best regards
Michael

---

From: Randall Stewart <[email-masked]>
Date: Tue, 31 May 2016 11:19:50 -0400

Jana:

So this discussion leads me to a question for this activity (maybe others feel that way too)…

Is this activity to:

(a) document what Google as done and produce an RFC showing here is “Quic"
         <or>
(b) solving a specific problem

If it is (b) it would make sense to figure out what the problem is and how
it could be solved most easily. In this case one could consider evolving TCP (or even
SCTP … but hey I do TCP right now :D). In case of (a) then this is not even something
we need to discuss..

Thanks

R

--------
Randall Stewart
[email-masked]
803-317-4952