Subject: Re: [tsvwg] The List (of application-layer desired features)

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 6 Aug 2013 08:25:52 +0000

Well, part of this is also related to TCPM, but for simplicity I am fine with keeping the discussion on the TSVWG list right now. The community is mostly the same anyway...

Michael

> -----Original Message-----
> From: Roberto Peon [mailto:[email-masked]] 
> Sent: Tuesday, August 06, 2013 10:16 AM
> To: HTTP Working Group; [email-masked]
> Subject: Re: The List (of application-layer desired features)
> 
> Actually sending to the right list for TSVWG...
> 
> -=R
> 
> 
> On Tue, Aug 6, 2013 at 1:14 AM, Roberto Peon <[email-masked]> wrote:
> 
> 
> 	For those of you who missed it, at the HTTPBis/TSVG 
> joint session, a question about what applications want from 
> the transport (I really want to put quotes around that) came up.
> 
> 	Here is a rendition of what was on the note that I 
> jotted down in response to this question, and which I passed 
> to people at the mic.
> 
> 	(Apps-folks want the following) Deployed in 1996:
> 	-----------------------------------------
> 	- Prioritization
> 	- Partial Reliability
> 	- "Shared" congestion between multiple streams
> 	- Security
> 	- No HOL blocking on stream X when loss on stream Y
> 	- Cheap/Fast  channel/connection setup
> 	- Wide, "safe" deployment
> 	- Competes with TCP/HTTP/1.1 (performance-wise)
> 	- Multipath/roaming robustness, i.e. the "driveway" problem
> 
> 
> 	I'll reiterate that by far the most important feature 
> is "is deployed".
> 	Nothing else matters until that is true, at least at 
> the application-layer.
> 	
> 	-=R
> 	
> 
> 
> 
> 
>

---

From: Roberto Peon <[email-masked]>
Date: Tue, 06 Aug 2013 08:15:48 -0000

-=R

On Tue, Aug 6, 2013 at 1:14 AM, Roberto Peon <[email-masked]> wrote:

> For those of you who missed it, at the HTTPBis/TSVG joint session, a
> question about what applications want from the transport (I really want to
> put quotes around that) came up.
>
> Here is a rendition of what was on the note that I jotted down in response
> to this question, and which I passed to people at the mic.
>
> I'll reiterate that by far the most important feature is "is deployed".
> Nothing else matters until that is true, at least at the application-layer.
> -=R
>
>
>

---

From: Roberto Peon <[email-masked]>
Date: Tue, 06 Aug 2013 08:30:42 -0000

The most interesting part of this list, for me, was that I was nearly 100%
sure that every item on this list had been discussed and designed by folks
participating in TSVWG.
The part that I'm unsure that people realized is the extent of the hunger
to have something deployed "yesterday" so that the features could be
reliably used today.

A colleague reminded me that "prioritization" means different things in the
different areas (thanks, and thanks Allison for pointing it out in your
preso!). On this note, it meant both QoS like things and hints to the
application layer.

Something which I also failed to describe well was that "cheap"/"fast"
channel/connection setup to CPU/mem, yes, but perhaps more importantly to
the importance of minimizing latency in connection/channel setup,
especially while including features like security.
-=R

On Tue, Aug 6, 2013 at 1:15 AM, Roberto Peon <[email-masked]> wrote:

>> Here is a rendition of what was on the note that I jotted down in
>> response to this question, and which I passed to people at the mic.
>>

---

From: Mike Belshe <[email-masked]>
Date: Tue, 06 Aug 2013 09:30:42 -0000

On Tue, Aug 6, 2013 at 10:49 AM, Yoav Nir <[email-masked]> wrote:

>  I think most of that is addressed in SCTP.  Except the deployment part.
> Standards people can’t force vendors of operating systems or Linux
> distributions to include any feature. So we have a lot of “version 2”s in
> the IETF that take a very long time to get deployed. ****
>
> ** **
>
> It’s also much more attractive to define a new thing (like SCTP) than to
> make something old work a little better. SCTP was sexier than TCPM.****
>
> ** **
>
> So it took ages to get deployment of IPv6, IKEv2, TLS 1.2, and all three
> are still used far less than IPv4, IKEv1 and TLS 1.0. SCTP is used almost
> never. HTTP/2 will likely fare better because the vendors are more involved
> and committed, but it’s hard to make predictions, especially about the
> future.
>

You're right, SCTP is non-deployable, which makes it a non-starter.  SCTP
also does not address handshake issues or TLS issues.

I don't mean to sound inflammatory - but for all intents and purposes, the
next generation transport will need to be in user space and run on top of
UDP.  There simply is no other deployable option on the table.  QUIC is
already reasonably far at exploring these issues:
http://en.wikipedia.org/wiki/QUIC

Mike

> ****
>
> ** **
>
> Yoav****
>
> ** **
>
> *From:* Roberto Peon [mailto:[email-masked]]
> *Sent:* Tuesday, August 06, 2013 11:16 AM
>
> *To:* HTTP Working Group; [email-masked]
> *Subject:* Re: The List (of application-layer desired features)****
>
> ** **
>
> Actually sending to the right list for TSVWG...****
>
> ** **
>
> -=R****
>
> ** **
>
> On Tue, Aug 6, 2013 at 1:14 AM, Roberto Peon <[email-masked]> wrote:****
>
> put quotes around that) came up.****
>
> ** **
>
> to this question, and which I passed to people at the mic.****
>
> ** **
>
> (Apps-folks want the following) Deployed in 1996:****
>
> -----------------------------------------****
>
> - Prioritization****
>
> - Partial Reliability****
>
> - "Shared" congestion between multiple streams****
>
> - Security****
>
> - No HOL blocking on stream X when loss on stream Y****
>
> - Cheap/Fast  channel/connection setup****
>
> - Wide, "safe" deployment****
>
> - Competes with TCP/HTTP/1.1 (performance-wise)****
>
> - Multipath/roaming robustness, i.e. the "driveway" problem****
>
> ** **
>
> ** **
>
> I'll reiterate that by far the most important feature is "is deployed".***
> *
>
> ****
> -=R****
> ** **
> ** **
> ** **
> Email secured by Check Point ****
>

---

From: William Chan (陈智昌) <[email-masked]>
Date: Tue, 06 Aug 2013 10:48:26 -0000

Another thing that I think Roberto implied but didn't explicitly call out
is mitigating application incentives to open multiple connections

On Tue, Aug 6, 2013 at 1:30 AM, Roberto Peon <[email-masked]> wrote:

> A colleague reminded me that "prioritization" means different things in
> the different areas (thanks, and thanks Allison for pointing it out in your

Prioritization also includes stuff sitting in socket buffers.

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 27 Aug 2013 11:47:56 +0200

On Aug 6, 2013, at 11:30 AM, Mike Belshe <[email-masked]> wrote:

> I think most of that is addressed in SCTP.  Except the deployment part. Standards people can’t force vendors of operating systems or Linux distributions to include any feature. So we have a lot of “version 2”s in the IETF that take a very long time to get deployed.
> 
>  
> 
> It’s also much more attractive to define a new thing (like SCTP) than to make something old work a little better. SCTP was sexier than TCPM.
> 
>  
> 
> So it took ages to get deployment of IPv6, IKEv2, TLS 1.2, and all three are still used far less than IPv4, IKEv1 and TLS 1.0. SCTP is used almost never. HTTP/2 will likely fare better because the vendors are more involved and committed, but it’s hard to make predictions, especially about the future.
> 
> 
> You're right, SCTP is non-deployable, which makes it a non-starter.  SCTP also does not address handshake issues or TLS issues.
I agree that SCTP over IP can't be deployed now due to missing NAT support.
But why can't SCTP/UDP/IP (RFC 6951) be deployed now? Also SCTP as specified now
is not optimised for fast setup. It needs an RTT before it can send the first
user message. However, I think one can easily extend it to provide a fast association
setup...
> 
> I don't mean to sound inflammatory - but for all intents and purposes, the next generation transport will need to be in user space and run on top of UDP.  There simply is no other deployable option on the table.  QUIC is already reasonably far at 
And why does "be in user space and run on top of UDP" exclude SCTP? There is also a
userland implementation supporting it. It also supports SCTP/DTLS/UDP and is
included in Firefox (for RTCWeb support).

Best regards
Michael
> exploring these issues: http://en.wikipedia.org/wiki/QUIC
> 
> Mike
> 
> 
> 
> 
>  
> 
>  
> 
> Yoav
> 
>  
> 
> Sent: Tuesday, August 06, 2013 11:16 AM
> 
> 
> For those of you who missed it, at the HTTPBis/TSVG joint session, a question about what applications want from the transport (I really want to put quotes around that) came up.
> 
>  
> 
> Here is a rendition of what was on the note that I jotted down in response to this question, and which I passed to people at the mic.
> 
>  
> 
> Email secured by Check Point
> 
>

---

From: Michael Thornburgh <[email-masked]>
Date: Tue, 27 Aug 2013 14:46:01 -0700

if you're bringing up something like QUIC, i'll point out that RTMFP addresses almost every point on The List, and is currently widely deployed in the Internet (in Flash Player) with 5+ years of deployment experience.  RTMFP is a general-purpose transport protocol that runs on top of UDP and provides prioritization, parallel message channels ("flows"), partial reliability, shared congestion control, a generalized security framework, independent flow control for every flow/channel, no flow-flow HOL blocking, IP address mobility, and more.  RTMFP sessions start in 2 RTT, and arbitrary numbers of unidirectional flows in that session can be started in 0 RTT.  flows are named with arbitrary/opaque-to-RTMFP metadata instead of port or stream numbers.  "return flow association" generalizes bidirectional communication to arbitrarily complex trees of flows, which can more naturally model structured data transport (such as request/multiple-response).

RTMFP is described here:

  http://tools.ietf.org/html/draft-thornburgh-adobe-rtmfp

and is currently in the RFC Editor's queue for publication as an Informational RFC.

note that while SCTP might not technically have HOL blocking stream-stream, the structure of its acknowledgements and the Transmission Sequence Number semantics can cause a priority inversion during periods of congestion.  SCTP does not have independent flow control for each stream.

some thoughts on "fast channel startup with no state at the server":

  1) you will typically want some kind of handshake with the server to establish that the client is actually at the address it appears to be coming from, to avoid TCP's "SYN flood" problem.  that means at least one RTT when you have no current state at the server for a client.

  2) when designing RTMFP, we rejected a "connection reset" message that could be sent by a server to a client that thought it had a connection when no connection existed in the server (for example, if the server rebooted or timed out the connection but the client didn't know), as that would be an abusable denial-of-service vector.

  3) if you're willing to maintain some state at a server, you can leave an RTMFP session open for a long time, and use the "address mobility" capability to handle the case when a client changes addresses (for example, on getting a new translation in a NAT after a quiet period).  the method described in Section 3.5.4.2 of the RTMFP spec currently has a 1 RTT cost to verify the address change.  that method could be modified to allow a provisional change of address immediately, with a switch back to the old address if the verification fails.

  4) if you've been idle for long enough to time out a session (minutes?), then establishing a new session (even at a cost of a couple RTTs) shouldn't perceptually be a big deal.

-michael thornburgh

From: [email-masked] [mailto:[email-masked]] On Behalf Of Mike Belshe
Sent: Tuesday, August 06, 2013 2:31 AM
To: Yoav Nir
Cc: HTTP Working Group; [email-masked]
Subject: Re: [tsvwg] The List (of application-layer desired features)

On Tue, Aug 6, 2013 at 10:49 AM, Yoav Nir <[email-masked]<mailto:[email-masked]>> wrote:
I think most of that is addressed in SCTP.  Except the deployment part. Standards people can't force vendors of operating systems or Linux distributions to include any feature. So we have a lot of "version 2"s in the IETF that take a very long time to get deployed.

It's also much more attractive to define a new thing (like SCTP) than to make something old work a little better. SCTP was sexier than TCPM.

So it took ages to get deployment of IPv6, IKEv2, TLS 1.2, and all three are still used far less than IPv4, IKEv1 and TLS 1.0. SCTP is used almost never. HTTP/2 will likely fare better because the vendors are more involved and committed, but it's hard to make predictions, especially about the future.

I don't mean to sound inflammatory - but for all intents and purposes, the next generation transport will need to be in user space and run on top of UDP.  There simply is no other deployable option on the table.  QUIC is already reasonably far at exploring these issues: http://en.wikipedia.org/wiki/QUIC

Mike

Yoav

From: Roberto Peon [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Tuesday, August 06, 2013 11:16 AM

To: HTTP Working Group; [email-masked]<mailto:[email-masked]>
Subject: Re: The List (of application-layer desired features)

-=R

On Tue, Aug 6, 2013 at 1:14 AM, Roberto Peon <[email-masked]<mailto:[email-masked]>> wrote:
For those of you who missed it, at the HTTPBis/TSVG joint session, a question about what applications want from the transport (I really want to put quotes around that) came up.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 27 Aug 2013 15:22:47 -0700

0-additional-RTTs whenever safe means LOTS of money, actually.
Latency and latency variance are highly correlated with bounce rates, etc.

I should have probably added "free to use" to the list of requirements, but
other than that, if it gets results, it is something which should be
examined and iterated upon.
I am personally quite agnostic to the name, format, company, etc. of who
develops and deploys the next better thing. The important part is that we
keep moving forward.

"Keep moving forward" implies that we're willing to continue to try the
next better thing, which we can do only be lowering the barrier to entry
(in safe ways).

-=R

On Tue, Aug 27, 2013 at 3:12 PM, Mike Belshe <[email-masked]> wrote:

>
>
>
> On Tue, Aug 27, 2013 at 2:46 PM, Michael Thornburgh <[email-masked]>wrote:
>
>> if you’re bringing up something like QUIC, i’ll point out that RTMFP
>> addresses almost every point on The List, and is currently widely deployed
>> in the Internet (in Flash Player) with 5+ years of deployment experience.
>> RTMFP is a general-purpose transport protocol that runs on top of UDP and
>> provides prioritization, parallel message channels (“flows”), partial
>> reliability, shared congestion control, a generalized security framework,
>> independent flow control for every flow/channel, no flow-flow HOL blocking,
>> IP address mobility, and more.  RTMFP sessions start in 2 RTT, and
>> arbitrary numbers of unidirectional flows in that session can be started in
>> 0 RTT.  flows are named with arbitrary/opaque-to-RTMFP metadata instead of
>> port or stream numbers.  “return flow association” generalizes
>> bidirectional communication to arbitrarily complex trees of flows, which
>> can more naturally model structured data transport (such as
>> request/multiple-response).****
>>
>> ** **
>>
>> RTMFP is described here:****
>>
>> ** **
>>
>>   http://tools.ietf.org/html/draft-thornburgh-adobe-rtmfp****
>>
>> ** **
>>
>> and is currently in the RFC Editor’s queue for publication as an
>> Informational RFC.
>>
>
> Sounds cool.  Why don't you benchmark it in a browser?  If it does all
> this stuff, it should be faster today.
>
>
>
>> ****
>>
>> ** **
>>
>> note that while SCTP might not technically have HOL blocking
>> stream-stream, the structure of its acknowledgements and the Transmission
>> Sequence Number semantics can cause a priority inversion during periods of
>> congestion.  SCTP does not have independent flow control for each stream.
>> ****
>>
>> ** **
>>
>> some thoughts on “fast channel startup with no state at the server”:****
>>
>> ** **
>>
>>   1) you will typically want some kind of handshake with the server to
>> establish that the client is actually at the address it appears to be
>> coming from, to avoid TCP’s “SYN flood” problem.  that means at least one
>> RTT when you have no current state at the server for a client.
>>
>
> When at Google, we devised several systems to counteract this.  The
> winning candidate (never implemented) was a hybrid solution:  you let the
> first one in for free (no RTT), and require RTTs for the secondary parallel
> requests until confidence is established.  This gives you most of the perf
> benefit.
>
>
>
>> ****
>>
>> ** **
>>
>>   2) when designing RTMFP, we rejected a “connection reset” message that
>> could be sent by a server to a client that thought it had a connection when
>> no connection existed in the server (for example, if the server rebooted or
>> timed out the connection but the client didn’t know), as that would be an
>> abusable denial-of-service vector.
>>
>
> Not if it ran atop a secured, encrypted channel :-)
>
>
>
>
>
>> ****
>>
>> ** **
>>
>>   3) if you’re willing to maintain some state at a server, you can leave
>> an RTMFP session open for a long time, and use the “address mobility”
>> capability to handle the case when a client changes addresses (for example,
>> on getting a new translation in a NAT after a quiet period).  the method
>> described in Section 3.5.4.2 of the RTMFP spec currently has a 1 RTT cost
>> to verify the address change.  that method could be modified to allow a
>> provisional change of address immediately, with a switch back to the old
>> address if the verification fails.
>>
>
> Maybe, but generally, I think that doesn't scales enough or as well as it
> could....  I was thinking client state, but happy to be proven wrong.
>
>
>> ****
>>
>> ** **
>>
>>   4) if you’ve been idle for long enough to time out a session
>> (minutes?), then establishing a new session (even at a cost of a couple
>> RTTs) shouldn’t perceptually be a big deal.
>>
>
> I disagree :-) RTTs are still 100+ms.  Thats money.
>
> Mike
>
>
>
>
>
>> ****
>>
>> ** **
>>
>> -michael thornburgh****
>>
>> ** **
>>
>> ** **
>>
>> *From:* [email-masked] [mailto:[email-masked]] *On
>> Behalf Of *Mike Belshe
>> *Sent:* Tuesday, August 06, 2013 2:31 AM
>> *To:* Yoav Nir
>> *Cc:* HTTP Working Group; [email-masked]
>> *Subject:* Re: [tsvwg] The List (of application-layer desired features)**
>> **
>>
>> ** **
>>
>> ** **
>>
>> ** **
>>
>> On Tue, Aug 6, 2013 at 10:49 AM, Yoav Nir <[email-masked]> wrote:***
>> *
>>
>> future.****
>> also does not address handshake issues or TLS issues.****
>>
>> ** **
>>
>> I don't mean to sound inflammatory - but for all intents and purposes,
>> the next generation transport will need to be in user space and run on top
>> of UDP.  There simply is no other deployable option on the table.  QUIC is
>> http://en.wikipedia.org/wiki/QUIC****
>>
>> ** **
>>
>> Mike****
>>
>> ** **
>>
>> ** **
>>
>> ** **
>>
>> ** **
>>
>>  ****
>>
>>  ****
>>
>> Yoav****
>>
>>  ****
>>
>> *Sent:* Tuesday, August 06, 2013 11:16 AM****
>>
>>
>> response to this question, and which I passed to people at the mic.****
>>
>>  ****
>>
>> I'll reiterate that by far the most important feature is "is deployed".**
>> **
>>
>> Nothing else matters until that is true, at least at the
>> application-layer.****
>>
>> -=R****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>
>>

---

From: Michael Welzl <[email-masked]>
Date: Wed, 28 Aug 2013 09:58:59 +0200

Hi,

> I agree that SCTP over IP can't be deployed now due to missing NAT support.

Indeed that's not an argument against SCTP/UDP/IP, but I also wonder why, instead of saying "can't be deployed", people don't just go ahead and use it whenever it's there and works, with a fall-back to TCP? This could be done with (this version of) Happy Eyeballs:
http://tools.ietf.org/html/draft-wing-tsvwg-happy-eyeballs-sctp-02

Good reasons against doing this are... what? Anyone?

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Wed, 28 Aug 2013 13:53:29 +0200

On 28. aug. 2013, at 11:53, William Chan (陈智昌) wrote:

> On Aug 28, 2013 4:01 PM, "Michael Welzl" <[email-masked]> wrote:
> >
> > Hi,
> >
> Implementation usefulness. Why bother adding code that barely gets used (and that is unlikely to improve in the near future), adds complexity, code bloat, etc...?
> 
Fair point. That's why I think the OS should in fact do Happy Eyeballs for you!

> SCTP/UDP has a much higher likelihood of usefulness. But as Roberto has mentioned, it still has deficiencies, mostly around RTTs (connection + DTLS setup). If they can be fixed, great. Let's do it.
> 
Why shouldn't it be possible to fix SCTP to do whatever you want? Anyway it sounds to me like a simpler approach than building a whole new protocol. Of course, SCTP++ isn't the nicest acronym...  then again, RTMFP isn't either, if you ask me, sounds almost like RTFM...  QUIC is great though!

Cheers,
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 28 Aug 2013 15:41:38 +0200

On Aug 28, 2013, at 3:34 PM, William Chan (陈智昌) <[email-masked]> wrote:

> On Wed, Aug 28, 2013 at 7:53 PM, Michael Welzl <[email-masked]> wrote:
> 
> I'm not sure if you're trolling me. In case you aren't, you may want to look at the graph at: http://gs.statcounter.com/#os-ww-monthly-201207-201307. Windows XP (released in 2001) is still around 20% of browser usage. If you have the ability to get Microsoft to backport SCTP/IP onto their XP stack, I'd love to know. We're not going to ignore large segments of our user base when we could use UDP and deploy for all relevant OSes. That may be acceptable for some applications, but not for the browser I work on.
You can build SCTP in your browser and run it on top of UDP. This is what is done
in RTCWeb. They use SCTP over DTLS over UDP with SCTP and DTLS running in the
application layer. This is available in recent versions of Firefox and running
on Windows XP...

Best regards
Michael
> 
> This is why Roberto said:
> """
> Wide, "safe" deployment
> """
> I have no attachments to the protocol name or frame format or whatever. Look at what we're doing in HTTP/2 which was inspired by SPDY but now has undergone substantial changes. We're serious about this. As long as the transport provides all the features we need, we'll use it. This conversation got started because tsvwg asked httpbis what the application layer wants from the transport. We're telling you. I think the constructive next step is for tsvwg folks to ask for clarification on any requirement they don't understand, discuss whether or not the requirements are reasonable, and discuss what may need to be done to address them.
>  
> 
> Cheers,
> Michael
> 
>

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 28 Aug 2013 16:31:19 +0200

On Aug 28, 2013, at 3:44 PM, William Chan (陈智昌) <[email-masked]> wrote:

> On Wed, Aug 28, 2013 at 9:41 PM, Michael Tuexen <[email-masked]> wrote:
> You may have missed the fact that Michael Welzl was talking about SCTP/IP and I was responding to that. You may also have missed my note below about SCTP over UDP where I said: "SCTP/UDP has a much higher likelihood of usefulness."
Ahh, OK. Thanks for the clarification.

Best regards
Michael
> Cheers.

---

From: Brian Trammell <[email-masked]>
Date: Wed, 28 Aug 2013 17:34:27 +0200

hi Michael, all,

> Fair point. That's why I think the OS should in fact do Happy Eyeballs for you!

My only concern here -- and I say this as someone who's a big fan of the general approach of endpoints being smart in how they work around deficiency in the network -- is that SCTP and TCP provide different services. This is unlike the case with IPv4 and IPv6, the differences between which are completely hidden behind the API (in the common case, in the modern world, anyway). 

So even if the OS handles the fallback (which it should), the application still has to determine whether the SCTP features it wants to use are available, unless the OS also provides SOCK_SEQPACKET (or equivalent) service with partial reliability and/or multihoming and/or streams over TCP.

> Why shouldn't it be possible to fix SCTP to do whatever you want? Anyway it sounds to me like a simpler approach than building a whole new protocol.

This, absolutely. But in order for it to take off, we need to make it as simple as possible for application developers to adapt to the availability different transport capabilities and services, which is more an API than a protocol issue.

Cheers,

Brian

---

From: Scott Brim <[email-masked]>
Date: Wed, 28 Aug 2013 11:52:37 -0400

On Wed, Aug 28, 2013 at 11:34 AM, Brian Trammell <[email-masked]>wrote:

> My only concern here -- and I say this as someone who's a big fan of the
> general approach of endpoints being smart in how they work around
> deficiency in the network -- is that SCTP and TCP provide different
> services. ...
>
> So even if the OS handles the fallback (which it should), the application
> still has to determine whether the SCTP features it wants to use are
> available
>

Brian, this has been the case since (at least) IBM access method assembler
macros.  Somehow the apps that want to do adjust, if you give them the
hooks to do so.

We don't need to make it simple for everyone, just simple for those who
want to use it.

Scott

---

From: Michael Thornburgh <[email-masked]>
Date: Wed, 28 Aug 2013 10:23:31 -0700

in case there's confusion, RTMFP is free to use:

  https://datatracker.ietf.org/ipr/1942/ (section VI)

(disclaimer: i'm not a lawyer! but that's our intention)

i agree that 0 additional RTTs when safe is highly valuable. i'm apparently more conservative in what "safe" means than several posters on this thread.  :)

-michael thornburgh

From: Roberto Peon [mailto:[email-masked]]
Sent: Tuesday, August 27, 2013 3:23 PM
To: Mike Belshe
Cc: Michael Thornburgh; Yoav Nir; HTTP Working Group; [email-masked]
Subject: Re: [tsvwg] The List (of application-layer desired features)

I should have probably added "free to use" to the list of requirements, but other than that, if it gets results, it is something which should be examined and iterated upon.
I am personally quite agnostic to the name, format, company, etc. of who develops and deploys the next better thing. The important part is that we keep moving forward.

"Keep moving forward" implies that we're willing to continue to try the next better thing, which we can do only be lowering the barrier to entry (in safe ways).

-=R

On Tue, Aug 27, 2013 at 3:12 PM, Mike Belshe <[email-masked]<mailto:[email-masked]>> wrote:

On Tue, Aug 27, 2013 at 2:46 PM, Michael Thornburgh <[email-masked]<mailto:[email-masked]>> wrote:

RTMFP is described here:

Sounds cool.  Why don't you benchmark it in a browser?  If it does all this stuff, it should be faster today.

When at Google, we devised several systems to counteract this.  The winning candidate (never implemented) was a hybrid solution:  you let the first one in for free (no RTT), and require RTTs for the secondary parallel requests until confidence is established.  This gives you most of the perf benefit.

Maybe, but generally, I think that doesn't scales enough or as well as it could....  I was thinking client state, but happy to be proven wrong.

Mike

-michael thornburgh

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Mike Belshe
Sent: Tuesday, August 06, 2013 2:31 AM
To: Yoav Nir
Cc: HTTP Working Group; [email-masked]<mailto:[email-masked]>
Subject: Re: [tsvwg] The List (of application-layer desired features)

Mike

Yoav

-=R

- Prioritization

- Partial Reliability

- Security

-=R

---

From: Michael Thornburgh <[email-masked]>
Date: Wed, 28 Aug 2013 10:41:44 -0700

replies inline.

> From: Mike Belshe [mailto:[email-masked]]
> Sent: Tuesday, August 27, 2013 3:12 PM
> 
> 
> On Tue, Aug 27, 2013 at 2:46 PM, Michael Thornburgh <[email-masked]> wrote:
> 
> 
> 	if you're bringing up something like QUIC, i'll point out that RTMFP addresses almost every
> point on The List, and is currently widely deployed in the Internet (in Flash Player) with 5+ years of
> deployment experience.  RTMFP is a general-purpose transport protocol that runs on top of UDP and
> provides prioritization, parallel message channels ("flows"), partial reliability, shared congestion
> control, a generalized security framework, independent flow control for every flow/channel, no flow-
> flow HOL blocking, IP address mobility, and more.  RTMFP sessions start in 2 RTT, and arbitrary
> numbers of unidirectional flows in that session can be started in 0 RTT.  flows are named with
> arbitrary/opaque-to-RTMFP metadata instead of port or stream numbers.  "return flow association"
> generalizes bidirectional communication to arbitrarily complex trees of flows, which can more
> naturally model structured data transport (such as request/multiple-response).
> 
> 
> 
> 	RTMFP is described here:
> 
> 
> 

i'll add that to the list.
 
> 	note that while SCTP might not technically have HOL blocking stream-stream, the structure of its
> acknowledgements and the Transmission Sequence Number semantics can cause a priority inversion during
> periods of congestion.  SCTP does not have independent flow control for each stream.
> 
> 
> 
> 	  1) you will typically want some kind of handshake with the server to establish that the client
> is actually at the address it appears to be coming from, to avoid TCP's "SYN flood" problem.  that
> means at least one RTT when you have no current state at the server for a client.
> 
> 
> When at Google, we devised several systems to counteract this.  The winning candidate (never
> implemented) was a hybrid solution:  you let the first one in for free (no RTT), and require RTTs for
> the secondary parallel requests until confidence is established.  This gives you most of the perf
> benefit.

i don't understand how that approach addresses the "a bad person initiates 100M one-packet connection attempts from different [forged/pwned] addresses and doesn't follow through" SYN flood problem, where every packet coming in would be "the first one", thereby creating connection state in the server.

> 	  2) when designing RTMFP, we rejected a "connection reset" message that could be sent by a
> server to a client that thought it had a connection when no connection existed in the server (for
> example, if the server rebooted or timed out the connection but the client didn't know), as that would
> be an abusable denial-of-service vector.
> 
> 

the issue here is there is no secured, encrypted channel over which to send a connection reset message, from a server with no connection state to a client with a connection state.  if you were going to go to the PK/CPU trouble to compute new session keys at the server just to send a reset, you might as well not reset and just continue with the session.  RTMFP does have an in-session "abrupt close" mechanism, and flows in a session can be rejected at any time by the receiver.

> 	  3) if you're willing to maintain some state at a server, you can leave an RTMFP session open
> for a long time, and use the "address mobility" capability to handle the case when a client changes
> addresses (for example, on getting a new translation in a NAT after a quiet period).  the method
> described in Section 3.5.4.2 of the RTMFP spec currently has a 1 RTT cost to verify the address
> change.  that method could be modified to allow a provisional change of address immediately, with a
> switch back to the old address if the verification fails.
> 
> 
> Maybe, but generally, I think that doesn't scales enough or as well as it could....  I was thinking
> client state, but happy to be proven wrong.
> 
> 
> 	  4) if you've been idle for long enough to time out a session (minutes?), then establishing a
> new session (even at a cost of a couple RTTs) shouldn't perceptually be a big deal.
> 
> 

there's no reason a timeout couldn't be tuned (per site? and communicated to the client to behave accordingly) to balance resource use at the servers for idle connections with expected users' idle periods between accesses to the site.

-michael thornburgh

> 	I think most of that is addressed in SCTP.  Except the deployment part. Standards people can't
> force vendors of operating systems or Linux distributions to include any feature. So we have a lot of
> "version 2"s in the IETF that take a very long time to get deployed.
> 
> 
> 
> 	It's also much more attractive to define a new thing (like SCTP) than to make something old work
> a little better. SCTP was sexier than TCPM.
> 
> 
> 
> 	So it took ages to get deployment of IPv6, IKEv2, TLS 1.2, and all three are still used far less
> than IPv4, IKEv1 and TLS 1.0. SCTP is used almost never. HTTP/2 will likely fare better because the
> vendors are more involved and committed, but it's hard to make predictions, especially about the
> future.
> 
> 
> 
> 	I don't mean to sound inflammatory - but for all intents and purposes, the next generation
> transport will need to be in user space and run on top of UDP.  There simply is no other deployable
> option on the table.  QUIC is already reasonably far at exploring these issues:
> 		For those of you who missed it, at the HTTPBis/TSVG joint session, a question about what
> applications want from the transport (I really want to put quotes around that) came up.
> 
> 
> 
> 		Here is a rendition of what was on the note that I jotted down in response to this
> question, and which I passed to people at the mic.
> 
> 
>

---

From: Ted Hardie <[email-masked]>
Date: Tue, 27 Aug 2013 15:08:45 -0700

On Tue, Aug 27, 2013 at 2:46 PM, Michael Thornburgh <[email-masked]>wrote:

> Informational RFC.****

Those reviewing that document may also wish to review:

https://datatracker.ietf.org/ipr/1942/

regards,

Ted Hardie

---

From: William Chan (陈智昌) <[email-masked]>
Date: Wed, 28 Aug 2013 17:53:54 +0800

On Aug 28, 2013 4:01 PM, "Michael Welzl" <[email-masked]> wrote:
>>> You're right, SCTP is non-deployable, which makes it a non-starter.
 SCTP also does not address handshake issues or TLS issues.
> Indeed that's not an argument against SCTP/UDP/IP, but I also wonder why,
instead of saying "can't be deployed", people don't just go ahead and use
it whenever it's there and works, with a fall-back to TCP? This could be
> Good reasons against doing this are... what? Anyone?

Implementation usefulness. Why bother adding code that barely gets used
(and that is unlikely to improve in the near future), adds complexity, code
bloat, etc...?

SCTP/UDP has a much higher likelihood of usefulness. But as Roberto has
mentioned, it still has deficiencies, mostly around RTTs (connection + DTLS
setup). If they can be fixed, great. Let's do it.

---

From: William Chan (陈智昌) <[email-masked]>
Date: Wed, 28 Aug 2013 21:34:53 +0800

On Wed, Aug 28, 2013 at 7:53 PM, Michael Welzl <[email-masked]> wrote:

> > Indeed that's not an argument against SCTP/UDP/IP, but I also wonder
> why, instead of saying "can't be deployed", people don't just go ahead and
> use it whenever it's there and works, with a fall-back to TCP? This could
> be done with (this version of) Happy Eyeballs:
I'm not sure if you're trolling me. In case you aren't, you may want to
look at the graph at: http://gs.statcounter.com/#os-ww-monthly-201207-201307.
Windows XP (released in 2001) is still around 20% of browser usage. If you
have the ability to get Microsoft to backport SCTP/IP onto their XP stack,
I'd love to know. We're not going to ignore large segments of our user base
when we could use UDP and deploy for all relevant OSes. That may be
acceptable for some applications, but not for the browser I work on.

> Why shouldn't it be possible to fix SCTP to do whatever you want? Anyway
> it sounds to me like a simpler approach than building a whole new protocol.
> Of course, SCTP++ isn't the nicest acronym...  then again, RTMFP isn't
> either, if you ask me, sounds almost like RTFM...  QUIC is great though!
>

I have no attachments to the protocol name or frame format or whatever.
Look at what we're doing in HTTP/2 which was inspired by SPDY but now has
undergone substantial changes. We're serious about this. As long as the
transport provides all the features we need, we'll use it. This
conversation got started because tsvwg asked httpbis what the application
layer wants from the transport. We're telling you. I think the constructive
next step is for tsvwg folks to ask for clarification on any requirement
they don't understand, discuss whether or not the requirements are
reasonable, and discuss what may need to be done to address them.

---

From: William Chan (陈智昌) <[email-masked]>
Date: Wed, 28 Aug 2013 21:44:38 +0800

On Wed, Aug 28, 2013 at 9:41 PM, Michael Tuexen <
[email-masked]> wrote:

> look at the graph at:
> http://gs.statcounter.com/#os-ww-monthly-201207-201307. Windows XP
> (released in 2001) is still around 20% of browser usage. If you have the
> ability to get Microsoft to backport SCTP/IP onto their XP stack, I'd love
> to know. We're not going to ignore large segments of our user base when we
> could use UDP and deploy for all relevant OSes. That may be acceptable for
> some applications, but not for the browser I work on.

You may have missed the fact that Michael Welzl was talking about SCTP/IP
and I was responding to that. You may also have missed my note below about
SCTP over UDP where I said: "SCTP/UDP has a much higher likelihood of
usefulness."

Cheers.

---

From: Mike Belshe <[email-masked]>
Date: Tue, 27 Aug 2013 15:12:25 -0700

> and is currently in the RFC Editor’s queue for publication as an

Sounds cool.  Why don't you benchmark it in a browser?  If it does all this
stuff, it should be faster today.

> congestion.  SCTP does not have independent flow control for each stream.*
> ***
>
> ** **
>

When at Google, we devised several systems to counteract this.  The winning
candidate (never implemented) was a hybrid solution:  you let the first one
in for free (no RTT), and require RTTs for the secondary parallel requests
until confidence is established.  This gives you most of the perf benefit.

> ****
>
> ** **
>
>   4) if you’ve been idle for long enough to time out a session (minutes?),
> then establishing a new session (even at a cost of a couple RTTs) shouldn’t
> perceptually be a big deal.
>

Mike

> ****
>
> ** **
>
> -michael thornburgh****
>
> ** **
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Mike Belshe
> *Subject:* Re: [tsvwg] The List (of application-layer desired features)***
> *
>
> ** **
>
> ** **
>
> ** **
>
> On Tue, Aug 6, 2013 at 10:49 AM, Yoav Nir <[email-masked]> wrote:****
>

---

From: Mike Belshe <[email-masked]>
Date: Wed, 28 Aug 2013 11:13:30 -0700

On Wed, Aug 28, 2013 at 10:41 AM, Michael Thornburgh <[email-masked]>wrote:

> stream-stream, the structure of its
> winning candidate (never
> i don't understand how that approach addresses the "a bad person initiates
> 100M one-packet connection attempts from different [forged/pwned] addresses
> and doesn't follow through" SYN flood problem, where every packet coming in
> would be "the first one", thereby creating connection state in the server.
>
> the issue here is there is no secured, encrypted channel over which to
> send a connection reset message, from a server with no connection state to
> a client with a connection state.  if you were going to go to the PK/CPU
> trouble to compute new session keys at the server just to send a reset, you
> might as well not reset and just continue with the session.  RTMFP does
> have an in-session "abrupt close" mechanism, and flows in a session can be
> rejected at any time by the receiver.
>
> there's no reason a timeout couldn't be tuned (per site? and communicated
> to the client to behave accordingly) to balance resource use at the servers
> for idle connections with expected users' idle periods between accesses to
> the site.
>

At this point, I think we're going in circles - this all sounds great.  I
encourage you and others to continue down this path.

At this point we need to move the discussion from theory to execution.  I
hear statements like this one all the time - "there is no reason a timeout
couldn't be.....".  You're right, no reason except the devil of the details
when you put it all together and realize you forgot something :-)  It needs
to be put to practice in real product, benchmarked, documented, tested, and
iterated.

The first person to actually implement, rather than theorize, will be the
one who defines the next protocol.  If that person borrows mostly from SCTP
or RTMFP or QUIC or something else doesn't matter.  Theory is nothing -
execution is everything. :-)

Mike

> >       You're right, SCTP is non-deployable, which makes it a
> non-starter.  SCTP also does not address
> > handshake issues or TLS issues.
> >
> >
> >
> application-layer.

---

From: Dave Taht <[email-masked]>
Date: Wed, 28 Aug 2013 12:32:28 -0700

On Wed, Aug 28, 2013 at 11:13 AM, Mike Belshe <[email-masked]> wrote:

>> i don't understand how that approach addresses the "a bad person
>> initiates 100M one-packet connection attempts from different [forged/pwned]
>> addresses and doesn't follow through" SYN flood problem, where every packet
>> coming in would be "the first one", thereby creating connection state in
>> the server.
>>
+10.

-- 
Dave Täht

Fixing bufferbloat with fq_codel:
http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos#Dave-Taumlhts-FQ_codel-World-Tour-March-June-2013

---

From: Michael Welzl <[email-masked]>
Date: Thu, 29 Aug 2013 09:35:50 +0200

On 28. aug. 2013, at 17:52, Scott Brim wrote:

> On Wed, Aug 28, 2013 at 11:34 AM, Brian Trammell <[email-masked]> wrote:
> My only concern here -- and I say this as someone who's a big fan of the general approach of endpoints being smart in how they work around deficiency in the network -- is that SCTP and TCP provide different services. ...
> 
> So even if the OS handles the fallback (which it should), the application still has to determine whether the SCTP features it wants to use are available
> 
> Brian, this has been the case since (at least) IBM access method assembler macros.  Somehow the apps that want to do adjust, if you give them the hooks to do so.
> 
> We don't need to make it simple for everyone, just simple for those who want to use it.

Agree 100%.

But I'll also say that in fact, streams should be hidden, and so should multihoming (all an application wants is to efficiently communicate to the other host).

As for the actual application services provided by SCTP, such as unordered delivery and partial reliability, it seems to me that they're semantically consistent with TCP's ordered total reliable transfer, so the fall-back should just make things less efficient, but still correct for what the application expects.

Either way, we are, at this point, very much leaving the field that Mike would like to stay in when he says "Theory is nothing - execution is everything."  Being an academic, as much as I do value execution, I just can't agree that theory is nothing  :-)

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Thu, 29 Aug 2013 09:56:04 +0200

> I'm not sure if you're trolling me. In case you aren't, you may want to look at the graph at: http://gs.statcounter.com/#os-ww-monthly-201207-201307. Windows XP (released in 2001) is still around 2…

I'm not trolling you.

So, first, I think we all agree that SCTP over UDP could be a useful approach anyway, and indeed maybe that could be the long-term path towards wider OS-level deployment. Imagine we had SCTP-over-UDP used by browsers all the time, this might be an argument for putting it in the OS. The way things are now, OS folks won't do it cause applications don't use it, and vice versa. Having a multitude of application-specific solutions over UDP that all pretty much do the same things as SCTP doesn't help that, I don't imagine that QUIC et al are going to end up in the OS at some point.

Why is it even so important that things would be in the OS and not in user space? I don't know, maybe it isn't... I figure that some timing things could be done better, but maybe it really doesn't matter much. So yes, SCTP over UDP sounds like the way to go for me.

But just to explain my argument: for Happy Eyeballs with OS-level SCTP, my point was that one could use OS-level SCTP whenever it's there. When it's not, do the UDP thing. Yes it's more complicated than simply going for the over-UDP solution at all times, and may not be worth it (are there numbers showing that OS-level SCTP is better than userspace SCTP?) - I'm just trying to explain what I had in mind.

Anyway, when I wrote "the OS should do it for you", I was thinking about something quite different, and not a recommendation for what you put in your browser at all. Never mind.

> I have no attachments to the protocol name or frame format or whatever. Look at what we're doing in HTTP/2 which was inspired by SPDY but now has undergone substantial changes. We're serious about this. As long as the transport 

Sorry if my statement about acronyms rubbed you the wrong way, I was only trying to be a bit ironic here. What I mean is: SCTP seems to do a lot of the things that RTMFP or QUIC are meant to do. Rather than re-inventing the wheel over and over again, it might be better to have a discussion in tsvwg about what you guys think is missing in SCTP and seeing how it could be fixed.

I understand that speed of deployment is a key issue here, and trying to update SCTP in the IETF according to your wishes may take too long, but then just extending the code as you want might have been an even faster solution?! Well that doesn't matter and is of course your decision alone - either way, it's probably worth to look at the requirements addressed by RTMFP and QUIC, and not addressed by SCTP, and then discuss if or in which way SCTP should be extended.

> provides all the features we need, we'll use it. This conversation got started because tsvwg asked httpbis what the application layer wants from the transport. We're telling you. I think the constructive next step is for tsvwg folks to ask for clarification on any requirement they don't understand, discuss whether or not the requirements are reasonable, and discuss what may need to be done to address them.

Yes, absolutely.

Cheers,
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 29 Aug 2013 17:40:19 +0200

On Aug 29, 2013, at 4:36 PM, Mike Belshe <[email-masked]> wrote:

> 
> 
> 
> On Thu, Aug 29, 2013 at 12:56 AM, Michael Welzl <[email-masked]> wrote:
> 
> I find this thread frustrating because it inadvertently replaces our real end-user goals with technogeek protocol goals that will likely fail:
>     Step 1:  make broad claims that existing protocol XYZ already solves a problem like web page loading times  (even though it has never been proven to do so)
>     Step 2:  claim that the goal should be to deploy protocol XYZ since it already solves the problem ("rather than reinvent the wheel")
> 
> No no no!  By doing this we swapped the real goals for fake ones.  The result will be a bunch of nerds wondering why nobody is using their glorious new protocol.  Next we'll be writing a "Happy Eyeballs" style spec to help them see the light!   Look, if the protocol was so lackluster we had to write a spec to tell people how to deploy it, we failed to solve interesting problems.  It's time to start over, not time to keep re-brandishing the same tired old protocol.
> 
> To be clear: 
>     - OS-level deployment of any protocols is not a goal
>     - deployment of SCTP is not a goal
>     - getting SCTP into the kernel is not a goal
None of this is a goal. I like to understand which services/features/properties
are considered to be useful or required by the HTTP people. Then we can see
which protocol fits best (maybe TCP, maybe TCP Minion, maybe SCTP with extensions,
maybe QUIC, maybe something which doesn't exist yet). Evaluation at this
step really requires prototyping. I agree completely here.
> 
> Useful goals would be:
>       - making web pages load 10x faster
>       - making servers that scale to 1B users
>       - making the web accessible to more 10x people
>       - eliminating privacy leaks, phishing scams, etc
I agree to all of them. But we have to find a way from these goals to
technical features of the protocols...

Best regards
Michael
> Wishful thinking alert!  --  If it wasn't designed for a particular goal, it likely doesn't solve that particular goal.  And no, SCTP was not designed for low-latency web transactions.
> 
> Don't let me discourage you too much, I'd love to see what you say here turn out to be true.  But if these protocols worked as you say, they'd be deployed already.  You think nobody tried HTTP over SCTP?  Of course we did!  It's a marginal improvement at best, after you get past all the complexity and bugginess of protocols that were not designed for this purpose. 
> 
> I hope this doesn't come across too harsh.
> 
> Best,
> Mike
> 
> 
> 
> 
>  
>

---

From: Michael Welzl <[email-masked]>
Date: Fri, 30 Aug 2013 09:29:25 +0200

Hi,

> Wishful thinking alert!  --  If it wasn't designed for a particular goal, it likely doesn't solve that particular goal.  And no, SCTP was not designed for low-latency web transactions.

Low-latency signaling messages (which it was designed for) don't strike me as being extremely different from low-latency web transactions... well, maybe the devil is in the detail.

> Don't let me discourage you too much, I'd love to see what you say here turn out to be true.  But if these protocols worked as you say, they'd be deployed already.  You think nobody tried HTTP over SCTP?  Of course we did!  It's a

Just curious, which implementation did you test?
This group had posted pretty good results achieved with HTTP over SCTP long ago:
http://www.eecis.udel.edu/~amer/PEL/index.html
- although the only paper I can dig up from their page about this now is this one:
http://www.eecis.udel.edu/~amer/PEL/poc/pdf/SIGCOMM-NSDR2008-NatarajanMultistreamed-transport-developing-regions.pdf

and now their page mentions a "SPDY over SCTP project supported by Google". So are you saying that the outcome of this project is disappointing already?

> marginal improvement at best, after you get past all the complexity and bugginess of protocols that were not designed for this purpose. 
> 

No it doesn't - it's interesting and makes me want to dig deeper into QUIC and RTFMP to better understand what makes it perform so much better than SCTP.

( I figured that multi-streaming at the transport layer would be the big win, and since SCTP already gives you that, well...  though I did notice the mention of e.g. lack of per-stream flow control in SCTP, which I also found to be a problem long ago... I had thought that this might have been fixed by now? Shouldn't be too hard to fix. )

Cheers,
Michael