Subject: fyi: An Experimental Study of Web Transport Protocols in Cellular Networks

From: gabriel montenegro <[email-masked]>
Date: Thu, 29 Mar 2012 02:27:48 -0700 (PDT)

Hi folks,

Just heard of this study which is very relevant to the HTTP 2.0 discussions:

Binoy Chemmagate (Aalto University, Finland) 
An Experimental Study of Web Transport Protocols in Cellular Networks 
M.S. Thesis, October 2010 - September 2011
http://eggert.org/students/chemmagate-thesis.pdf

Gabriel

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Thu, 29 Mar 2012 10:17:19 +0000

that's very interesting.
  
There's another very interesting study done by Google into TCP fast 
open (SYN + data).
  
research.google.com/pubs/archive/37517.pdf
  
It also havs very interesting stats on latency in the real world.
Seems to me we should be pushing harder to get that working, it's 
basically a "free" RTT.

------ Original Message ------
From: "gabriel montenegro" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 29/03/2012 10:27:48 p.m.
Subject: fyi: An Experimental Study of Web Transport Protocols in 
Cellular Networks

---

From: Mike Belshe <[email-masked]>
Date: Thu, 29 Mar 2012 14:13:14 +0200

I worked with Binoy a fair bit when he was getting this setup, as he used
software from the spdy codebase extensively.

The data is getting a little old at this point - I believe all of these
tests were done over a year ago now.  But Binoy could confirm.

He also didn't test the impact of SSL at all, although his conclusion makes
references to SSL, it wasn't actually part of the tested code.  Binoy has
confirmed this for me.

Mike

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Thu, 29 Mar 2012 12:34:19 +0000

------ Original Message ------
From: "Mike Belshe" <[email-masked]>
To: "gabriel montenegro" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 30/03/2012 1:13:14 a.m.
Subject: Re: fyi: An Experimental Study of Web Transport Protocols in 
Cellular Networks
>I worked with Binoy a fair bit when he was getting this setup, as he 
>used software from the spdy codebase extensively.  
>The data is getting a little old at this point - I believe all of 
>these tests were done over a year ago now.  But Binoy could confirm.
>
>He also didn't test the impact of SSL at all, although his conclusion 
>makes references to SSL, it wasn't actually part of the tested code.  
>Binoy has confirmed this for me.
  
yep, bottom of page 35 it's mentioned.  He also mentions (bottom p45) 
removing time (1.8s on average for HSPA) of the TCP RTT.
  
So I'm finding it difficult to draw clear conclusions.  
  
Adrien
>
>Mike
>
>
>On Thu, Mar 29, 2012 at 11:27 AM, gabriel montenegro <[email-masked]> wrote:
> Hi folks,
>

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 29 Mar 2012 08:54:24 -0400

On 3/29/2012 6:17 AM, Adrien W. de Croy wrote:
>  
> that's very interesting.
>  
> There's another very interesting study done by Google into TCP fast open
> (SYN + data).
>  

The IETF is already working on this:
http://tools.ietf.org/html/draft-ietf-tcpm-fastopen-00

-- 
Wes Eddy
MTI Systems

---

From: <[email-masked]>
Date: Thu, 29 Mar 2012 12:58:29 +0000

Hi,

Binoy's not on the list, AFAIK, but I was also working with him in the project.

What Mike says is true. Most of the tests were done about a year ago, with whatever Chrome and the Flip server had at that time, with a lot of help from Mike. I'm not sure if the SPDY specification/implementation has changed considerably in this regard since then. SSL was not included.

What we did was that we setup a copy of the contents of a bunch of real websites on the Flip server supporting SPDY, and accessed then over the mobile network with both SPDY and HTTP. We emulated the website setup also what comes to the use of multiple domains, so that the number of TCP connections used would match real life. We ran the tests primarily on top of GPRS (~100 kbps, 400 ms RTT) and WCDMA HSPA (~1 Mbps, 100 ms RTT). We run a on a real cellular access network and run a large number of runs to avoid random effects. We measured mainly just the time from the first TCP SYN going out to receiving the last byte of content of all objects on the main page, i.e. the full page load time. DNS was not included either.

The results we got were that SPDY was indeed faster than HTTP as expected. It very much depended on the site, but over GPRS most of the sites had 5-15% smaller page load times, while in HSPA it was 10-20% (discluding the initial radio channel setup delay). In GPRS the overall page load times were often in the order of 30-90 seconds, while in HSPA around 10 seconds, so you can calculate the absolute savings from there. Majority of savings seemed to stem from compression, but the better multiplexing of SPDY vs. HTTP also helped on some sites.

There were some unideal cases in our tests. For instance, SPDY still opened as many TCP connections as HTTP, even if it only used one per server. I think that has been fixed since, and would perhaps improve SPDY's figures a little bit. Also the compression gains were not always very intuititive, so there could have been something faulty with that in some cases. So I would not treat these as absolute numbers on how much SPDY is faster than HTTP in mobile networks, but it gives at a least a starting point.

BTW, we also tested the effect of TCP initial window of 3 vs. 10 at the same time. To our amazement, the effect was nearly unnoticeable, less than 5% in majority of cases. The access to the very first objects on the page should benefit more than that, but we didn't measure it.

One conclusion was also that in mobile the best way to make web faster is to upgrade the access networks :) HSPA is superior to GPRS and LTE is even better than that. But also the HTTP/2.0 effort and SPDY would give very useful improvements! So based on our results HTTP/2.0 work clearly makes sense. If we can include the battery saving aspects that Gabriel's draft talks about, even the better. For most people battery is actually more important to speed these days when it comes to smartphones.

Markus

From: ext Mike Belshe [mailto:[email-masked]]
Sent: 29 March, 2012 15:13
To: gabriel montenegro
Cc: [email-masked]
Subject: Re: fyi: An Experimental Study of Web Transport Protocols in Cellular Networks

I worked with Binoy a fair bit when he was getting this setup, as he used software from the spdy codebase extensively.

The data is getting a little old at this point - I believe all of these tests were done over a year ago now.  But Binoy could confirm.

He also didn't test the impact of SSL at all, although his conclusion makes references to SSL, it wasn't actually part of the tested code.  Binoy has confirmed this for me.

Mike

On Thu, Mar 29, 2012 at 11:27 AM, gabriel montenegro <[email-masked]<mailto:[email-masked]>> wrote:
Hi folks,

Gabriel