Subject: [v6ops] Measuring the Effectiveness of Happy Eyeballs

From: "Bajpai, Vaibhav" <[email-masked]>
Date: Thu, 4 Jul 2013 13:02:53 +0000

Hello,

I would like to request a 10-minute presentation slot 
at the upcoming IETF 87 to present my PhD work:

Title:		 Measuring the Effectiveness of Happy Eyeballs
Authors:	 Vaibhav Bajpai and Jürgen Schönwälder
URL:             http://tools.ietf.org/html/draft-bajpai-happy-00

Abstract:

  The IETF has developed solutions that promote a healthy IPv4 and IPv6
  co-existence.  The happy eyeballs algorithm for instance, provides
  recommendations to application developers to help prevent bad user
  experience in situations where IPv6 connectivity is broken.  This
  document describes a metric used to measure the effectiveness of the
  happy eyeballs algorithm.  The insights uncovered by analysing the
  data from multiple locations is discussed.

Thank you!

Best, Vaibhav

-----------------------------------------------------
Vaibhav Bajpai

Research I, Room 86
Computer Networks and Distributed Systems  (CNDS) Lab
School of Engineering and Sciences
Jacobs University Bremen, Germany

www.vaibhavbajpai.com

---

From: Dan Wing <[email-masked]>
Date: Mon, 8 Jul 2013 09:05:56 -0700

>  data from multiple locations is discussed.

I noticed when this was published and presented earlier at https://ripe66.ripe.net/presentations/263-ripe66-happy-slides.pdf.  It's nice to see it published as an Internet Draft, but disappointing that the underlying Effective Measurement is testing something other than Happy Eyeballs.

Happy Eyeballs is doing what it was designed to do -- provide a good user experience when the IPv6 (or IPv4) path is down.  However, draft-bajpai-happy did not evaluable how well Happy Eyeballs handles a broken IPv6 or broken IPv4 path.  Instead, what draft-bajpai-happy measured was how well the selected path functioned.  Happy Eyeballs biases its path selection towards IPv6 by design (150-250ms timeout is suggested in http://tools.ietf.org/html/rfc6555#section-5.5).  The justification for this delay is explained in "Delay IPv4", http://tools.ietf.org/html/rfc6555#section-4.1, and was consensus of the working group primarily because it (a) mimics the long-standing IETF view that IPv6 should be preferred over IPv4 (b) reduces connection attempts on servers and (c) minimizes the harm to IPv4-only devices sharing an IPv4 address with the dual-stack client (as those IPv4-only devices cannot use IPv6).  Happy Eyeballs' algorithm differs from Apple's algorithm (introduced in OS X 10.7) which uses whichever path connects first (for details see http://lists.apple.com/archives/ipv6-dev/2011/Jul/msg00009.html), which I expect would result in better results if tested using the test methodology of draft-bajpai-happy.  However, even with Apple's algorithm if a path connects quickly but the path performs poorly (e.g., low bandwidth) the user experience will suffer.  Happy Eyeballs can perform similarly to Apple's algorithm (but not identically) by setting its connection delay to 0ms.

-d

> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops

---

From: "Tony Hain" <[email-masked]>
Date: Mon, 8 Jul 2013 12:09:08 -0700

Dan Wing wrote:
> On Jul 4, 2013, at 6:02 AM, "Bajpai, Vaibhav"
<[email-masked]>
> wrote:
> 
> > Hello,
> >
> > I would like to request a 10-minute presentation slot at the upcoming
> > IETF 87 to present my PhD work:
> >
> I noticed when this was published and presented earlier at
> https://ripe66.ripe.net/presentations/263-ripe66-happy-slides.pdf.  It's
nice
> to see it published as an Internet Draft, but disappointing that the
underlying
> Effective Measurement is testing something other than Happy Eyeballs.

It doesn't even acknowledge that the target nodes differ. DNS resolution of
a name does not tell you anything about deployed topology.

> 
> Happy Eyeballs is doing what it was designed to do -- provide a good user
> experience when the IPv6 (or IPv4) path is down.  However, draft-bajpai-
> happy did not evaluable how well Happy Eyeballs handles a broken IPv6 or
> broken IPv4 path.  Instead, what draft-bajpai-happy measured was how well
> the selected path functioned.  

Well, it offer an ROFL moment with 
"will never use Teredo IPv6 unless IPv4 connectivity is broken"
clearly lacking the understanding that IPv4 is required to work for Teredo
to function...

> Happy Eyeballs biases its path selection
> towards IPv6 by design (150-250ms timeout is suggested in
> http://tools.ietf.org/html/rfc6555#section-5.5).  The justification for
this
> delay is explained in "Delay IPv4",
http://tools.ietf.org/html/rfc6555#section-
> 4.1, and was consensus of the working group primarily because it (a)
mimics
> the long-standing IETF view that IPv6 should be preferred over IPv4 (b)
> reduces connection attempts on servers and (c) minimizes the harm to IPv4-
> only devices sharing an IPv4 address with the dual-stack client (as those
IPv4-
> only devices cannot use IPv6).  

If the IPv6 path is not given an automated preference the traffic will never
move. Even with emerging deployment on 'services', the cache topologies are
not identical in every instance, and getting them there requires
demonstrated traffic. People continue to complain that IPv6 traffic levels
are low, but they will continue that way until ISP's and content providers
make everything identical between the protocol versions. Given that many use
lack of traffic at other sites as an excuse for not doing their own
deployment, breaking the stalemate requires a bias.

> Happy Eyeballs' algorithm differs from Apple's
> algorithm (introduced in OS X 10.7) which uses whichever path connects
first
> (for details see http://lists.apple.com/archives/ipv6-
> dev/2011/Jul/msg00009.html), which I expect would result in better results
if
> tested using the test methodology of draft-bajpai-happy.  However, even
> with Apple's algorithm if a path connects quickly but the path performs
> poorly (e.g., low bandwidth) the user experience will suffer.  Happy
Eyeballs
> can perform similarly to Apple's algorithm (but not identically) by
setting its
> connection delay to 0ms.

Apple's implementation is "not helpful". They start by refusing to allow
modifications to the 3484 policy table to meet local policies, then follow
that with a refusal to ack that the IPv4 cache nodes are generally closer to
the request than the IPv6 ones, so traffic stays on IPv4 when it could &
should have moved. Anything less than a 100ms delay for the IPv4 request is
essentially guaranteeing that the traffic stays on IPv4, until the cache
topology catches up, which will never happen if the traffic loads can't
justify it. 

Tony

---

From: Lorenzo Colitti <[email-masked]>
Date: Wed, 10 Jul 2013 18:23:56 +0900

On Tue, Jul 9, 2013 at 4:09 AM, Tony Hain <[email-masked]> wrote:

> move. Even with emerging deployment on 'services', the cache topologies
> are not identical in every instance, and getting them there requires
> demonstrated traffic.

Actually it's worse than that. Even when topology is congruent and IPv4 and
IPv6 have exactly the same latency, a "use the first connection that
completes" implementation with no preference will use IPv4 50% of the time,
just due to statistical noise (introduced, if nothing else, by hashing
along the path).

> should have moved.

They also bias in favour of IPv4 because they send the A query before the
AAAA query (or at least they used to).

---

From: Erik Kline <[email-masked]>
Date: Wed, 10 Jul 2013 18:48:47 +0900

On 10 July 2013 18:23, Lorenzo Colitti <[email-masked]> wrote:
> just due to statistical noise (introduced, if nothing else, by hashing along
> the path).
>
>>

Some data.

Here's a slide from my presentation at the IPv6 congress in Paris this
past March.

The pink is % of connections that arrived over IPv6.  The client
population is a set of gigabit FTTH native IPv6 users, where IPv4 is
provided encap'd within IPv6.

The mapping between clients and HE implementation is basically:

    Nexus 7:  basically a special subset of Chrome (below)
    MSIE on NT6+:  the well-known IPv6 probe on network attach
    Chrome:  still the 300ms head start from just before W6D in 2011
    Safari on OS X 10.[78]:  connection racing

Even when IPv4 is provided as an encap'd service in a native IPv6
network you can see that connection racing does not help the provider
obtain much of the return on investment in IPv6 that might have come
from migrating traffic away from IPv4 (e.g. in a CON environment).

---

From: Arturo Servin <[email-masked]>
Date: Wed, 10 Jul 2013 09:58:02 -0300

Is the destination the same in all the samples?

    (I assume it is and that is why it is so weird but better ask).

Regards
as

---

From: "Bajpai, Vaibhav" <[email-masked]>
Date: Wed, 10 Jul 2013 13:53:14 +0000

On Jul 8, 2013, at 6:05 PM, Dan Wing <[email-masked]> wrote:

>> [...]
> https://ripe66.ripe.net/presentations/263-ripe66-happy-slides.pdf.  

Thank you for noticing it :-)

> It's nice to see it published as an Internet Draft, but disappointing
> that the underlying Effective Measurement is testing something other
> than Happy Eyeballs.
> 
> experience when the IPv6 (or IPv4) path is down.  However, draft-bajpai-happy
> did not evaluable how well Happy Eyeballs handles a broken IPv6 or broken IPv4
> path.  Instead, what draft-bajpai-happy measured was how well the selected
> path functioned.  

Yes! this is true. This is because we _wanted_ to measure in a real 
uncontrolled environment. We wanted to know: if all the applications today
get happy eyeballed, how would the dual-stack user experience be? This why
we made significant effort to find and deploy probes and measure from 
behind a residential gateway of a real dual-stack host.

We don't change the network conditions, we only measure what the end-host
might experience from a happy eyeballed application. Therefore, we think
we do measure happy eyeballs on how it would perform in the wild today.

> Happy Eyeballs biases its path selection towards IPv6 by
> design (150-250ms timeout is suggested in
> http://tools.ietf.org/html/rfc6555#section-5.5).  

The draft mentions that 300ms is used in Chrome and Firefox. Since, the
implementations are using their own timer value than what is recommended,
it would be nice to know what would be a better timer value, by directly
measuring from deployed dual-stacked hosts, no?

> The justification for this delay is explained in "Delay IPv4",
> http://tools.ietf.org/html/rfc6555#section-4.1, and was consensus of the
> working group primarily because it (a) mimics the long-standing IETF view that
> IPv6 should be preferred over IPv4 (b) reduces connection attempts on servers
> and (c) minimizes the harm to IPv4-only devices sharing an IPv4 address with
> the dual-stack client (as those IPv4-only devices cannot use IPv6).

We understand it's a policy discussion to facilitate IPv6 adoption. We will
add a section describing this policy decision. We will also add a section in
the draft to clarify the motivation of RFC6555.

Best, Vaibhav

> Happy Eyeballs' algorithm differs from Apple's algorithm (introduced in OS X
> 10.7) which uses whichever path connects first (for details see
> http://lists.apple.com/archives/ipv6-dev/2011/Jul/msg00009.html), which I
> expect would result in better results if tested using the test methodology of
> draft-bajpai-happy.  However, even with Apple's algorithm if a path connects
> quickly but the path performs poorly (e.g., low bandwidth) the user experience
> will suffer.  Happy Eyeballs can perform similarly to Apple's algorithm (but
> not identically) by setting its connection delay to 0ms.

Research I, Room 86

www.vaibhavbajpai.com

---

From: "Bajpai, Vaibhav" <[email-masked]>
Date: Wed, 10 Jul 2013 14:01:21 +0000

Hello,

Thank you for the review!

> a name does not tell you anything about deployed topology.

We understand that the target nodes differ. We will update the
text to explicitly mention it.

> Well, it offer an ROFL moment with "will never use Teredo IPv6 unless
> IPv4 connectivity is broken" clearly lacking the understanding that
> IPv4 is required to work for Teredo to function…

We meant when the IPv4 reachability of the destination service is broken
(not of the end-host). We will amend the text. Thank you for noticing this.

Best, Vaibhav

Research I, Room 86

www.vaibhavbajpai.com

---

From: Lorenzo Colitti <[email-masked]>
Date: Wed, 10 Jul 2013 23:14:36 +0900

On Wed, Jul 10, 2013 at 10:53 PM, Bajpai, Vaibhav <
[email-masked]> wrote:

> The draft mentions that 300ms is used in Chrome and Firefox.
>

Does Firefox use 300ms too? I thought Firefox did something more similar to
Apple than to Chrome.

---

From: "Bajpai, Vaibhav" <[email-masked]>
Date: Wed, 10 Jul 2013 15:42:12 +0000

On Jul 10, 2013, at 4:14 PM, Lorenzo Colitti <[email-masked]> wrote:

> On Wed, Jul 10, 2013 at 10:53 PM, Bajpai, Vaibhav <[email-masked]> wrote:
> Does Firefox use 300ms too? I thought Firefox did something more
> similar to Apple than to Chrome. 

The more aggressive approach needs to be enabled by setting 
network.http.fast-fallback-to-IPv4 parameter to true. Firefox 
then starts parallel TCP connections to the first endpoint of
each address family [1].

[1] http://www.potaroo.net/ispcol/2012-05/notquite.html

Best, Vaibhav

Research I, Room 86

www.vaibhavbajpai.com

---

From: Lorenzo Colitti <[email-masked]>
Date: Thu, 11 Jul 2013 00:51:50 +0900

> then starts parallel TCP connections to the first endpoint of

Actually, as far as I can see, Firefox (at least the version I have, v22 on
Ubuntu) has that parameter set to true by default (if you go to
about:config, you can tell if it's been modified). If you set it to false
it doesn't do HE at all.

It appears to fall back to IPv6 after ~250ms, at least in this case where I
connect to a port that doesn't respond:

00:46:51.836461 IP6 2400:2410:20c0:100:656d:e13c:6236:d37a.45342 >
2404:6800:4004:803::1012.81: Flags [S], seq 2062516864, win 14400, options
[mss 1440,sackOK,TS val 179759058 ecr 0,nop,wscale 7], length 0
00:46:52.087045 IP 192.168.3.5.55797 > 74.125.235.115.81: Flags [S], seq
4216986802, win 14600, options [mss 1460,sackOK,TS val 179759120 ecr
0,nop,wscale 7], length 0

> [1] http://www.potaroo.net/ispcol/2012-05/notquite.html

I wouldn't take that as the gospel truth. It also says that in Chrome "The
choice of which protocol is preferred for the initial connection attempt is
based on the DNS response time", which as far as I know is not true.
Whether it prefers IPv6 or not depends only on what getaddrinfo returns
first.

---

From: "Bajpai, Vaibhav" <[email-masked]>
Date: Sun, 14 Jul 2013 21:04:23 +0000

Hello,

Thank you so much for all the feedback on the -00 version.

I have uploaded a new version of the draft with requested 
changes and with a new title:

Revision:	 01
Title:		 Measuring the Effects of Happy Eyeballs
Htmlized:        http://tools.ietf.org/html/draft-bajpai-happy-01
Diff:            http://www.ietf.org/rfcdiff?url2=draft-bajpai-happy-01

It would be nice to get a 10 minute agenda slot to 
present this at the upcoming IETF.

The changes reflect from these discussions:

On Jul 10, 2013, at 3:16 PM, "Bajpai, Vaibhav" <[email-masked]> wrote:

> On Jul 10, 2013, at 4:29 PM, Lorenzo Colitti <[email-masked]> wrote:
> 
>> […] If you do want to measure this, I think you might have to call it something
>> other than "effectiveness". […]

done! changed the title.

> We argue that since applications on top of TCP will not be happy eyeballed
> only in scenarios where IPv6 connectivity is broken, but also in scenarios
> where the dual-stack host enjoys perfect IPv6 connectivity, we want to
> measure how much imposition does such a user experience in reality by
> measuring the effect of the 300ms timer value.
> 
>> Actually, that's not a good definition either. If the dual-stack host enjoys
>> perfect IPv6 connectivity, then by definition IPv4 can never be better (since
>> IPv6 is perfect), the 300ms timer will never fire. So by "perfect" I think what
>> you really mean is "perfectly reliable".

done! rephrased the definition.

> On Jul 5, 2013, at 4:30 PM, Andrew Yourtchenko <[email-masked]> wrote:
> 
>> First, the overall goal. The RFC6555 was not designed to [...]
> 
> […] We will add a section in the draft to clarify the motivation of RFC6555.

done! added a new section clearly describing the goals of happy eyeballs.

>> The justification for this delay is explained in "Delay IPv4", [...]
> 
> add a section describing this policy decision. [...]

done! added a new section on IPv6 upgrade policy.

>> It doesn't even acknowledge that the target nodes differ. [...]
> 

done! added the text acknowledging the difference.

>> […] clearly lacking the understanding that IPv4 is required
>> to work for Teredo to function…
> 

done! rephrased the statement.

Best, Vaibhav

Research I, Room 86

www.vaibhavbajpai.com