Subject: Re: [v6ops] Comments on draft-wing-v6ops-happy-eyeballs-ipv6-01

From: Andrew Yourtchenko <[email-masked]>
Date: Wed, 2 Mar 2011 05:21:57 +0100 (CET)

Hi Bjoern,

Apologies for the latency, just got to editing everything...

First of all - thanks a lot for the very detailed comments!

On Thu, 27 Jan 2011, Bjoern A. Zeeb wrote:

> Hi,
>
> (ignore the things that have been said since October already)
>
> General comments upfront:
>
> - I don't like the name of compnay-of-the-days being used as samples
>  in standards, especially when there is no reference attached to it
>  and if it's just one in a zillion.  You should generalize this.
>

s/Facebook/social networks/; s/Google/content provider/.

> - I left some private notes (XXX-BZ) in there that I had at the time
>  of reading.  They may either provide information or be addressed at
>  the end.
>

thanks!

>
>> 1.  Introduction
>>
>>    In order to use HTTP successfully over IPv6, it is necessary that the
>>    user enjoys nearly identical performance as compared to IPv4.  A
>>    combination of today's applications, IPv6 tunneling and IPv6 service
>>    providers, and some of today's content providers all cause the user
>>    experience to suffer (Section 3).  For IPv6, Google ensures a
>
> The thing that makes me currently suffer the most wrt to IPv6 is
> `magic (or call it bug) in a commercial OS resolver tralala` forcing me
> to flush caches up to 70 times a day to be able to get to ipv6 enabled
> sites.  This comment to my best understanding is very related to the
> roots of your draft (but I have yet limited understanding of the actual
> problem causing this bug to show up for people; it's been reported by
> multiple, yet not fixed).

if you have time in Prague, would be very interesting to debug together. I use 
don't use that resolver, so no means to debug.

>
>
>>   While the application recommendations in this document are described
>>   in the context of HTTP clients ("web browsers"), but is useful and
>>   applicable to other time-sensitive applications.
>
> I'd remove the `other` here as "web browsing" is all but time-sensitive
> in the classical meaning.  The 300ms you will not really notice for a
> page-load-time can for some other network usage ruin entire entire
> companies or cost them fortunes.

s/time-sensitive/interactive/

>
>
>> 3.  Problem Statement
>> ..
>>   As discussed in more detail in Section 3.2, IPv6 connectivity is
>>   sometimes broken entirely or slower than native IPv4 connectivity.
>
> Contrary to my findings over the last years, where IPv6 often was
> faster or more reliable especially on trans-continental links between
> the same two dual-stack systems (as seen from DE to CA.US, DE to CA and
> DE to AU).
> I have even found that going via some 3rd party IPv6 rather than directly
> into the destination network at the next IXP using their internal IPv4
> transport, IPv6 was still faster.  I also like the variance, especially
> as in the following samples the IPv6 is actually tunnelled and still
> better.
>
> (DE - AU)
> round-trip min/avg/max/stddev  = 332.616/357.183/408.802/28.150 ms
> round-trip min/avg/max/std-dev = 336.952/338.048/340.461/0.973 ms
>
> (DE - CA [east coast])
> round-trip min/avg/max/stddev = 129.852/136.686/145.283/5.634 ms
> round-trip min/avg/max/std-dev = 118.636/120.272/121.628/0.996 ms
>
> [I wonder if someone may have the data of a lot more probes somewhere?]
>

Interesting. In my measurements the IPv6 was usually a little bit slower, but 
then I had quite some of tunneled IPv6...

> So I think, even though pointing to the other section, the above
> should be a lot more precise or limit things to "very special
> circumstances" as I think it's just the very obvious brokeness you try
> to address but you'll also catch the 10ms deltas in other areas
> unfortunately.

Yeah, in the beginning it may catch and amplify the noise. Means to be 
preferred, one service must be better than the other one (possibly with some 
headstart for IPv6). I don't have a good idea on what specifically to change - 
not putting edits for now, let's talk this more.

>
>
>> 3.2.  IPv6
>> 
>> ...
>>   Reasons for such failure include no connection to the IPv6 Internet,
>>   broken 6to4 or Teredo tunnels, and broken IPv6 peering.
>
> Hmm, if there is no connection you should get the almost instant
> notification back and move on.  Broken IPv6 peering as much in
> broken IPv4 peering (I don't think the legacy world is any better).
>
>
>>         1.    |<--www.example.com A?-----|                       |
>>         2.    |<--www.example.com AAAA?--|                       |
>>         3.    |---192.0.2.1------------->|                       |
>>         4.    |---2001:dba::1----------->|                       |
>
> Hmm, www.example.com has A and AAAA records; either use those or
> better 2001:db8:: rather than 2001:dba:: (here and throughout the document).
>
>
> The problem described here, really is not that much IPv6 specific and
> not much different to just multiple-A records, is it?

correct - though the assumption is that the connectivity properties within the 
AF are more homogenous than between different AFs, and that at a given point of 
time one AF will have better connectivity than the other.

>
>
>>   The client attempts to connect using IPv6 to the server, but the IPv6
>>   path is broken (6-8), which consumes several seconds of time.
>
> The entire thing strikes me as as general TCP behaviour (often tunable
> by global MIB variables or per socket options depending on the OS)
> told how to handle since at least RFC1122 (STD3) section 4.2.3.5.
>

Yes. Seems like the defaults are not very friendly to the user, though.

>
>> 4.1.  IPv6
>
> I think the section title, if not already for 3.2, at least here is
> misleading.  The very best it descripes "Dual Stack" behaviour.
>
>
>>  In diagram above, ...
>
> Add 'the'.
>
> XXX-BZ revisit (what if there is more than one record returned per address
> family (AF))

Whatever happens today (first of the answers?) seems to work in IPv4. Whether 
the same approach would work with IPv6 - maybe, maybe not. Agree we should 
talk more about this.

>
>>   using IPv4.  The IPv6 path is retried until the application gives up
>>   (10).
>
> I think given the diagram you should s/the application gives up/TCP
> times out/ here.

No. It's when IPv4 happened to work all right and we give up on trying the IPv6 
for this connection.

>
>
>>   The absolute value of
>>   P is the measure of a delay before initiating a connection attempt on
>>   the other address family.
>
> A 'connection attempt' in literature seems to be more the one of the
> real upper layer protocoal (ULP) connection but what you want to say is
> "the DNS lookup and connection attempt ..."

s//a DNS lookup and /

>
>
>>   The TCP client application starts two threads in order to minimize
>>   the user-noticeable delay ("dead time") during the connection
>>   attempts:
>
> I think the draft should not enforce how to implement things.
> Threads are just one way to implement the expected (concurrent)
> behaviour.  I would love to see the word 'thread' to go away from the
> draft completely or detailed that other ways to implement the expected
> behaviour may equally be chosen.

Erik also had the same comment. I've put it this way:

"The TCP client application starts two concurrent execution flows (they will be 
referred to as "threads" but this reference does not imply the implementation 
detail of using the threading library, merely the property of mutual 
concurrency) in order to minimize the user-noticeable delay ("dead time") during 
the connection attempts:"

The "flow" is used in other contexts as well in other places, so really we need 
a third word. I don't have a good one to do a global search/replace with.

>
>
>>       *  If P<0, wait for absolute value of p*10 milliseconds
>
> Mix of 'P' and 'p'.  Should be all upper-case?
>
> XXX-BZ revist - which P is this - the per destination or the global?
> (should be disambiguated throughout the entire draft Ps (P socket) and
> Pa (P application) or similar and one or the other or both mentioned
> explicitly where appropriate.

That's something which should really be marked as "to be experimented with". 
Back when we did the very first experiments, I simply had a single value that 
gave decent user experience.

>
>
>>   The value of P
>>   is incremented (decremented) each time an IPv6 (IPv4) connection is
>>   successfully made.
>
> I felt this is inaccurate.  Both IPv4 and IPv6 connections could
> succeed, even before you can cancel the other (connection|thread),
> so I think you mean s/is successfully made/wins the race/.

Yes.

>
>
>>   When a connection using the less-preferred
>>   address family is successful, it indicates the wrong address family
>>   was used and the P is halved:
>>  ...
>
> For the two items:
>>   o  If P>0  ...
>>   o  If P<0  ...
>
> I'd prefer to have the exact same wording with just s/[46]/[64]/
> s/[<>]/[><]/.

Is is more about the order of the two paragraphs ?

The 'elevator pitch' is for all of these:

The value of P is incremented (decremented) each time an IPv6 (IPv4) connection 
wins the race if it was preferred originally, and halved if an AF wins the race 
while not being the preferred protocol.

>
>>   o  If P=0 (indicating equal preference), P is incremented if the
>>      first thread to complete was the IPv6 thread, or decremented if
>>      the first thread to complete was the IPv4 thread.
>
> "incremented by 1" "decremented by 1"
>
>>   which
>>   is similar to the value used by many IPv6-capable TCP client
>>   applications to switch to an alternate A or AAAA record.
>
> Again, it's not the application usually (unless using setsockopt) but
> the intial connection setup timeout of the TCP stack.
>
>
>>      Note:  Proof of concept tests on fast networks show that even
>>      smaller value (around 0.5 seconds) is practical.  More extensive
>>      testing would be useful to find the best upper boundary that still
>>      ensures a good user experience.
>
> I think in both cases here and the paragraph above a value of P should
> not be in seconds but the absolute value of 400 or 50 (given it's
> |P|==400 [ * 10ms = 4s ] ).
>
>
> I feel that |P|==50 is optimistic given that it can easily be 450+ms
> from Europe to Australia meaning losing the race twice the application
> would immediately favour the other AF.

I've done s/is/may be/. Indeed more testing is needed.

>
> XXX-BZ revist and explain the general problem with the maths in here
> as I understand it; general failover, larger caches, bad expiry
>
> XXX-BZ again, what about multiple AAAA or multiple A records and how
> to count that if the first records wins but we move on to the second
> for an application failure on that AF?

You mean the connection succeeds but we decide to drop it by the application and 
reestablish ?

>
>
>> 4.2.3.  Flush or Expire Cache
>> ...
>>   However, in some instances the
>>   application and the host are unaware the network connectivity has
>>   changed so it is RECOMMENDED that per-destination values expire after
>>   10 minutes of inactivity.
>
> XXX-BZ see further down
>
>> 4.2.4.  Determining Address Type
>
> I would remove that section entirely.  It would mean just more magic
> that might not be needed at all.  If replacement transition technolgy
> would work better than native, so be it;-)
>

This needs more discussion since I got both the "keep" and "remove" opinions :-)
On one hand, it does look logical, but then if we're preferring a client-facing 
NAT64 prefix IPv6 address it is not a whole lot different than preferring IPv4 
to IPv6 as far as the rest of the Internet is concerned.

>
>> 4.2.5.  Debugging and Troubleshooting
>> ...
>> mechanism to temporarily use only one address family.
>
> The much I'll love once we are there, this is somewhat out of scope I fear
> as you can write an entire draft about the pros and cons of just that if
> you have the time.  I'd much rather like to see the fallback to revert to
> the well-known beviour without this draft (whatever it is, mostly usually
> disabling the concurrency and trying on after the other).
> If you disable one address family a debug trace will usually only show you
> half of the information you may want/need.

"... that regard, the applications implementing the proposal in this
       document SHOULD also provide a mechanism to revert the behavior
       to that of a default provided by the operating system.</t>"

I've also added this blurb, which I think can help make the troubleshooting a 
more automated process for those who care:

[[[ To be discussed.
       Some sites may wish to be informed when the the hosts adjust their "P" value,
       in order to troubleshoot the underlying cause. To help these sites,
       a strawman proposal is to send a syslog message or other notification
       to an address that may be configured
       by a site administrator in a centralized fashion.
       (The exact method TBD - DHCP option, domain name, etc.)
       This syslog message should be sent only first N times that the host
       expects to prefer IPv6 but has to use IPv4. I.e. the first N times it decreases
       the value of P. N - TBD.
]]]

i.e. give the host an ability to send the cry for help if the administrator 
instructed it to. Given that the whole idea is around the user experience, the 
operator then can:

1) aggregate: measure the number of "cries for help" to see how bad the problem 
is (how many heads to fix it)
2) look at specifics and proactively contact those users.

This makes a troubleshooting part of me think that this is an improvement
compared to what we have now.

>
>
>
> Ok, some of the XXX-BZ to be addressed still in general outside the
> scope of a specific paragraph:
>
> "Which P?"
>
> I feel the two 'P's need to be better explained, as in when to update
> which one and when to pick which one.  If I had to guess I would go
> for a specific implementation but that does not mean anyone else would
> go for the same.  Be more explicit about that.
>

Let's hash this out in the bar BoF in Prague
(or if there are good arguments on this topic before that, would be good as 
well on-list).

>
> "What about multiple AAAA or A records?"
>
> I know the abstract says "dual stack client" to "dual stack server"
> but while in a majority of the cases that might still be true, it's an
> ideal world that seems to be especially untrue for the top n sites.
>
> What should an application do if it gets back only two AAAA records
> or 2 AAAA and 4 A records?
>
> It may well be one of the machines is in Europe and the other is in the
> Asia?  Do you want to try those in sequence only if one fails or do
> you also want to measure the 150ms out that the other one could be
> faster?  When, in your model, will you then start more threads?

No. It'll be up to the DNS server to serve me a decently suitable answer as the 
first answer within the AF.

>
> What if the first A wins your match above but the ULP errors? How will
> the retry work or the moving on to another record? Will we try the
> other AF then depsite having clsoed the conneciton there?
>
> Depsite they usually should do not expect application to behave exactly
> the same no matter which AF you are using.
>

hm. why ? I thought the whole point of dualstackness was more or less to treat 
both protocols the same way from the logical point of view.

>
> "general failover, larger caches, bad expiry"
>
> Flushing entries from the cache after 10 min. of inactivity has multiple
> problems:
>
> - Say my browser has about the same uptime as my machine.
> - Say I keep random-modern-ajax-website open in a window all the time.
> - The website will refresh itself every minute or even have "pseudo"
>  persistent connections (there's a lot of variance in there what
>  you can observe).
>
> The result of this will be kind of like: my preference will grow to the
> max until there is a network hickup.  The question then is will it
> last long enough so the preference will flip or will I just see the
> 3s delay?  Or will it cause the converse and put me on the other
> address family for the next week despite my conenctivity in my
> actually prefered address family would be all fine again after a
> couple of minutes?

That will depend. You might indeed get on a different address family for the 
next week, if they work comparatively the same.

>
> Next problem is that you are creating a shadow cache in each
> application on (hostname,port,P,timeout) which can easily get
> into the thousands in 10 minute sliding window of inactivity depending
> on the application.  That seems wrong to me but I guess people will fix
> it with a more intelligent purging strategy (depending on application) --
> say I close my browser tab for a site, purge the entry if it's the last
> tab with a connection to the host.  Too bad if it was my "home page".
> People will be creative once they hit an in-application specific "upper
> bound" of cache size despite memory being cheap.
>
>
> Another thing I am worried is that the experience in the non-"fast"
> networks in the non-ideal world will kill IPv6 more often than you
> wished, especially in the early transition time as it will be for
> most users.   The reason for that is:  while these days you often
> get a local A mirror close by for the larger sites, the AAAA machine(s)
> are more often further away (which can be in another state or country or
> even continent).  This will impose extra delay for IPv6
> "lookup+connection attempt" times possibly quickly degrading P to be
> IPv4 friendly.  It's unpredictable how the content will move to IPv6
> yet, as I see it but I am almost sure the world wide global "IPv6 day"
> turn on switch everywhere will not happen.  The preference will slowly
> change towards v6 but in the mean time you want to push it rather than
> slowing it down.
> Again, as said at the beginning, it will work well for the really broken
> cases but it will also affect the "10 ms difference" setups.
>

Okay. So, I am a user. Why do I (in a user hat) have to suffer a slower internet 
then ? Will I get a discount because of the worse experience ? Again - users are 
not buying IPv4 or IPv6. They buy the chance to do their other "stuff". We want 
to nudge the IPv6 a bit - but not get too overzealous in it.

This also has a property that if a dataplane for a particular AF is less of a 
capacity than the other one, the excess load will quietly "leak over" to the 
other AF. This is good for avoiding the short term disasters, is a pain from the 
troubleshooting side - but, the above "cries for help" should help on the edge 
ISP.

> So will it be a better user experience for me if I'll go through a CGN
> which may still is quicker than the native IPv6?

You said in one of the paragraphs above "if the transition technology faster 
than native, could as well use it" :-) - This is the contradiction with that. 
Besides the details of whether it is AFT or CGN - the result for the world is 
the same.

>
>
> One way to circumvent the aforementioned weekly flap would be to flush
> every cached record every n minutes (a time also long enough for a route
> flap to happen and propagate, or just an hour or a day or worst the
> TTL of the record returned the application will never learn).  It's
> something that you can spend a year of research on to properly tune that
> for most places around the globe.  Time noone has, which makes this bad.
>
>
> I said that a |P|==50 sounds too optimistic for me. What will be a good
> absolute value of P?  Should the be a static recommendation or not?
>

Can we have a dynamic value here ? What would it be ?
Something like a median time required to establish the X-percentile of TCP 
connections on the host can be a good value ?

>
> What about non-TCP?  The first mention of TCP is in 4.1 (ignoring the
> earlier diagram).  Should it equally apply to non-TCP or should TCP be
> mentioned already in the abstract?

The overall approach does not have to be TCP-specific. But, seems the TCP is the 
most active problem ?

>
>
> The very last thing is that the winning address you can connect to
> fastest does not have to be the address that gives you the best user
> experience as it could be the least loaded machine for other reasons:
> the application could hang there, ... the application could still have
> problems with connections on this address family, ...  Will that help
> user experience?  Yes it's another problem domain but "user
> experience" doesn't see the difference.

Agreed - this is not perfect. It's just the only approximation that is close 
enough to the network/transport layer to talk about. Should we have a knob for 
the L7 to tweak P ? To disable it ? Do the two GET /favicon.ico in case it is 
HTTP, over two AFs, to determine which of the two is better ?

Or try to measure the user's behaviour while browsing, and correlate that with 
single- or dualstack- nature of the servers ? (far stretch, but presumably there 
might be some correlation there). Could be - the connection setup time is by far 
not perfect. It's just that it was the only thing that looked tractable as 
opposed to "Bad connectivity ? too bad. your internet is off!" approach.

Again, thanks for the detailed feedback - these are certainly very useful 
discussions to have.

cheers,
andrew

>
>
> /bz
>
> -- 
> Bjoern A. Zeeb                                 You have to have visions!
>        <ks> Going to jail sucks -- <bz> All my daemons like it!
>  http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails.html
>