Subject: Re: [v6ops] PI [ULA draft revision #2 Regarding isolated networks]

From: Gert Doering <[email-masked]>
Date: Sun, 1 Jun 2014 12:24:08 +0200

Hi,

On Sun, Jun 01, 2014 at 08:23:21AM +1000, Mark Andrews wrote:
> > > But I have nothing to update my DNS zones. How do I reflect which links 
> > > are up or down? Is there even a draft for that? What's the BCP for TTL
> > > values, DNSSEC, etc?
> > 
> > This is where things get interesting.  You, Owen, I are not "the 99% home
> > users out there" - home users don't do DNS zones, because they do not 
> > control a DNS server...  (they do mDNS because it's automatic and works
[..]
> The IETF has published exactly one method for updating the DNS (RFC 2136).
> It has standardized several methods for securing that update.

While technically fully correct, this is completely missing the point :-)

 - how does the homenet CPE/host know *which* domain to update?
 - how does the homenet CPE/host know *how* to update this domain (end 
   users do not understand "entering keys in bind config on authoritative 
   name servers, and then copy these keys to their CPE")?
 - *when* should the CPE/host do the update, particularily, when one of the 
   ISP uplinks goes down, *should* it remove the records using that ISP's
   prefix?  If yes, after which time of non-availability?  (The ISP might
   never come back, it might be a move to a new ISP, after all)
 - if, on the same ISP, the prefix changes, how should the update look
   like?  Flash replace, staggered add/remove?

these are the interesting questions.  The mechanics of doing a zone update
via DNS protocol are interesting for DNS software implementors, but fairly 
uninteresting as far as the yet-unsolved questions surrounding a CPE in
a non-managed home or SME network go :-)

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444           USt-IdNr.: DE813185279

---

From: Ted Lemon <[email-masked]>
Date: Sun, 1 Jun 2014 12:51:01 -0400

On Jun 1, 2014, at 6:24 AM, Gert Doering <[email-masked]> wrote:
> [etc]

Daniel Migault has been working on this problem for a while, and there hasn't been a whole lot of working group participation and review.   I got that some people didn't like the solution Daniel proposed, but we need to work on this--it would be nice if people could try to come up with proposals rather than just pointing out the well-known issues that need to be addressed (although that was certainly a nice summary).

---

From: Ted Lemon <[email-masked]>
Date: Sun, 1 Jun 2014 12:54:31 -0400

On May 31, 2014, at 5:55 PM, Philip Homburg <[email-masked]> wrote:
> You suggest that every possible piece of software should implement HE?

Software that doesn't implement some kind of happy eyeballs mechanism is going to work poorly on an IPv6 internet.   Ask rather whether there could be some support for HE-like behavior that is relatively easy for app developers to use, so that they don't fail to use it and wind up making applications with crappy behaviors during partial outages.

---

From: Lorenzo Colitti <[email-masked]>
Date: Mon, 2 Jun 2014 02:36:50 +0900

On Mon, Jun 2, 2014 at 1:54 AM, Ted Lemon <[email-masked]> wrote:

> Software that doesn't implement some kind of happy eyeballs mechanism is
> going to work poorly on an IPv6 internet.

Because...?

---

From: Ted Lemon <[email-masked]>
Date: Sun, 1 Jun 2014 16:10:18 -0400

> Because...?

Because any situation where you are multi-homed, if either home fails and your software can't try both to see which one is working, there's a good chance it will fail to connect.

---

From: Philip Homburg <[email-masked]>
Date: Sun, 01 Jun 2014 22:24:36 +0200

In your letter dated Sun, 1 Jun 2014 16:10:18 -0400 you wrote:
>Because any situation where you are multi-homed, if either home fails 
>and your software can't try both to see which one is working, there's a 
>good chance it will fail to connect.

Nice how this thread evolves:
- Use PI space, it works (but is not really scalable)
- No, no, get with the program. Get multi-homed on PA.
- We don't actually have anything for servers and multi-homed PA.
- Every piece of software has to do HE because of how multi-homed PA fails.

Great way of advertising IPv6.

---

From: Ted Lemon <[email-masked]>
Date: Sun, 1 Jun 2014 16:31:52 -0400

> Great way of advertising IPv6.

Every piece of software that uses the network has to use DNS for ease-of-use, and has to use some kind of API to establish connections or send datagrams.   There is no reason why this API can't do happy eyeballs for the application without requiring any complexity _in_ the application.   Indeed, the API provided by Apple already provides this capability for free.

In comparison to what most apps have to do to work because of the higher-layer protocols they use (e.g., RESTful APIs have to support HTTP and TLS, streaming APIs have to support loss detection and data rate adaptation), doing happy eyeballs is the picture of simplicity.   The only reason it seems like it's not is that your mental model of a network API is (apparently) the BSD sockets API, which, while extremely useful, is pretty bare-bones.

---

From: joel jaeggli <[email-masked]>
Date: Sun, 01 Jun 2014 11:02:43 -0700

On 6/1/14, 9:54 AM, Ted Lemon wrote:
> Software that doesn't implement some kind of happy eyeballs mechanism
> is going to work poorly on an IPv6 internet.   Ask rather whether
> there could be some support for HE-like behavior that is relatively
> easy for app developers to use, so that they don't fail to use it and
> wind up making applications with crappy behaviors during partial
> outages.

Assumption of the Existence of HE  is a pretty dodgy proposition. if
you're an application and you support it great. but v6-only connected
hosts even with 4 translation probably don't bother or even know to try
for example.

> _______________________________________________ v6ops mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: Ted Lemon <[email-masked]>
Date: Sun, 1 Jun 2014 16:38:41 -0400

On Jun 1, 2014, at 2:02 PM, joel jaeggli <[email-masked]> wrote:
> Assumption of the Existence of HE  is a pretty dodgy proposition.

Then we can't do multihoming without NAT.

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 02 Jun 2014 08:45:22 +1200

> Every piece of software that uses the network has to use DNS for ease-of-use, and has to use some kind of API to establish connections or send datagrams.   There is no reason why this API can't do…

There's a lot I'm tempted to say about this, but I've already said it elsewhere:
http://www.sigcomm.org/ccr/papers/2014/April/0000000.0000008

N.B. PLEASE don't debate that article on this list.

   Brian

> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
> .
>

---

From: Gert Doering <[email-masked]>
Date: Sun, 1 Jun 2014 22:51:11 +0200

Hi,

> Then we can't do multihoming without NAT.

For multihoming *clients*, you don't need HE (or, specifically, HE doesn't
solve the issue of *source* address failover anyway in it's current form).

Server multihoming works without HE if you can afford to be patient,
like, you're an SMTP client and it doesn't matter whether the TCP session
connects in 10ms or 2 minutes, as long as it transmits the mail in question
(which, incidentially, HE doesn't guarantee - it will give you "that 
address gave back SYN-ACK first" but no guarantees that the TCP path 
actually works for packets with data in)

Gert Doering
        -- NetMaster

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 02 Jun 2014 10:23:40 +1200

> Then we can't do multihoming without NAT.

That's not what RFC 7157 says. Not even close.

   Brian

---

From: Mark Andrews <[email-masked]>
Date: Mon, 02 Jun 2014 09:11:40 +1000

> Great way of advertising IPv6.

Every piece of client software should already support multi-homing.
This is about supporting it *better* so there isn't the big delay
when switching to the second address because the link at the other
end is down.  The stack should be choosing source addresses which
have working outbound paths.  Selecting destination addresses is
up to the application.

This is about making the error cases work better which quite frankly
should have been done years ago for IPv4.  The only ones that don't
benefit from this is GLB developers.

Mark

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Lorenzo Colitti <[email-masked]>
Date: Mon, 2 Jun 2014 10:01:18 +0900

On Mon, Jun 2, 2014 at 5:10 AM, Ted Lemon <[email-masked]> wrote:

> Because any situation where you are multi-homed, if either home fails and
> your software can't try both to see which one is working, there's a good
> chance it will fail to connect.
>

And this problem is specific to IPv6 because...?

---

From: Ted Lemon <[email-masked]>
Date: Sun, 1 Jun 2014 21:06:16 -0400

> And this problem is specific to IPv6 because...? 

IPv4 never claimed to provide ubiquitous support for multihoming.   IPv6 kind of does.

---

From: Lorenzo Colitti <[email-masked]>
Date: Mon, 2 Jun 2014 10:12:53 +0900

> IPv4 never claimed to provide ubiquitous support for multihoming.   IPv6

There is no such claim. The only difference between IPv4 and IPv6 with
regard to multihoming is that IPv6 supports more than one IPv6 address on
the same interface, whereas some IPv4 implementations don't.

---

From: Ted Lemon <[email-masked]>
Date: Sun, 1 Jun 2014 21:19:05 -0400

On Jun 1, 2014, at 9:12 PM, Lorenzo Colitti <[email-masked]> wrote:
> There is no such claim. The only difference between IPv4 and IPv6 with regard to multihoming is that IPv6 supports more than one IPv6 address on the same interface, whereas some IPv4 implementations don't.

I would like multihoming to work, and I'd like it to work without NAT.   If you would like something else, that's okay.  But my impression is that I am not alone in thinking this is a key benefit of IPv6 over IPv4, even if it's still not fully baked (RFC 7157 notwithstanding).

---

From: Mark Andrews <[email-masked]>
Date: Mon, 02 Jun 2014 11:38:29 +1000

In message <[email-masked]>, Ted Lemon writes
> IPv4 never claimed to provide ubiquitous support for multihoming.   IPv6 kind

IPv6 adds multiple addresses on a interface.   Multihoming support is
transport family agnostic.

Multihoming support is supposed to be done in IPv4 applications
unless you have a very good reason to not support it or do you think
SHOULD means it is a optional part of IPv4?

RFC 1123
 
   2.3  Applications on Multihomed hosts

      When the remote host is multihomed, the name-to-address
      translation will return a list of alternative IP addresses.  As
      specified in Section 6.1.3.4, this list should be in order of
      decreasing preference.  Application protocol implementations
      SHOULD be prepared to try multiple addresses from the list until
      success is obtained.  More specific requirements for SMTP are
      given in Section 5.3.4.

      When the local host is multihomed, a UDP-based request/response
      application SHOULD send the response with an IP source address
      that is the same as the specific destination address of the UDP
      request datagram.  The "specific destination address" is defined
      in the "IP Addressing" section of the companion RFC [INTRO:1].

      Similarly, a server application that opens multiple TCP
      connections to the same client SHOULD use the same local IP
      address for all.

All dual stack machines are multihomed.  HE is basically about fast
failover when the destination addresses are unreachable.  This is not
hard to do.  From memory BSD 4.2 supported non blocking connect
which is all that is required from the IP stack to do fast failover.

Mark

---

From: Ted Lemon <[email-masked]>
Date: Sun, 1 Jun 2014 21:56:50 -0400

> SHOULD means it is a optional part of IPv4?

Your quote from RFC 1123 is focused on incoming connections, not outgoing connections.   This is a different problem than what I'm talking about.

---

From: joel jaeggli <[email-masked]>
Date: Sun, 01 Jun 2014 18:59:32 -0700

On 6/1/14, 6:19 PM, Ted Lemon wrote:
>> There is no such claim. The only difference between IPv4 and IPv6
>> with regard to multihoming is that IPv6 supports more than one IPv6
>> address on the same interface, whereas some IPv4 implementations
>> don't.
> 
> I would like multihoming to work, and I'd like it to work without
> NAT.   If you would like something else, that's okay.   But my
> impression is that I am not alone in thinking this is a key benefit
> of IPv6 over IPv4, even if it's still not fully baked (RFC 7157
> notwithstanding).

Ted I think you need rethink what you're claiming. for one rfc 6555
makes no claims about ipv6 address selection between multiple addresses.
it in fact sets aside the issue of having more than the two address
familes to chose from. I daresay if you have six v6 routes you probably
don't want to make six requests and see which one is fastest as a
general rule. you'remore likely to do ecmp if you simply treat all
nexthops as though they were equivalent, which you might not want to do
if they aren't (backup via LTE is probably not something you want to
exercise all the time even if it's equivlant speed to your cable
connection because it's more costly and has a datacap).

source address selection based on nexthop is pretty straight forward.
most unixes can do that. invalidating a nexthop when the route should no
longer be used is propertly the domain of routing.  This is not even a
benifit of ipv6 in the sense that it can work in ipv4 as anyone with an
appropriately configured router  can attest (it's what the multihomed
nat box actually does after all).

212.162.4.234
149.11.21.194
62.115.37.146

are all on same router for example.

---

From: Ted Lemon <[email-masked]>
Date: Sun, 1 Jun 2014 22:10:10 -0400

On Jun 1, 2014, at 9:59 PM, joel jaeggli <[email-masked]> wrote:
> general rule.

How is this different than doing happy eyeballs between v4 and v6?   Is it different because you're sextuply multi-homed?   I agree that that begins to look a bit problematic, but the more usual two-provider case is pretty much the same as regular happy eyeballs.

> you'remore likely to do ecmp if you simply treat all

Right, that's a problem.

> nat box actually does after all).

I've never operated one of these, but I'll take your word for it.   The fact that it works with IPv4 and NAT isn't very interesting to me.

Correct me if I am wrong, but what you have written appears to be a more detailed way of saying "happy eyeballs for multihoming won't happen, in my opinion, and multihoming for connection originators probably won't happen either, at least without NAT."   If so, that's a fine opinion to have, and may even be correct, but it's not an opinion I am willing to just accept without trying to do better (according to my opinion as to what's "better").

---

From: joel jaeggli <[email-masked]>
Date: Sun, 01 Jun 2014 19:28:12 -0700

On 6/1/14, 7:10 PM, Ted Lemon wrote:
>> Ted I think you need rethink what you're claiming. for one rfc
>> 6555 makes no claims about ipv6 address selection between multiple
>> addresses. it in fact sets aside the issue of having more than the
>> two address familes to chose from. I daresay if you have six v6
>> routes you probably don't want to make six requests and see which
>> one is fastest as a general rule.
> 
> How is this different than doing happy eyeballs between v4 and v6?
> Is it different because you're sextuply multi-homed?   I agree that
> that begins to look a bit problematic, but the more usual
> two-provider case is pretty much the same as regular happy eyeballs.
> 
>> you'remore likely to do ecmp if you simply treat all nexthops as
>> though they were equivalent, which you might not want to do if they
>> aren't (backup via LTE is probably not something you want to 
>> source address selection based on nexthop is pretty straight
>> forward. most unixes can do that. invalidating a nexthop when the
>> route should no longer be used is propertly the domain of routing.
>> This is not even a benifit of ipv6 in the sense that it can work in
>> ipv4 as anyone with an appropriately configured router  can attest
>> (it's what the multihomed nat box actually does after all).
> 
> I've never operated one of these, but I'll take your word for it.
> The fact that it works with IPv4 and NAT isn't very interesting to
> me.
> 
> Correct me if I am wrong, but what you have written appears to be a
> more detailed way of saying "happy eyeballs for multihoming won't
> happen, in my opinion, and multihoming for connection originators
> probably won't happen either, at least without NAT."

nope that's not what  I said.

>   If so, that's
> a fine opinion to have, and may even be correct, but it's not an
> opinion I am willing to just accept without trying to do better
> (according to my opinion as to what's "better").

So don't asssume that every application needs to be aware of and test
for which path is working, not only is that needless complexity pushed
up into the stack. let your routers do their job and cease routing when
their upstream connectivity is lacking.

---

From: Ted Lemon <[email-masked]>
Date: Sun, 1 Jun 2014 22:30:28 -0400

> their upstream connectivity is lacking.

Right, but this doesn't actually work.

---

From: Mark Andrews <[email-masked]>
Date: Mon, 02 Jun 2014 12:58:33 +1000

In message <[email-masked]>, Ted Lemon writes
> Right, but this doesn't actually work.

Setting the prefix's preferred lifetime to 0 in RAs when the uplink
goes and restoring it when the uplink works does work for non-broken
stacks.  Yes this is crude but so is advertising default.  Withdrawing
default is what routers should be doing when there isn't a working
default. 

There is also nothing to prevent a stack trying multiple source
addresses when it hasn't been explicitly bound to a source address
by the application.  The stack doesn't have to complete TCP handshakes
if it doesn't need to.

If all else fails the application layer is free to explictly try
different source addresses.  Hiding this detail in a library routine
is not hard.

---

From: Randy Bush <[email-masked]>
Date: Sun, 01 Jun 2014 20:19:43 -0700

> - Every piece of software has to do HE because of how multi-homed PA

you left out
  - discussion is completely irrelevant.  we're talking about INTRAnet

randy

---

From: Mark ZZZ Smith <[email-masked]>
Date: Sun, 1 Jun 2014 20:37:53 -0700 (PDT)

----- Original Message -----
> From: Mark Andrews <[email-masked]>
> To: Ted Lemon <[email-masked]>
> Cc: Philip Homburg <[email-masked]>; V6 Ops List <[email-masked]>
> Sent: Monday, 2 June 2014 11:38 AM
> Subject: Re: [v6ops] PI [ULA draft revision #2 Regarding isolated networks]
> 
> 

<snip>

> which is all that is required from the IP stack to do fast failover.

Fred wrote a very interesting draft on this, in particular including references to multihoming support/discussion earlier than RFC1123:

"Happier Eyeballs"
http://tools.ietf.org/html/draft-baker-happier-eyeballs-00

"Abstract

   Multihoming in modern networks has problems with differing quality of
   routes.  This note generalizes the concept of happy eyeballs across a
   variety of multihoming scenarios."

---

From: Gert Doering <[email-masked]>
Date: Mon, 2 Jun 2014 10:09:49 +0200

Hi,

> How is this different than doing happy eyeballs between v4 and v6?   Is it different because you're sextuply multi-homed?   I agree that that begins to look a bit problematic, but the more usual tw…

Actually, implementation-wise, there is a huge difference between

 source A       ---> talk to destination X, Y
 source A, B    ---> talk to destination Z

and it's time the IETF provides some answers how to handle the second case
(which HE doesn't talk about, and even ietf-mif-happy-eyeballs-extension
only addresses for the specific case where A and B are not on the same
interface).

6724 will not help very much in the second case - it will ensure that
"one of the GUAs" is used, but due to lack of further rules, will fall
through to the "most bits match" rule - which is only the "best" path
for some limited cases, like "talk to other hosts direclty in the A/B /32".

Worse, if you pick source A and that does not work because ISP A has 
an issue, no getaddrinfo() / HE implementation will fall back to try 
source B  (supposedly MacOS does, but I haven't verified that yet), as
the whole topic of source address probing / source address failover is
completely underspecified yet.  As in "it's complicated, leave us alone
with that stuff".

(But I'm sure we can do a lengthy discussion whether this could be solved
with RA or DHCPv6 options)

Gert Doering
        -- NetMaster

---

From: Gert Doering <[email-masked]>
Date: Mon, 2 Jun 2014 10:17:43 +0200

Hi,

> their upstream connectivity is lacking.

Well.  It might not have been your intention, but what you said is very
clear "we're not going to make multihoming in the homenet case work".

In the case

    host --> router --> ISP A ---->  ISP C --------+
	       +------> ISP B ---->  ISP D ------> ISP E --> target

where "host" has source addresses A and B, and there is a problem at 
ISP *C*, preventing connectivity via A->C->, how exactly is the homenet
router "router" supposed to *know* that, or even solve it?

Assuming "as long as connectivity to ISP A is there, the whole Internet
will be reachable via ISP A" is completely ignoring reality.  Blackhole
issues happen (where even BGP to the customer router won't help you)...

OTOH, proper source address failover on "host" to source address B would
very nicely solve this whole category of connectivity issues - and (by
enforcing halfway symmetric return traffic via ISP B) would actually solve
it *better* than BGP routing, which might need manual fiddling with the
router to remove "ISP C" from the path.

(IETF is complaining about lack of operator input.  Here is operator input.
Don't ignore it, otherwise operators get bored and find solutions outside
IETF space, like, with NAT)

Gert Doering
        -- NetMaster

---

From: Gert Doering <[email-masked]>
Date: Mon, 2 Jun 2014 10:18:53 +0200

Hi,

On Mon, Jun 02, 2014 at 12:58:33PM +1000, Mark Andrews wrote:
> by the application.  

Well, yes, nothing *prevents* this.  But given lack of IETF guidance on
this, stack implementors do not seem to be keenly interested in actually
implementing this.

(Does BIND do this...?)

Gert Doering
        -- NetMaster

---

From: Gert Doering <[email-masked]>
Date: Mon, 2 Jun 2014 10:30:46 +0200

Hi,

> http://tools.ietf.org/html/draft-baker-happier-eyeballs-00

Thanks.  I forgot about that one, and only found the mif draft, which
focuses on "multiple interfaces".

*This* draft is actually a very nice starting point, listing the issues
and providing some thoughts how to tackle them.

Gert Doering
        -- NetMaster

---

From: t.petch <[email-masked]>
Date: Mon, 2 Jun 2014 10:11:38 +0100

----- Original Message -----
From: "Owen DeLong" <[email-masked]>
To: "t.petch" <[email-masked]>
Cc: "Tore Anderson" <[email-masked]>; "Brian E Carpenter"
<[email-masked]>; "V6 Ops List" <[email-masked]>
Sent: Friday, May 30, 2014 6:18 PM
>
>> Yes, we need to change the fundamental way we deal with routing and
come
>> up
>> with a more scalable solution than the current everyone knows every
>> prefix
>> model.
>>
> <tp>
> Owen,
>
> yes, and that is what the RRG did.  It came up with two solutions and,
> being an IRTF and not an IETF WG, one solution was declared
appropriate
> while the other (LISP) is being developed by the IETF:-)

Not sure why your mailer can’t do quoting correctly, but I’ve fixed it
here
for you.

<tp>
Sorry about that, my mailer doesn't cope  with quoted-printable so if we
could if we could just eliminate all those funny European characters and
live with USASCII - sigh, perhaps not

Meanwhile ...

>
> The key to both is that while the number of prefixes is likely to grow
> beyond the capacity of routers, the number of locations involved is
> likely to remain much smaller so that routing will be based on
> locations, with a mapping from prefix to location prior to routing.

LISP is way over-complicated for the problem space in question, though
it’s
probably the right general idea.

In reality, what is needed is a simpler solution probably related to
doing
the IDR portion of routing on destination ASN. Unfortunately, we don’t
have
a good way in the current protocol to encode the destination ASN onto
the
packet. My thinking is that a revised 44 octet header would do the trick
and
that during the transition process, having routers that border between
44-octet header capable zones and non-44-octet header capable zones
would
perform the header replacement. Unfortunately, this has PMTU-D issues
which
I haven’t figured out how to address yet.

<tp>

... I commend RFC 6115 to you, although the I-Ds that led up to it are
perhaps better. And there is the RRG list archive (which makes v6ops
look like a low volume list:-).

A lot of options were explored, ILNP got the WG chair's recommendation,
LISP became an IETF WG; I liked SEAL (and am always glad to read what
Fred has to say) and I think that Ivip never got the attention it
deserved.

Between the IAB report in 2006 and the RFC in 2011, a lot of work went
into this area, with results as above.  Multi-homing, PI v. PA, PMTUD,
tunnels, NAT, IPv6 rate of migration, DFZ churn, the impact of 2B cell
phones on IPv6, network API, ... all of life is there and I suspect that
revisting the topic will bring little new (pace a revision of the
predicted chip capacity and hence a longer life for BGP4))

Tom Petch

Owen

>
> All in the RRG archives.
>
> Tom Petch
>
> However, at any likely growth rate, if we actually start working on
the
> problem
> instead of simply inflicting limitations and calling it handled, then
we
> have
> time to address the issue in IPv6. Hopefully the RRG will wake up and
> start
> addressing this issue. For now, it is out of scope for v6ops as I
> understand
> the WG charter.
>
>> And I think that every SME who has lost business with the
> unreliability
>> of their ISP will want multi-homing and will think that with IPv6 and
> PI
>> the constraints have gone, and the number of such SMEs can only
> approach
>> 10M over time.
>
> All the more reason this issue needs to get addressed instead of
> ignored.
>
>> So, Brian is spot on, and just as the IETF did little about IPv4
>> addresses running out until the event loomed large, so I expect
> history
>> to repeat itself with the growth of PI in IPv6.
>
> He’s somewhat right about the problem. He’s absolutely wrong in
> believing that
> the current limitations are a solution.
>
> Owen
>

---

From: Ted Lemon <[email-masked]>
Date: Mon, 2 Jun 2014 07:50:52 -0400

> source A, B    ---> talk to destination Z

Thank you for explaining that--I was talking about case 2, but I didn't make it explicit, and I think people thought I was talking about case 1.

The MIF happy eyeballs document will probably be updated according to the MPVD architecture document when that's done (hopefully soon).

---

From: Owen DeLong <[email-masked]>
Date: Mon, 2 Jun 2014 14:29:07 +0100

> On Jun 2, 2014, at 2:19 AM, Ted Lemon <[email-masked]> wrote:
> 

Multihoming works just fine without NAT as is. I run a network at home which is multihomed in both IPv4 and IPv6 without NAT.

Multihoming is not what is actually being discussed here. As Lorenzo pointed out this discussion seems to be limited to the (pathological) case of multihoming which involves a separate selection of source address for each upstream which is, for the most part, unique to IPv6 and does, indeed, pose additional challenges not present in traditional multihoming.

Owen

---

From: Gert Doering <[email-masked]>
Date: Mon, 2 Jun 2014 16:33:21 +0200

Hi,

> Multihoming is not what is actually being discussed here. As Lorenzo pointed out this discussion seems to be limited to the (pathological) case of multihoming which involves a separate selection of…

It would be really helpful if you could label your soapbox properly, and
if you talk about "BGP based multihoming with PI addresses", then please
*label* it as such.

The term "multihoming" does not imply more than "multihoming", in particular,
it does not require a specific technology or addressing policy to be used.

Gert Doering
        -- NetMaster

---

From: Mark Andrews <[email-masked]>
Date: Tue, 03 Jun 2014 01:08:03 +1000

> this, stack implementors do not seem to be keenly interested in actually

We don't automatically try multiple source addresses per family but
you can specify different sources addresses for different destination
addresses.

At this point multiple PA prefixes are not common yet.

ULA + PA (or PI) just works at the moment.

That said I would most probable implement it something like <D1,S1>,
<D2,S2>, <D3,S1>, <D1,S2>, <D2,S1>, <D3,S2> assume S1 and S2 are from
different PA prefixes.

On top of detecting network failures nameservers also have to deal
with broken nameservers / firewalls that drop queries because they
don't like something in the query despite it being perfectly in
spec.

Then there is RRL processing which drops queries if the server sees
excesive traffic as part of amplication mitigation.

---

From: joel jaeggli <[email-masked]>
Date: Mon, 02 Jun 2014 08:10:33 -0700

> clear "we're not going to make multihoming in the homenet case work".

No sorry I did not. What I said is that I believe you're pushing an
expectation of a certain amount of complexity onto hosts in the
interests of measurement and detection of topological corner cases. some
hosts might choose to do that in the interest of improved connectivity,
the expectation that they all should is onerous, you be should have an
expection that devices and applications will work in the absence of HE.

having a router invalidate itself as a nexthop because it has detected
that it's paritioned from a provider or the internet seems like an
incredibly basic concession that you'd want to have happen even were
everything HE enabled.

> router "router" supposed to *know* that, or even solve it?

It doesn't.

> issues happen (where even BGP to the customer router won't help you)...

yes, bgp multihoming doesn't solve it either. that has not rendered it
unsuitable for use.

---

From: Gert Doering <[email-masked]>
Date: Mon, 2 Jun 2014 17:26:42 +0200

Hi,

> incredibly basic concession that you'd want to have happen even were

You seem to be using a different Internet than I do, which seem to consist
of "the only way things can break is if my CPE loses contact to it's ISP,
which is always easily detectable, but as soon as the packet reaches the
ISP, it will be 100 percent reliably transported and answered, all the
time".

My Internet, unfortunately, seems to have less boolean cases where
packets just disappear when being sent into certain directions, without
even adjacent devices noticing, much less "devices 10 hops away".

> > router "router" supposed to *know* that, or even solve it?

Thanks for acknowledging that.

> yes, bgp multihoming doesn't solve it either. that has not rendered it

BGP multihoming has it's uses, but I think all but Owen agree that it's
not going to solve the "billions of leaf networks without IT knowledge"
case.

Multi-Address multihoming has the potential to be a radical improvement
for "leaf networks without IT knowledge".  

Especially for cases that BGP based multihoming will not handle well today, 
like "black hole problems due to people's MPLS stuff running amock", 
"stateful firewalls in the packet path that do not handle asymmetric 
traffic well", "selection of the ISP used by the user, not by the CPE 
router", etc.

It's certainly not a replacement for BGP based multihoming for "networks
with lots of servers and qualified IT staff".  Just to avoid that particular
avenue of ratholing.

> > IETF space, like, with NAT)

Maybe I should include this in of my signature when writing to IETF lists...

Gert Doering
        -- NetMaster

---

From: joel jaeggli <[email-masked]>
Date: Mon, 02 Jun 2014 09:12:05 -0700

> not going to solve the "billions of leaf networks without IT knowledge"

Right, nor are we suggesting it is. Lets say it, the attachment costs,
borne individually and collectively make it entirely unsuitable for this.

> for "leaf networks without IT knowledge".  

You and I agree on this.

Where I disagree is that the devices attached to network should be
assumed to bear the cost of fine grained decision making. We should not
assume that they all should, and multihomed homenets should function
reasonably well even if they do not.

---

From: Philip Homburg <[email-masked]>
Date: Mon, 02 Jun 2014 18:28:45 +0200

>benefit from this is GLB developers.

I'd like to see IPv6 deployed sooner rather than later. And in my experience,
everywhere where IPv6 is different from IPv4 it causes problems and extra
delays.

So if you say 'every piece of client software should already support
multi-homing' then that is simply not true in the IPv4 world. At least while
giving reasonable performance.

If all IPv4 software supported multi-homing properly, then there was no need 
for happy eyeballs. And many of the features that discussed in the context
of happy eyeballs were not implemented in a large scale before.

As a result, there also no standard API for doing HE. The Berkeley socket
interface is still the same because in practice, eveybody is not doing
multi-homed. 

Doing HE with minimal overhead is very tricky. It is very easy for most
client systems to take a full matrix of source and destination addresses and
connect to all of them. I'm quite sure that most operators of server systems
would completely hate that approach.

Simply put, the more things we come up with that have to be done before someone
can deplay IPv6, the longer it will take.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 2 Jun 2014 12:37:01 -0400

> Then we can't do multihoming without NAT.

This rather badly qualified comment appears to have generated a lot of discussion, so to head off further discussion I'd like to clarify a few points.

First, if you are multihomed, and both paths are working, then a host that does nothing special will work.   Second, if one connection goes out, and the routing updates, then an application that just retries when it doesn't get a connection will get a connection; for applications for which delays on the order of the time it takes for routing to update are harmless, redundant multihoming will work with no changes.

So the statement I made was false when interpreted in this context.   I know that it's false, and you don't need to keep arguing with me to convince me it's false.   I'm sorry for having been sufficiently unspecific that people thought I meant my statement was true for the above cases.

The cases I care about are in fact edge cases.   They don't happen all the time, and depending on the application they may or may not be important.   I care about them because I've been affected by failures, and I'd like to be able to say how to get these cases right without resorting to the use of NAT or NPT.   I'd like for there to be better tools for application developers to use so that they can address these edge cases relatively cheaply; possibly even more cheaply than they now do network programming that doesn't address these edge cases.   If you don't care about these edge cases, you will have little concern for this, and will not consider dealing with these edge cases important.

I, and no doubt most participants on this mailing list, am deeply uninterested in hearing "it works okay for me, so your edge cases don't matter"   I am also not interested in trying to get you to say "I care deeply about these edge cases."   It's okay with me if you don't care about them, and don't consider it important that they be addressed.   You don't need to respond to this message with yet another assertion that there is no problem here--it won't convince me, and what I want to do about this probably won't affect your application anyway.

If you are interested in thinking about this problem, the place to discuss it is probably the MIF working group--it's related to the work that's being done there, although it's not the entirety of that work.   If you are not interested, please let's just let this drop.   I'm sorry for the troll--I didn't intend it to be a troll, but I could have phrased it in a way that was less trollish.  I will try to do better next time.

---

From: Gert Doering <[email-masked]>
Date: Mon, 2 Jun 2014 18:41:45 +0200

Hi,

> can deplay IPv6, the longer it will take.

Strictly speaking, the bits of this discussion that turn around leaf-site
multihoming are not relevant to "being able to deply IPv6".  

They are relevant to "build a solution for millions of barber-shop style 
SMEs and pure home networks that like to have two ISPs because internet is
important", but this is not something happening *now* but "in the next
few years".

What we have now works about as well as IPv4 - but we can make it better.

Gert Doering
        -- NetMaster

---

From: Philip Homburg <[email-masked]>
Date: Mon, 02 Jun 2014 18:49:13 +0200

In your letter dated Mon, 2 Jun 2014 10:17:43 +0200 you wrote:
>where "host" has source addresses A and B, and there is a problem at=20

One philosophical question is whether it is the host or network that is
responsible for dealing with such brokenness?

The nice thing about the IPv4 architecture is the split between the network
and the transport layer on the hosts: the network transports packets from
source to destination, occasionally dropping packets, reordering them and 
in extreme cases duplicating them.

The host transport layer retransmits and reorders packets and filters out
duplicates. Host do not know about the network, they just pass packets to 
the default router (or other routers in case of ICMP redirects).

Making the host resposible for good source address selection in case network
failures violates that model. 

If we make the router responsible dealing with failures, then it helps to 
separate mechanism from policy.

I think it would already help enormously if routers could inform hosts in a 
standard way what prefixes do and do not work. One option is to play
tricks with the preference lifetime RAs. But how does that work in the
context of DHCPv6? How do we update DNS? 

For policy, even if you would have to tell a router manually that link is 
broken, it would help already enormously if entire site would then reconfigure
automatically. 

Detecting a broken uplink can be added easily and is a very common cause of
failure.

Beyond that, many different types of pluggable detectors are possible.
If for me, say, github is important, then having a module that can poll a
website regularly would do the trick.

Beyond that having an SNMP interface would allow a separate measurement device
to inform routers about what has failed.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 2 Jun 2014 12:52:17 -0400

> responsible for dealing with such brokenness?

It doesn't have to be one or the other.   The network can take care of what it can take care of, and that's important.   But the host can do things the network can't. Thinking of this as a big philosophical question that we have to settle once and for all is a recipe for endless debate.

---

From: Gert Doering <[email-masked]>
Date: Mon, 2 Jun 2014 18:57:31 +0200

Hi,

> > for "leaf networks without IT knowledge".  

This is good :-)

> reasonably well even if they do not.

OK, I do also agree (as this is a "should be assumed").

There's multiple aspects to it.

 - if the network devices do not care, and just follow 6724 rules, and
   the CPEs will handle "basic" failures right (detect ISP down, assign
   prefix with preferred lifetime = 0), they will generally just work,
   but the source address = ISP selection will not generally be optimal
   (whatever that means in the specific case - latency, throughput, 
   price?), and certain failure cases might involve user action ("plug
   the pull to ISP A").

 - if the network devices *do* care, and we get a bit more work done, the
   user experience can be improved
 
    - give the user (!) control over ISP selection, like in "I want my
      bittorrent to use my cable ISP, and my web surfing to use the DSL
      line" - I think this is a major benefit compared to solutions with
      a single prefix and a CPE-controlled policy, because for the normal
      end user, the CPE is a magic box that is not touched or configured
      (Owen is not a normal end user).

    - handle black-holing failure cases that only affect the packet path
      for one of the available prefixes, but for the one that might be
      selected by 6724

to point out what is obvious to me, but might not be for someone reading
this:

 - this is NOT about "managed environments".  In an "enterprise" network
   that has IT staff and policies set by "management" regarding ISP usage,
   the enterprise admins will NOT want the end systems to control policy
   - so there, "single prefix, CPE controls policy" is a more adequate 
   solution.  Which might imply NATs, or proxies, etc. - enterprisey stuff,
   according to that specific network's needs.

 - and that implies that there is not "one" solution for multihomed leaf sites

thanks :)

Gert Doering
        -- NetMaster

---

From: Gert Doering <[email-masked]>
Date: Mon, 2 Jun 2014 19:49:04 +0200

Hi,

> responsible for dealing with such brokenness?

I'm not so much concerned about philosophy, I'm an engineer.

> failures violates that model. 

This theory is very nice.  Unfortunately, it's, uh, theory, and today's
Internet does not really know very much about this.  

It *does* know about budget ISPs that are too cheap to pay engineers to 
properly monitor their network, about de-coupled control and data planes 
that advertise reachability but sink the data packets, extremely overloaded
links between ISPs fighting (de-)peering wars, and other unfriendlyness.

> separate mechanism from policy.

In the failure scenario I described, there is no (reasonable) way a home 
router can know that "ISP C" has messed up their MPLS network (again!) 
and it's no use sending packets out ISP A to reach host Z.

Even if it knew, other targets might work perfectly well via ISP A, while
yet other parts are broken via ISP B.  So "withdraw a prefix as soon as
something is broken upstream" will usually give you "zero working prefixes",
given that something is always broken somewhere.

[..]
> broken, it would help already enormously if entire site would then reconfigure

Please understand that it's not "the link" that is broken, or even "ISP A"
- these cases are trivial, and reconfiguring the site is also quite trivial,
except - to some extent - for externally published DNS records.

[..]
> website regularly would do the trick.

Yeah.  I can easily see *end users* do that, or router vendors update their
gear two years after it has been solved to monitor the Next Big Thing on
the Internet.

The *host* knows where it wants to connect to, and that it has multiple
options in case one doesn't work.

Coming back to your example and extending it a bit: what do you do if 
github is broken via ISP A, while netflix is broken via ISP B?  If you 
can solve that with prefix announcements, I'm all ears.  Until then, I
maintain that "dual prefixes with source address failover *plus* a way
to present the prefix selection to the application(!) user" is a more
workable way in that scenario.

(There are a number of ways such a "magic CPE" could achieve the end result
with dirty tricks, like involving NPT66 to circumvent non-working paths, but
I can't see a way to make it work without resolving to those)

Gert Doering
        -- NetMaster

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 03 Jun 2014 08:42:55 +1200

On 02/06/2014 20:17, Gert Doering wrote:
...
> router to remove "ISP C" from the path.

I know that shim6 isn't popular around here, but if you actually
want to achieve this effect - for any transport protocol, and
any application protocol, unmodified - run linshim6 at both ends.

> IETF space, like, with NAT)

Not to pretend that shim6 doesn't have operational issues too
(like needing SADR and tolerant firewalls).

   Brian
> 
> Gert Doering
>         -- NetMaster
> 
> 
> ------------------------------------------------------------------------
>

---

From: Gert Doering <[email-masked]>
Date: Mon, 2 Jun 2014 22:47:30 +0200

Hi,

> any application protocol, unmodified - run linshim6 at both ends.

I've heard very good results about failover tests with shim6 (much faster
convergence than BGP).  

I'm not convinced we particularily *need* it, though, as - as far as I 
understood - shim6 will primarily serve to ensure session survivability, 
while in most scenarios, sessions are so shortlived that "oh, ISP broken,
use other one" will be a matter of clicking reload in the browser...

Or will it also take care of selective non-reachability at session setup
("something in the ISP A path to Z broken")?

Gert Doering
        -- NetMaster

---

From: Ted Lemon <[email-masked]>
Date: Mon, 2 Jun 2014 16:57:03 -0400

> use other one" will be a matter of clicking reload in the browser...

The long-lived connection I'd be most interested in would be a download or a video stream.   TBH, I don't know the state of the art on streaming and multiply homed devices, though.   I know that there are techniques that spread streams across flows, but I don't know if they spread flows across links without operator intervention on a dual-homed home network.

---

From: Mark ZZZ Smith <[email-masked]>
Date: Mon, 2 Jun 2014 14:27:23 -0700 (PDT)

----- Original Message -----
> From: Philip Homburg <[email-masked]>
> To: V6 Ops List <[email-masked]>
> Cc: 
> Sent: Tuesday, 3 June 2014 2:28 AM

So if IPv6 was no different to IPv4 ... it'd be IPv4. So naturally, no matter how similar or different to IPv4 IPv6 was made, there was always going to be additional complexity when deploying it in parallel with IPv4.

> giving reasonable performance.

I think that is actually an observation on how ubiquitous, reliable and reliably operated IPv4 is. The parallel HE approach clearly makes just as much sense for IPv4 instead of attempting each A RR IPv4 address sequentially after a timeout.

In some ways the it is surprising that the HE approach hasn't be necessary under IPv4. Google seem to consider that they need to facilitate IPv4 redundancy in this way, as a query for A records for google.com returns 16 different A records. I doubt they'd be doing that unless they had some evidence that it is useful to some portion of end-users. They are shuffling the addresses around on each query, however they could do that but still only return one.

Another way to think of the HE technique is that it is a variation of recovery from TCP SYN packet loss. Traditional recovery took place "horizontally", within a single TCP connection. HE recovery is taking place "vertically", across multiple different TCP connections (and possibly different layer 3 protocols.)

> would completely hate that approach.

So I think if you made them think about it, they'd probably change their mind. The servers exist to serve, and therefore anything that provides better service to the clients is certainly worth considering, if not implementing. They will consequently earn a reputation as being a reliable and high performing server operator.

They might probably also want to consider Multipath TCP at the same time, for the same reasons - better server/service performance and reliability. HE + MPTCP would provide more of both.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 03 Jun 2014 11:11:21 +1200

> ("something in the ISP A path to Z broken")?

No. One of the design features of shim6 is *not* to create any
overhead for short-lived sessions, which means that by definition
it only wakes up after the first packets of a session have flowed.

I think that there are some very interesting ideas in shim6, but
it could be that it needs some redesign at this point in history,
espicially given the experience and thinking around MPTCP
and Happy Eyeballs.

    Brian

---

From: Mark Andrews <[email-masked]>
Date: Tue, 03 Jun 2014 09:39:43 +1000

Border NAT (IPv4) and depreferencing a prefix + SADR (IPv6) are
equivalent in terms of failure reachability.  Depreferencing a
prefix + SADR (IPv6) just moves the source address selection from
the NAT to the host.  This is what most homes will get to because
there is unlikely to be more than link up/down on the upstream links.

If you add BGP then external reachability can be added to the source
address selection of border NAT.  But if you are up to running BGP
then PI is the way to go.

Extending HE to do multiple source addresses (one per prefix) within
a adddress family will give pretty much the same robustness as
running BGP at the cost of some embryonic connections if the initial
attempt doesn't work.

Mark

In message <[email-masked]>, Ted Lemon writes
:
> This rather badly qualified comment appears to have generated a lot of discus
> sion, so to head off further discussion I'd like to clarify a few points.
> 
> First, if you are multihomed, and both paths are working, then a host that do
> es nothing special will work.   Second, if one connection goes out, and the r
> outing updates, then an application that just retries when it doesn't get a c
> onnection will get a connection; for applications for which delays on the ord
> er of the time it takes for routing to update are harmless, redundant multiho
> ming will work with no changes.
> 
> So the statement I made was false when interpreted in this context.   I know 
> that it's false, and you don't need to keep arguing with me to convince me it
> 's false.   I'm sorry for having been sufficiently unspecific that people tho
> ught I meant my statement was true for the above cases.
> 
> The cases I care about are in fact edge cases.   They don't happen all the ti
> me, and depending on the application they may or may not be important.   I ca
> re about them because I've been affected by failures, and I'd like to be able
>  to say how to get these cases right without resorting to the use of NAT or N
> PT.   I'd like for there to be better tools for application developers to use
>  so that they can address these edge cases relatively cheaply; possibly even 
> more cheaply than they now do network programming that doesn't address these 
> edge cases.   If you don't care about these edge cases, you will have little 
> concern for this, and will not consider dealing with these edge cases importa
> nt.
> 
> I, and no doubt most participants on this mailing list, am deeply unintereste
> d in hearing "it works okay for me, so your edge cases don't matter"   I am a
> lso not interested in trying to get you to say "I care deeply about these edg
> e cases."   It's okay with me if you don't care about them, and don't conside
> r it important that they be addressed.   You don't need to respond to this me
> ssage with yet another assertion that there is no problem here--it won't conv
> ince me, and what I want to do about this probably won't affect your applicat
> ion anyway.
> 
> If you are interested in thinking about this problem, the place to discuss it
>  is probably the MIF working group--it's related to the work that's being don
> e there, although it's not the entirety of that work.   If you are not intere
> sted, please let's just let this drop.   I'm sorry for the troll--I didn't in
> tend it to be a troll, but I could have phrased it in a way that was less tro
> llish.  I will try to do better next time.
>

---

From: Owen DeLong <[email-masked]>
Date: Mon, 2 Jun 2014 21:51:24 -0700

> it does not require a specific technology or addressing policy to be used.

I didn’t say it did.

Owen

---

From: Olivier Bonaventure <[email-masked]>
Date: Tue, 03 Jun 2014 09:34:09 +0200

Gert, Brian,
>> any application protocol, unmodified - run linshim6 at both ends.

Having worked actively with both shim6 and Multipath TCP and supervised 
their implementation in the Linux kernel, I'm convinced that solving the 
problem in the transport layer is much better than in the network layer. 
Multipath TCP can deal with the case that you discuss and we'd be happy 
to perform tests with the Multipath TCP implementation in the Linux 
kernel (see http://www.multipath-tcp.org )

> I've heard very good results about failover tests with shim6 (much faster

shim6 would converge faster, but Multipath TCP is even better. It gives 
you many additional features that cannot be provided by shim6 alone. 
Multipath TCP gets continuous feedback about the quality of the Internet 
paths that it uses. It can detect failures and move traffic away from 
congested links. With a network-level solution like shim6 or LISP, one 
hides the changes in the path to the transport protocol. This is not a 
good idea because the congestion control scheme sends packets over paths 
that it cannot control.

> use other one" will be a matter of clicking reload in the browser...

There are many short sessions, but most of the traffic is transported in 
very long TCP flows. Furthermore, the deployent of SPDY will increase 
the lifetime of the TCP connections.

> ("something in the ISP A path to Z broken")?

Multipath TCP  could be tuned to do that. There are many use cases where 
Multipath TCP would provide lots of benefits for failover, traffic 
engineering, ... Feel free to bring this operator input to the mptcp 
mailing list

Olivier

-- 
INL, ICTEAM, UCLouvain, Belgium, http://inl.info.ucl.ac.be

---

From: Gert Doering <[email-masked]>
Date: Tue, 3 Jun 2014 15:05:22 +0200

Hi,

> running BGP at the cost of some embryonic connections if the initial

It will give you *better* robustness than BGP, because BGP can not send
your packets around dataplane failures where control plane still advertises
reachability.

I find this very important to point out, again and again :-)

Gert Doering
        -- NetMaster

---

From: Philip Homburg <[email-masked]>
Date: Tue, 03 Jun 2014 15:24:53 +0200

>I find this very important to point out, again and again :-)

The good news is that in a network that provides hosts with just one prefix
this feature should be harmless.

But I'm not holding my breath for getting any kind of wide spread
implementation of such a feature.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 04 Jun 2014 08:16:46 +1200

Olivier,

> kernel (see http://www.multipath-tcp.org )

But, of course, the great advantage of shim6, as you know, is that it
applies to all transport layers. MPTCP is very elegant but only works
for TCP applications. So we have a bit of a dilemma in this area.
It's not a v6ops issue though.

    Brian

---

From: Scott Brim <[email-masked]>
Date: Tue, 3 Jun 2014 16:24:32 -0400

> It's not a v6ops issue though.

And the disadvantage of shim6 is that it's gone, sorry Brian but it's
not coming back. In the meantime, while MPTCP itself has little
deployment, the framework in it and SCTP are strongly present in HTTP
2.0, RTCWEB, and web browser space in general, and there is refreshing
work being done on transport layer improvements. If and when
multipathing proves its usefulness (imho the jury is still out, eg.
when you have paths with highly different quality) there will be
several convenient paths for its deployment.

Scott

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 04 Jun 2014 08:40:48 +1200

> several convenient paths for its deployment.

My point is that there are useful learnings in shim6 as well as in MPTCP
and SCTP. However, it will be very sad if we have to re-solve these
problems in every application suite rather than solving them once
in the stack.

    Brian

---

From: Scott Brim <[email-masked]>
Date: Tue, 3 Jun 2014 17:27:17 -0400

> problems in every application suite rather than solving them once

Got it. Agree completely. But looking at the momentum, I'm betting on
higher layers. Content is driving the bus.

Scott

---

From: George Michaelson <[email-masked]>
Date: Wed, 4 Jun 2014 08:49:00 +1000

Its only a personal opinion, but I feel (based on what we see in V6
measurements that HE) has been a net overall mistake.

 * its not consistently implemented.
 * it collides with other mechanisms for address/family selection such as
the apple RTT estimation model
 * its inherently non-deterministic
 * its not under (much) end user control

I cannot believe that making it an across the board, or adding more
complexity is going to make it better.

-G

On Tue, Jun 3, 2014 at 11:24 PM, Philip Homburg <[email-masked]
> wrote:

---

From: Doug Barton <[email-masked]>
Date: Tue, 03 Jun 2014 22:57:53 -0700

> At this point multiple PA prefixes are not common yet.

It's actually fairly common in the US.

Doug

---

From: Olivier Bonaventure <[email-masked]>
Date: Wed, 04 Jun 2014 12:48:14 +0200

Scott,

> higher layers. Content is driving the bus.

I'd prefer transport than application. The problem with solving 
multihoming or other issues at the application layer is that this is 
complex and requires a lot of networking expertise. Some app developpers
have this expertise, but they are the minority. We should not expect 
each app developper to have to deal with multihoming, the stack should 
solve the issue. This is the same problem as for reliability. We've 
placed reliability in the transport layer so that applications can built 
upon it. There are niche applications that want to reinvent reliable 
delivery, but this is not the best utilisation of scarce developper time...

Olivier

---

From: Scott Brim <[email-masked]>
Date: Wed, 4 Jun 2014 06:57:19 -0400

On Wed, Jun 4, 2014 at 6:48 AM, Olivier Bonaventure
<[email-masked]> wrote:
> I'd prefer transport than application. The problem with solving multihoming
> or other issues at the application layer is that this is complex and
> requires a lot of networking expertise. Some app developpers
> have this expertise, but they are the minority. We should not expect each
> app developper to have to deal with multihoming, the stack should solve the
> issue. This is the same problem as for reliability. We've placed reliability
> in the transport layer so that applications can built upon it. There are
> niche applications that want to reinvent reliable delivery, but this is not
> the best utilisation of scarce developper time...
>
> Olivier

Olivier: yes but there is tremendous momentum in user space / apps.
... but we digress.

---

From: Lorenzo Colitti <[email-masked]>
Date: Thu, 5 Jun 2014 22:53:20 +0900

On Tue, Jun 3, 2014 at 6:27 AM, Mark ZZZ Smith <[email-masked]>
wrote:

> Google seem to consider that they need to facilitate IPv4 redundancy in
> this way, as a query for A records for google.com returns 16 different A
> records. I doubt they'd be doing that unless they had some evidence that it
> is useful to some portion of end-users. They are shuffling the addresses
> around on each query, however they could do that but still only return one.
>

AFAIK that is not done for fallback reasons.

---

From: Alexandru Petrescu <[email-masked]>
Date: Mon, 23 Jun 2014 16:56:35 +0200

Old email, sorry.

Le 28/05/2014 23:24, Mark ZZZ Smith a écrit :
[...]
> The use case I like to imagine is a home user streaming a video from
> their NAS to their TV over their internal network. That should use
> ULAs so that a failure of their Internet connection/Global addressing
> has no impact on watching their movie. If the movie failed because
> the Internet connection/global addressing failed, that user will call
> the ISP's helpdesk. But why should it fail! The home users internal
> network is fine, it is only external connectivity that has failed.

I hope this requirement is considered in the homenet architecture 
discussion.

Too often the in-home Internet-disconnected network can not work internally.

Alex

>
> People might argue that enterprise networks are different and they
> are - they're simpler! Enterprise networks have technical staff on
> hand that can troubleshoot networks and resolve faults. If you can
> make IPv6 work seamlessly for home networks and their non-technical
> "operators", you've solved the harder problem. Since enterprise
> networks also value the same things home networks do - seamless
> operation, robustness against failure, stable internal connectivity,
> you've solved most of the enterprise network problems too.
>
>> It doesn't conform to RFC 7084 either, which clearly
>
>> states that "prefix(es) (and ULA prefix if configured...)" must be
>> advertised.
>>
>
> The precursor to 7084 (6204) was in development at the time, but it
> wouldn't have helped because they were ignoring our feedback.
> (Another CPE vendor was much better and almost too keen - they were
> sending me new software to test within 24 hours after reporting an
> IPv6 issue.)
>
> Regards, Mark.
>

---

From: Alexandru Petrescu <[email-masked]>
Date: Mon, 23 Jun 2014 17:12:09 +0200

Le 02/06/2014 03:38, Mark Andrews a écrit :
> IPv6 adds multiple addresses on a interface.

Hmmm, I have recollections of discovering multiple IPv4 addresses per 
interface before there was an IPv6 stack.

Alex

    Multihoming support is

---

From: Tim Chown <[email-masked]>
Date: Mon, 23 Jun 2014 16:18:15 +0100

On 23 Jun 2014, at 15:56, Alexandru Petrescu <[email-masked]> wrote:

> I hope this requirement is considered in the homenet architecture discussion.

It is already covered, e.g. in 3.4.2.

Tim

---

From: Ted Lemon <[email-masked]>
Date: Mon, 23 Jun 2014 11:29:41 -0400

On Jun 23, 2014, at 11:12 AM, Alexandru Petrescu <[email-masked]> wrote:
> Hmmm, I have recollections of discovering multiple IPv4 addresses per interface before there was an IPv6 stack.

It's certainly possible, but you will never see an IPv4 host do it automatically.  It has to be configured manually, and rarely makes much sense other than for servers that simply need an additional public identifier, or routers that need to do network address translation and hence need an IP address on both the public and private subnets.

---

From: Alexandru Petrescu <[email-masked]>
Date: Mon, 23 Jun 2014 17:41:50 +0200

> It is already covered, e.g. in 3.4.2.

Thanks, it's already covered and makes sense.

However, I have one minor issue with it.

I do not understand why it requires an in-home Router to not advertise 
self as a default router whenever ULA prefixes are involved.

>    In cases where ULA prefixes are in use within a homenet but there is
>    no external IPv6 connectivity (and thus no GUAs in use),
>    recommendations ULA-5, L-3 and L-4 in RFC 6204 should be followed to
>    ensure correct operation, in particular where the homenet may be
>    dual-stack with IPv4 external connectivity.  The use of the Route
>    Information Option described in [RFC4191] provides a mechanism to
>    advertise such more-specific ULA routes.

(or I can't remember the earlier discussions).

It may be that this requirement will make the in-home Hosts not capable 
of talking to each other simply because not implementing RFC4191.  And, 
certainly RFC4191 is not for Router-to-Router communications.

The benefit of the doubt.

Alex

---

From: Tim Chown <[email-masked]>
Date: Mon, 23 Jun 2014 17:05:17 +0100

HI,

On 23 Jun 2014, at 16:41, Alexandru Petrescu <[email-masked]> wrote:

> I do not understand why it requires an in-home Router to not advertise self as a default router whenever ULA prefixes are involved.
> 
> It may be that this requirement will make the in-home Hosts not capable of talking to each other simply because not implementing RFC4191.  And, certainly RFC4191 is not for Router-to-Router communications.
> 

This topic was discussed a year or so ago in the context of the draft on ULA usage, e.g. see
http://www.ietf.org/mail-archive/web/v6ops/current/msg16405.html

It would be good to ensure this is captured in 
http://tools.ietf.org/html/draft-ietf-v6ops-ula-usage-recommendations-02

Tim

---

From: Randy Bush <[email-masked]>
Date: Tue, 24 Jun 2014 08:00:24 +0900

>> has no impact on watching their movie.

this is yet another red herring (which this wg seems to love).  if there
is no external connection, whatever address prefix the LAN is using will
still work.

randy

---

From: Mark Andrews <[email-masked]>
Date: Tue, 24 Jun 2014 09:11:32 +1000

> is no external connection, whatever address prefix the LAN is using will

But choosing the wrong one will potentially cause problems when the
net reconnects.  You can't safely keep using a PD assigned prefix
after the lease has expired across a reconnect event.

Note we have already agreed that there is no such thing as a
permanently disconnected network.

---

From: "Liubing (Leo)" <[email-masked]>
Date: Tue, 24 Jun 2014 02:51:48 +0000

Hi Tim,

[Bing] The default route had been mentioned in Section 3.2 and Section 3.3. But we didn't mentioned RFC4191 relevant consideration, I think it would be good to add some text.
Thanks for your remind.

Best regards,
Bing

Tim

Alex

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 24 Jun 2014 11:10:16 +0100

On 23/06/2014 16:29, Ted Lemon wrote:
> It's certainly possible, but you will never see an IPv4 host do it
> automatically.  It has to be configured manually, and rarely makes much
> sense other than for servers that simply need an additional public
> identifier, or routers that need to do network address translation and
> hence need an IP address on both the public and private subnets.

...or any of the situations where it might make sense in the same way that
it might make sense for ipv6, which is why all major operating systems have
GUI access to make it easy to add secondary ipv4 addresses on any
interface.  Secondary ipv4 addresses are completely mainstream, even if we
all used them before they were cool.

Nick

---

From: Mark Andrews <[email-masked]>
Date: Tue, 24 Jun 2014 20:19:57 +1000

> all used them before they were cool.

Yet there is equipment on sale as I write that doesn't support them.  It is
not a feature you can depend upon being present in *all* the equipment
connected to the network.

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 24 Jun 2014 11:38:49 +0100

On 24/06/2014 11:19, Mark Andrews wrote:
> Yet there is equipment on sale as I write that doesn't support them.

Mark, you quoted a single Brother HL-4040CN which ok, I'm sure it doesn't
have a gui option to support multiple ipv4 addresses, but I'd also suspect
that it uses a linux kernel underneath the glitz and that this is purely a
UI issue.

Either way, Mark Smith's original point:

> RFC1918s have provided that internal connectivity robustness to both home
> networks and enterprise networks. Of course the drawback is that in IPv4 it
> is binary - hosts either have RFC1918s or public addresses, so if you have
> RFC1918s you have to use NAT to access external destinations on the
> Internet.

... is demonstrably incorrect on a matter of fact, and can we just accept
that supporting multiple IPv4 addresses on the same interface is normal,
mainstream and widely supported, including on consumer grade hardware?  And
then can we move on?  thanks,

Nick

---

From: [email-masked]
Date: Tue, 24 Jun 2014 13:27:14 +0200 (CEST)

> all used them before they were cool.

Absolutely agreed. We tend to put each service on its own IP address -
makes it much easier to move the service later without affecting other
services on the same server.

Steinar Haug, AS 2116

---

From: Ted Lemon <[email-masked]>
Date: Tue, 24 Jun 2014 09:14:38 -0400

> all used them before they were cool.

So e.g. you would use a secondary IPv4 address on a host to do multihoming?   And you would argue that this works sufficiently well that we should treat it as commonplace?

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 24 Jun 2014 15:23:20 +0100

On 24/06/2014 14:14, Ted Lemon wrote:
> So e.g. you would use a secondary IPv4 address on a host to do
> multihoming?

your question is badly specified because you don't say what you mean by
"multihoming" and it's not obvious from the context.

If you're referring to local connectivity on the same l2 domain where there
are multiple prefixes used on that lan, then adding multiple ipv4 addresses
to the same interface works without problems and has always done so.

If you're referring to something else which involves routing the traffic to
a different network by selecting a next-hop gateway depending on the host
address used, then that works as well as in ipv6, which is to say not at all.

Nick

---

From: Ted Lemon <[email-masked]>
Date: Tue, 24 Jun 2014 11:55:37 -0400

> then can we move on?  thanks,

Nick, it's certainly "supported," but it's not configured automatically (nor is there a standard mechanism for doing so), and it doesn't support multihoming or intelligent source address selection.

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 24 Jun 2014 17:40:48 +0100

On 24/06/2014 16:55, Ted Lemon wrote:
> Nick, it's certainly "supported," but it's not configured automatically
> (nor is there a standard mechanism for doing so), and it doesn't support
> multihoming or intelligent source address selection.

you still haven't defined what you mean by multihoming in this context.

ipv4 source address selection is handled by vendor implementations and the
ietf has no particular guidelines on how it should be managed.  Possibly
there is a correlation between these two things.

Nick

---

From: Ted Lemon <[email-masked]>
Date: Tue, 24 Jun 2014 13:38:05 -0400

> address used, then that works as well as in ipv6, which is to say not at all.

This isn't really true.   If your routing is set up right and source address selection is configured correctly, you can do multihoming.   It's not clear that every device supports it, and it's not particularly easy to make it work at the moment, but it is _certainly_ possible to make it work.   The same is not true of IPv4, and it's unlikely we (the IETF) will do the work to fix that, because nobody (that I know of) cares.

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 24 Jun 2014 19:49:46 +0100

On 24/06/2014 18:38, Ted Lemon wrote:
> This isn't really true.   If your routing is set up right and source
> address selection is configured correctly, you can do multihoming.

I'm trying to read between the lines here and am going to assume that
you're talking about end hosts with multiple addresses being contactable
over different links from other networks where next-hop interface selection
based on destination addresses (which is defined for ipv4) is used.

> It's not clear that every device supports it, and it's not particularly
> easy to make it work at the moment, but it is _certainly_ possible to
> make it work.

Most people need protocols which just work rather than protocols which
we're assured that it's possible that they can be made to work, assuming
that there is device support and that the routing is set up correctly.  And
that address selection is configured correctly.  And that you're doing
one-legged cartwheels on nights where there is both a full moon and
planetary alignment and you're waving dead chickens in the air.  I've spend
enough of my career dealing with that level of brokenness and it's not
worth it by any widely-accepted metric: financial / technical / etc.

> The same is not true of IPv4, and it's unlikely we (the
> IETF) will do the work to fix that, because nobody (that I know of)
> cares.

indeed because it's not necessary.  People who need reliable multihoming
for either ipv4 or ipv6 will acquire an address assignment or allocation
and will use that.  It just works, and the people who do it don't much mind
the small amount of global blood-letting it causes.

Nick

---

From: Gert Doering <[email-masked]>
Date: Tue, 24 Jun 2014 21:46:38 +0200

Hi,

> the small amount of global blood-letting it causes.

Mainly because some people keep repeating that BGP+PI is the only available
option.  Like, religiously.  You're not *that* old yet.

Gert Doering
        -- NetMaster

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 24 Jun 2014 20:49:23 +0100

> option.  Like, religiously.  You're not *that* old yet.

...then point me to another option which works reliably.

Nick

---

From: Gert Doering <[email-masked]>
Date: Tue, 24 Jun 2014 21:57:20 +0200

Hi,

> ...then point me to another option which works reliably.

"reliably" in itself is already a red herring.  Protecting you against
*what*, and under which assumptions?

For "I want my facebook and youtube and e-mail to work, even if one of 
my ISPs goes belly-up!", homenet + dual-prefix works.  It will not give
you perfectness just yet, aka "source address selection will not 
pick the best source IP for whatever definition of 'best'", but MIF is
working on optimizing this.  Your sessions will die if you are actively 
using the ISP that just died on you, but mp-tcp is one *demonstrably
working* way to handle that, get much faster session failover that BGP 
would give you (seconds, not minutes).

For "I need to run a datacenter full of stuff that need to have stable
addresses, no matter what happens elsewhere", I give you that BGP+PI
(or whatever other source of "your addresses") is a much better and
more reliable approach.  But for a barber shop that just wants their
internet radio to be there all the time, it's completely wrong.

Short form.  Longer discussion of the options and missing bits upstream
in this very thread.

Gert Doering
        -- NetMaster

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 24 Jun 2014 21:43:31 +0100

> would give you (seconds, not minutes).

this kinda gets back to issue of what we mean when we talk about
multihoming, which is why I poked Ted multiple times to try to get him to
define what he meant when he used the term.  At the moment it seems like
we're all talking about different things, and this is leading to some of
the confusion.

So I ask again: can we please define what we mean by multihoming in the
context of this discussion?  Until we have clarified what we're talking
about, the discussion is not going to lead anywhere.

Nick

---

From: Ted Lemon <[email-masked]>
Date: Tue, 24 Jun 2014 16:51:29 -0400

> about, the discussion is not going to lead anywhere.

I shut up about this because as far as I can tell, the question of multihoming has no bearing on this discussion.   That is, I'm fairly sure that even if I explained what I meant by multihoming, it would still not lead us anywhere useful.

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 24 Jun 2014 22:01:31 +0100

> I shut up about this because as far as I can tell, the question of multihoming has no bearing on this discussion.   That is, I'm fairly sure that even if I explained what I meant by multihoming, it…

you're right, it's off-topic.  Let's kill this ratholing exercise.

Nick

---

From: John Mann <[email-masked]>
Date: Wed, 25 Jun 2014 14:06:28 +1000

Hi,

On 28 May 2014 18:16, Owen DeLong <[email-masked]> wrote:

>
> On May 27, 2014, at 3:17 PM, Mark Andrews <[email-masked]> wrote:
>
> >
> > In message <[email-masked]>, Philip Homburg
> writes:
> >> In your letter dated Tue, 27 May 2014 08:52:02 -0400 you wrote:
> >>> The operational situation that's problematic is the large enterprise
> >>> scenario, where you have two large enterprises with their own ULAs that
> >>> merge.   If those ULAs happen to clash, you have to renumber at least
> >>> one of them.   If they don't clash, you still have to deal with routing
> >>> them (although I think the split-horizon complexity objection Mikael
> >>> raised ought to be thought through carefully before being asserted as
> >>> factual, because I think it can be addressed through routing and not
> >>> naming).
> >>
> >> If you are a large entrprise, just spend the 50 euro or so (RIPE
> service region) it
> >> costs to get your own prefix.
> >
> > A /56 is over AUD1180 (+10% GST) annually.  Thanks for playing.
> >
>
> Yes, APNIC now has the distinction of being the absolutely most expensive
> RIR on the planet.
>
> However, outside of the APNIC region, prices are much more reasonable.
>
> US 100 per resource (regardless of size) ARIN
> EU  50 (flat rate, regardless of resources) RIPE
> US 600 (up to a /35) LACNIC
> US 100 (per /48?) AfriNIC
>

> AU1180 (/56, logarithmic formula for larger blocks) APNIC

I think you are describing the APNIC costs poorly.

It is  AUD 1180 (up to a /34, logarithmic formula for larger blocks) APNIC

Pricing one /56

http://submit.apnic.net/cgi-bin/feecalc.pl?ipv4=&ipv6=%2F56&action=Calculate
says "= AUD 4", but the minimum annual fee is AUD 1,180  so that is the
actual fee.

Similarly, an IPv6 /48 says "= AUD 30", but the minimum annual fee is AUD
1,180  so that is the actual fee.

But a realistic situation for a SME with IPv4 and IPv6:
- an IPv4 /24 and an IPv6 /34

http://submit.apnic.net/cgi-bin/feecalc.pl?ipv4=%2F24&ipv6=%2F34&action=Calculate
Fees for each address type is AUD 1,180 ; take the maximum so the fee is
AUD 1,180

And a small ISP:
- an IPv4 /22 and/or an IPv6 /32 costs AUD 1,994

 So, it looks like at a little less than twice your next closest
> competitor, APNIC is the clear winner for the highest prices.
>

   John

---

From: Owen DeLong <[email-masked]>
Date: Wed, 25 Jun 2014 12:46:07 -0700

On Jun 24, 2014, at 21:06 , John Mann <[email-masked]> wrote:

> > In message <[email-masked]>, Philip Homburg writes:
> >> If you are a large entrprise, just spend the 50 euro or so (RIPE service region) it
> Yes, APNIC now has the distinction of being the absolutely most expensive RIR on the planet.
> 
> says "= AUD 4", but the minimum annual fee is AUD 1,180  so that is the actual fee.
> 
> Similarly, an IPv6 /48 says "= AUD 30", but the minimum annual fee is AUD 1,180  so that is the actual fee.
> 
> Fees for each address type is AUD 1,180 ; take the maximum so the fee is AUD 1,180
> 
> And a small ISP:
> So, it looks like at a little less than twice your next closest competitor, APNIC is the clear winner for the highest prices.
> 
>    John

None of this contradicts my statement that APNIC is at least twice as expensive as anyone else. (OK, not quite double LACNIC, but really close).

Owen

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Wed, 25 Jun 2014 23:51:04 +0000

> Software that doesn't implement some kind of happy eyeballs mechanism is going to work poorly on an IPv6 internet.   Ask rather whether there could be some support for HE-like behavior that is rela…

For the record, HE isn’t actually about IPv6, although the document is written that way. It’s about multihoming. If I have a multihomed device with two IPv4 addresses and one of them is intermittent for some reason, I’ll have exactly the same problem and should have exactly the same solution. Or ditto if I have two IPv6 addresses.

I actually have that problem right now. I have an HE tunnel to the residential side of my home and IPv4+IPv6 to my home office courtesy of Cisco. A few weeks ago, without notice, the HE tunnel stopped working. Not sure why, but I have seen comments from others to the effect. So, on the corporate side, I can ping or ping6 www.kame.net, but on the residential network I can only ping it, not ping6. Imagine IPv4 not working at all (that day will come, but not today). I’d like to be able to play happy eyeballs between my two IPv6 addresses.

<pull Ethernet plug, turn on wifi, wait a few seconds>

[FRED-M-20VW:~/Desktop] fred% ping6 -c 5 www.kame.net;ping -c 5 www.kame.net
PING6(56=40+8+8 bytes) 2001:1938:290:2:948e:3370:b5c6:74ff --> 2001:200:dff:fff1:216:3eff:feb1:44d7

--- orange.kame.net ping6 statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

PING orange.kame.net (203.178.141.194): 56 data bytes
64 bytes from 203.178.141.194: icmp_seq=0 ttl=49 time=124.782 ms
64 bytes from 203.178.141.194: icmp_seq=1 ttl=49 time=124.827 ms
64 bytes from 203.178.141.194: icmp_seq=2 ttl=49 time=126.614 ms
64 bytes from 203.178.141.194: icmp_seq=3 ttl=49 time=116.715 ms
64 bytes from 203.178.141.194: icmp_seq=4 ttl=49 time=128.197 ms

--- orange.kame.net ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss

<disable wifi, plug in Ethernet, wait a few seconds>

round-trip min/avg/max/stddev = 116.715/124.227/128.197/3.964 ms
[FRED-M-20VW:~/Desktop] fred% !p
ping6 -c 5 www.kame.net ; ping -c 5 www.kame.net
PING6(56=40+8+8 bytes) 2001:420:701:32:9cec:98d1:3718:ce3c --> 2001:200:dff:fff1:216:3eff:feb1:44d7
16 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7, icmp_seq=0 hlim=51 time=173.079 ms
16 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7, icmp_seq=1 hlim=51 time=172.896 ms
16 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7, icmp_seq=2 hlim=51 time=187.208 ms
16 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7, icmp_seq=3 hlim=51 time=171.177 ms
16 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7, icmp_seq=4 hlim=51 time=176.913 ms

--- orange.kame.net ping6 statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 171.177/176.255/187.208/5.789 ms
PING orange.kame.net (203.178.141.194): 56 data bytes
64 bytes from 203.178.141.194: icmp_seq=0 ttl=39 time=177.976 ms
64 bytes from 203.178.141.194: icmp_seq=1 ttl=39 time=180.668 ms
64 bytes from 203.178.141.194: icmp_seq=2 ttl=39 time=181.578 ms
64 bytes from 203.178.141.194: icmp_seq=3 ttl=39 time=183.519 ms
64 bytes from 203.178.141.194: icmp_seq=4 ttl=39 time=176.404 ms

--- orange.kame.net ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 176.404/180.029/183.519/2.543 ms

---

From: Ted Lemon <[email-masked]>
Date: Wed, 25 Jun 2014 21:57:40 -0400

> For the record, HE isn’t actually about IPv6, although the document is written that way. It’s about multihoming. If I have a multihomed device with two IPv4 addresses and one of them is intermitten…

Yup.   Hence the MIF happy eyeballs draft...