Subject: [RRG] Elegance and the rejection of SHIM6 host-based multihoming

From: Robin Whittle <[email-masked]>
Date: Wed, 17 Sep 2008 16:40:12 +1000

Hi Iljitsch,

There are several messages I want to respond to, and will next week,
but I wanted to first challenge something you wrote in "Re: [RRG]
Renumbering...":

> Stephen Sprunk wrote:
>
>>> The users leaned on the RIRs to do PI, and any attempt to get
>>> rid of PI would take even more ooomph than stopping them, and
>>> there clearly wasn't enough for that.
>>
>> Speaking as one of "them", the reason we proposed (and accepted)
>> IPv6 was that the IETF had provided no acceptable (to us)
>> alternative.
>
> That is nonsense. The shim6 effort was well under way at that
> point but not yet mature enough that it was possible to know
> whether it would solve the problem.

The functional goals of SHIM6 are well known and have been for a
while: to provide host-based multihoming.

End-user networks generally don't want host-based multihoming - they
want network based multihoming so an unknown number of hosts in
their network keep operating without a glitch when one link to an
ISP goes down.  They also need a solution which works for today's
Internet - IPv4.  I guess they require any changes to their
operations not to involve host changes or hosts being involved with
network functions such as routing and forwarding.

Also, people want address space which is portable between ISPs - so
they don't have to renumber their network, DNS etc. when selecting a
new ISP.

I believe these are perfectly reasonable desires and needs.  SHIM6
and IPv6's automated router and host renumbering just doesn't do
what most end-user networks need.

> So even if you are of the opinion that shim6 isn't an acceptable
> solution, at the point the decision was made this was still
> unknowable and thus the decision was wrong.

The limits of the usefulness of SHIM6 and IPv6's automated host
renumbering system are and were well known, so I think it is wrong
to state that people making a decision based on their needs and
SHIM6's limits made the wrong decision.

> (And the fact that this decision is made by random people in one
> part of the world rather than by the IETF or some other world wide
> organization with at least _some_ technical props shows how broken
> the RIR policy development system is.)

I guess that if the IETF was working on some promising techniques
which would have provided whatever it is that people need when they
need portability and network based multihoming - but with PA space -
then perhaps folks would have waited before approving the release of
PI space to end-users.

The IETF had and still has no such plans.  When people say they need
to keep their address space, either when selecting a new ISP and/or
for multihoming, it is a perfectly reasonable thing for a network to
need.   This can't be achieved with PI space.

LISP, APT, Ivip, TRRP and Six/One Router are attempts at providing
end-user networks with what they want and need: portability and
network-based multihoming with PI space, but in a scalable way.

I think it would be an unjustifiable over-reaction to require that
all hosts change their stacks and applications just to solve a
scaling problem in the DFZ, which is what ILNP and I guess some
other proposals involve.  Perhaps some of those proposals would
result in a cleaner, more elegant, Internet structure than those
which involve no host changes.

Most people, including me, are not prepared to change everything to
achieve something which optimised "elegance".  We just want the
Internet to keep working well with minimal disruption.

Elegance in English would involve a radical revamp of the language.

Elegance in driving would involve all countries agreeing to either
drive on the left or on the right.

Elegance in Western music notation would involve changing the crazy
C, C#/Df, D, D#/Ef, E, F, F#/Gf, G, G#/Af, A, A#/Bf, B nomenclature.
  (Backwards compatible with some pieces of music written 400 years
ago . . . )  Likewise the horrible layout of piano and organ
keyboards (also backwards compatible with 17th century organs) - a
different chord and melodic fingering pattern for each of the 12
transpositions - would be replaced by something like a 5 row button
accordion system, where a single chord or melody pattern is used no
matter what the transposition.

Elegance in keyboards would involve replacing the QWERTY keyboard
with something better.

(The driving, piano keyboard and computer keyboard questions are
also major questions of health and safety, with the former problem
causing death and injury and the latter two contributing to RSI.)

I wish all these things would happen, and I wish the Internet had
full end-to-end connectivity in a manner that a simple routing
system could scale indefinitely to support portability, multihoming
and mobility for billions of end-user networks.

However, since there are solutions which can provide scalable
portability, multihoming and mobility without host changes - and
since an elegant redesign would only be useful to most end-users
after the prohibitive hurdle of everyone changing their OSes and and
applications, and connecting to the new network - I don't support
such a radical revision of the Internet.  Even if I thought a
radical revision was a good idea, there's no way we could either
impose it on the world or persuade sufficient people to willingly
adopt it (unless we could somehow make it seamlessly interwork with
the current Internet, current applications etc.).

  - Robin

--
to unsubscribe send a message to [email-masked] with the
word 'unsubscribe' in a single line as the message text body.
archive: <http://psg.com/lists/rrg/> & ftp://psg.com/pub/lists/rrg

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 17 Sep 2008 10:19:22 +0200

On 17 sep 2008, at 8:40, Robin Whittle wrote:

>> The shim6 effort was well under way at that

> while: to provide host-based multihoming.

Well if all your hosts are multihomed your entire network is  
multihomed. End-users shouldn't concern themselves too much with the  
details of technology that aren't of immediate importance to them.

> They also need a solution which works for today's Internet - IPv4.

Which is orthogonal to the discussion at hand: PI in IPv6.

> they don't have to renumber their network, DNS etc. when selecting a

Shim6 could have given them that and maybe still could in the future  
by using a separate ID space.

> what most end-user networks need.

No, it doesn't do what they want. Like trained monkeys these users  
expect to be able to continue to do the same thing forever even though  
circumstances are changing. (Actually that's unfair to monkeys, humans  
and flies are pretty much the only creatures that keep doing the same  
thing expecting different results.)

> network-based multihoming with PI space, but in a scalable way.

At a high price of added complexity.

One or two of these solutions may be turned into something workable  
but that's not to say that that would be the best possible solution.

Mark my words: a loc/id split is going to open a big can of worms that  
will make life harder for a long time. Loc/id overload isn't so bad.

> scaling problem in the DFZ

There's no need to update applications for shim6 (IPv6 applications  
are already expected to try all addresses, that's all that shim6 needs).

There is ample precedent for this, like IPv6. Or if you prefer  
something that's in wider use: TCP congestion control. Hosts need to  
keep state anyway so solving the problem there is essentially free. It  
would be stupid to ignore that.

Host stack changes take a long time, but apart from that, they're  
actually a fairly easy way to solve problems because only a handful of  
organizations need to make investments (the host OS vendors).

> We just want the

Then switch to IPv6 before we run out of IPv4 addresses...

Oh wait, what you really want is to not have to deal with the problem.  
Fine. But then you pay the price in some other way.

> Elegance in English would involve a radical revamp of the language.

Which, of course, those of us who invested 10 years of our lives to  
learn how to spell it (most other languages take less than half that)  
wouldn't like.

> drive on the left or on the right.

Hey, I drive on the right, speak a language that's (arguably) more  
elegant than English, use tab notation rather than traditional  
notation and play a keyboardless instrument, use the metric system and  
run IPv6. I'm doing my part.  :-)

> Elegance in keyboards would involve replacing the QWERTY keyboard

Actually that's a myth. There was a lot of competition between various  
keyboard layouts in the early days of machine writing and QWERTY won  
for a reason. Talk that Dvorak is better comes from people with a  
financial interest and uncritical reporting. Of course it's possible  
to improve on QWERTY (or Dvorak) using more advanced systems where the  
system knows the language and multiple keys are pressed at the same  
time (velotype) but that's not a simple layout change.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 18 Sep 2008 09:29:53 +1200

I'd like to point out that "rejecting" shim6 isn't within the
ISPs' or even site operators' remit. The real issue is whether
o/s implementors care to include shim6 code in their stacks or
not. If they do, shim6 will self-deploy; if they don't, it won't.

(Pretty much the same is true of SCTP, except that it also
requires transport-using applications to be updated too. And no
doubt the same will be true of multipath transport when some
code comes along.)

What the ISPs have done, and some large users too, is state
that shim6 doesn't solve all their problems. Well, it wasn't
intended to -- to be blunt, it's the best that benighted protocol
designers could do absent any sign of change in the routing
system. Viewed from the multi6 WG in 2005, it seemed axiomatic that
we couldn't expect any change in routing on any reasonable timescale.
Three years later, we have shim6 code but no change in routing.

We should probably use RRG cycles to figure out how to change
routing.

    Brian

---

From: Jari Arkko <[email-masked]>
Date: Thu, 18 Sep 2008 20:15:54 +0200

> We should probably use RRG cycles to figure out how to change
Indeed.

Jari

---

From: Robin Whittle <[email-masked]>
Date: Mon, 22 Sep 2008 17:34:47 +1000

Short version:  Exploration of how SHIM6's host-based approach
                to multihoming failure detection and service
                restoration decision making is messier than
                LISP's, APT's and TRRP's approach.  Both
                approaches are inflexible and monolithically
                combined into their respective systems compared
                to Ivip's approach in which these functions are
                performed independently of the Ivip system, in
                any way the end-user network wishes to do so.

Hi Iljitsch,

You wrote:

>>> The shim6 effort was well under way at that point but not yet
>>> mature enough that it was possible to know whether it would
>>> solve the problem.
> 
>> The functional goals of SHIM6 are well known and have been for
>> a while: to provide host-based multihoming.
> 
> multihomed. End-users shouldn't concern themselves too much with
> the details of technology that aren't of immediate importance to
> them.

It is important to the administrator of an end-user network whether
portability and multihoming (TE too) can be controlled purely at a
router or some other central location, or whether every host in the
network needs to be coordinated in some way to achieve this.

For instance, each SHIM6 host does its own reachability testing and
multihoming service restoration decisions.  This is extraordinarily
messy compared to the Ivip approach of the administrator of the
network organising (typically via a company which specialises in
this) a single multihoming monitoring system for the entire network.
 That system uses whatever tests the administrator desires, and
makes decisions according to whatever criteria they require.  Then
that system changes the mapping, so the traffic flows to the new ETR
within a few seconds.

Ivip modularly separates the tasks of testing reachability and
deciding which ETR to use from the core-edge separation scheme
itself.  Ivip enables these functions to be performed however the
end-user wishes, including with as much or as little sophistication
and test packet overhead as they desire.  This will be a lot simpler
to debug and optimise for each multihoming situation than any system
such as LISP, APT, TRRP and I think Six/One Router - which integrate
these functions monolithically into the ITR (or translation router)
functions of the core-edge separation system itself.

SHIM6 is even worse - it builds these functions into the hosts.

With Ivip there is a single failure monitoring system and a single
decision process to handle the entire end-user network.

With LISP etc. these functions have to be done with very limited
flexibility by as many ITRs as are currently tunneling packets to
hosts in this end-user network.

If there were 100 such LISP etc. ITRs, this is 100 separate sets of
test packets to the ETR, with very limited decision flexibility, due
to the necessarily limited functionality these protocols can require
in all ITRs.  With Ivip, there need only be one set of test packets,
and one highly flexible, decision system - which can be constructed
in any manner at all, since it is not at all part of the Ivip system.

With SHIM6, the situation will often be worse than with LISP etc.
If there were 100 ITRs each handling packets from 10 hosts which are
sending packets to this one edge network, then this means there are
1000 hosts all doing their own reachability testing, making their
own decisions etc.

A system which completely isolates the hosts from multihoming,
portability etc. by providing them with a consistent IP address at
all times is arguably a lot simpler to run and debug than one which
pushes a lot of the complexity, responsibility and potential
problems into the host operating systems themselves.

> Which is orthogonal to the discussion at hand: PI in IPv6.

OK.

>> Also, people want address space which is portable between ISPs
>> - so they don't have to renumber their network, DNS etc. when
>> selecting a new ISP.
> 
> Shim6 could have given them that and maybe still could in the
> future by using a separate ID space.

I don't see how SHIM6 could be used for providing portable address
space.

>> I believe these are perfectly reasonable desires and needs.
>> SHIM6 and IPv6's automated router and host renumbering just
>> doesn't do what most end-user networks need.
> 
> No, it doesn't do what they want. Like trained monkeys these
> users expect to be able to continue to do the same thing forever
> even though circumstances are changing. (Actually that's unfair
> to monkeys, humans and flies are pretty much the only creatures
> that keep doing the same thing expecting different results.)

I think that the administrators of end-user networks are perfectly
justified in seeking a solution which isolates hosts from any
effects of multihoming service restoration, moving to another ISP etc.

>> LISP, APT, Ivip, TRRP and Six/One Router are attempts at
>> providing end-user networks with what they want and need:
>> portability and network-based multihoming with PI space, but in
>> a scalable way.
> 
> One or two of these solutions may be turned into something
> workable but that's not to say that that would be the best
> possible solution.
> 
> Mark my words: a loc/id split is going to open a big can of worms
> that will make life harder for a long time. 

Please document your concerns in detail.  Core-edge separation
schemes do involve extra complexity - but they do not require
changes to host operating systems or applications, and they have the
ongoing property of not involving hosts in portability, multihoming etc.

> Loc/id overload isn't so bad.

Do you mean the current situation where an IP address is arguably
used to define both the end-point device and how the packet needs to
be forwarded across the Internet to reach that endpoint?  If so, it
is bad because it leads to a scaling problem when more and more
end-user networks get what they need: PI space, in the only way
currently possible: via a BGP advertised prefix.

>> I think it would be an unjustifiable over-reaction to require
>> that all hosts change their stacks and applications just to
>> solve a scaling problem in the DFZ
> 
> There's no need to update applications for shim6 (IPv6
> applications are already expected to try all addresses, that's
> all that shim6 needs).

OK - has SHIM6 been extensively tested with a range of IPv6
applications?

Since reachability testing is built into the SHIM6 functionality of
each host, how are end-user network administrators going to be happy
with this, since they are likely to have a variety of desires about
what constitutes an "outage", what should cause packets to flow one
way or another etc.?

> something that's in wider use: TCP congestion control. Hosts need
> to keep state anyway so solving the problem there is essentially
> free. It would be stupid to ignore that.
> 
>  actually a fairly easy way to solve problems because only a
> handful of organizations need to make investments (the host OS
> vendors).

Yes, but there is a huge hurdle to overcome if the new system only
provided benefits when communicating with another upgraded system.
 SHIM6 still can't provide portable address space.  Nor could SHIM6
be the basis for a global approach to mobility such as this:

  http://www.firstpr.com.au/ip/ivip/TTR-Mobility.pdf

>> We just want the Internet to keep working well with minimal
>> disruption.
> 

That is rather disruptive, since there's almost no-one at home on
the IPv6 Internet.

> Oh wait, what you really want is to not have to deal with the
> problem. Fine. But then you pay the price in some other way.

We have to find a way of dealing with the scalability problem - and
then the shortage of address problem - which most people will want
to adopt due to the immediate benefits it brings.  IPv6 is currently
a non-starter because it does not provide connectivity to all hosts
(or at least all hosts used for any purpose important to Internet
users in general) - which IPv4 does.

> Which, of course, those of us who invested 10 years of our lives
> to learn how to spell it (most other languages take less than
> half that) wouldn't like.
> 
>> Elegance in driving would involve all countries agreeing to
>> either drive on the left or on the right.
> 
> Hey, I drive on the right, speak a language that's (arguably)
> more elegant than English, use tab notation rather than
> traditional notation and play a keyboardless instrument, use the
> metric system and run IPv6. I'm doing my part.  :-)

Very cool!  If the majority of Internet users were like you, they
would be keen to invest the time end effort into making a big switch
to another network such as IPv6 even when it is only partially
functional for a few years until everyone else gets around to using
it fully.

Our task is to come up with a system which solves the routing
scalability problem which ordinary folks will adopt for their own
immediate reasons - since most of them won't know or care about
routing scalability.

 - Robin

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 22 Sep 2008 13:12:30 +0200

> network needs to be coordinated in some way to achieve this.

Yes, but it's unimportant whether this control is an inherent part of  
a centralized design or is simply added to a decentralized design  
because it's an additional requirement. There is nothing stopping you  
from coming up with a shim6 control protocol.

> messy

It's also extremely effective, because this actually tells you whether  
you have connectivity, while the presence of a prefix in BGP doesn't  
tell you much because the unreachability will usually be aggregated  
away, and it's scalable. It also avoids the need for continuous  
keepalives that is present in many protocols.

> 1000 hosts all doing their own reachability testing, making their

Anarchy!  :-)

> problems into the host operating systems themselves.

Yes, but our architecture is such that it's impossible to communicate  
this type of information between the network and the hosts, so a lot  
of information is lost if you take this functionality out of the host.  
We do congestion control in hosts for a reason.

> I don't see how SHIM6 could be used for providing portable address

shim6 just negotiates extra addresses and switches to a different one  
when necessary. If you can boot strap the address negotiation without  
requiring an initially working address pair then you could simply use  
ULAs or some such as the identifiers. However, this would require an  
id->loc lookup of some sort, which is something we were reluctant to  
include in shim6.

> effects of multihoming service restoration, moving to another ISP etc.

Sure, and it's perfectly reasonable to go live in the desert and pump  
up water from deep beneath the surface to water your lawn and run an  
air conditioner 24/7 to keep cool.

Some things just don't work in the long run.

> currently possible: via a BGP advertised prefix.

Well, if we accept that then basically a loc/id separation is a way to  
avoid renumbering. Renumbering is only painful when you're actually  
doing it. Loc/id is painful for every packet and for every session...

> what constitutes an "outage", what should cause packets to flow one

Sorry, that doesn't make any sense to me.

> provided benefits when communicating with another upgraded system.

True.

> That is rather disruptive, since there's almost no-one at home on

There will be if everyone updates... The disruption will come to IPv4  
as it runs out of addresses. At that point there will be people who  
are on v4 and not of v6 and people who are on v6 but not on v4. That's  
not good. People who add v6 today can keep running v4 so there is no  
disruption.

> immediate reasons - since most of them won't know or care about

Right. But we should expect users to make _some_ changes to their  
routine if this is really necessary to make our efforts succesful.  
Maybe renumbering once a month is asking too much, but rejecting a  
solution just because it does some stuff in a decentralized manner is  
the other extreme. (I'm not arguing that shim6 solves everything for  
everyone, though, just that IPv6 PI was adopted way too easily.)

---

From: Robin Whittle <[email-masked]>
Date: Tue, 23 Sep 2008 13:27:20 +1000

Short version:   More on SHIM6's decentralises approach to
                 multihoming reachability testing, - and the
                 somewhat less decentralised approach of using
                 ITRs to do this (LISP, APT and TRRP) compared to
                 Ivip's more centralised approach.

                 Also, do we upgrade the "routing system" to
                 scalably do what it already does: provide stable
                 IP addresses to networks and hosts - or do we
                 assume this is impossible or undesirable, and
                 insist that host operating systems and applications
                 change to cope with IP addresses which could change
                 at any time?

                 SHIM6 does not provide portable address space.
                 Nor can it do reliable address referrals without
                 application changes, since it would be necessary to
                 refer with two or more IP addresses.

Hi Iljitsch,

You wrote:

> Yes, but it's unimportant whether this control is an inherent part of a
> centralized design or is simply added to a decentralized design because
> it's an additional requirement. There is nothing stopping you from
> coming up with a shim6 control protocol.

Adding a control protocol for altering the SHIM6 behavior of all
hosts in a network would be an extra layer of complexity - which is
important.  It would also involve scaling problems in larger
networks, and require either the management system to discover all
hosts, or all the hosts to be configured to auto-discover the
management system.  It would also presumably involve changes to SHIM6.

> away, and it's scalable. 

In principle it could be more effective than monitoring the
reachability of an edge network from a single location.  When I
suggest that most Ivip end-user networks would pay a special
multihoming monitoring company to continually monitor connectivity,
and change the mapping of their micronets if an outage occurred - I
also suggest that these companies would monitor from multiple locations.

That isn't necessarily as robust as each sending host doing its own
reachability testing, but it scales a lot better.  It also enables
the end-user network to precisely determine how their network's
reachability is probed, what criteria is used when deciding there is
an outage - and what mapping changes to make.  With SHIM6 or the
non-Ivip core-edge separation schemes, the testing and decision
making functionalities are implemented in the sending hosts, or the
ITRs, respectively.

> It also avoids the need for continuous keepalives that is present in 
> many protocols.

But how could an application protocol know it is running over SHIM6
and therefore decide it doesn't need to use keepalives?  How could
SHIM6 know what level of continual probing to conduct in the absence
of actual traffic packets?  The recent paper:

  http://conferences.sigcomm.org/hotnets/2008/papers/18.pdf

points out this difficulty: many protocols don't have a continual
flow of packets, so unless there is continual (inefficient and
unscalable) probing by the ITR (or by the SHIM6 layer of a sending
host) then the reachability problem is only discovered some time
after it occurs, when the sending host has data to send.  Then,
there would be seconds of delay while the application gets no
response and the SHIM6 layer somehow decides that the current
address doesn't work, before the application would try again (I
think this is how it would work) and the packet could be sent to
another address, via another ISP, which did work.

With Ivip, the multihoming monitoring system will detect the problem
pretty quickly - probably within seconds - and change the mapping,
again within a few seconds.

>> 1000 hosts all doing their own reachability testing, making their

Costs, inefficiency, unscalability . . . :{

> this type of information between the network and the hosts, so a lot of
> information is lost if you take this functionality out of the host. We
> do congestion control in hosts for a reason.

I don't understand this.

> id->loc lookup of some sort, which is something we were reluctant to

I don't clearly understand this.  From all I know, SHIM6 does not
enable a network to have portable address space.  So moving to
another ISP requires changes to all DNS entries, and to any other
place where IP addresses are used.  This includes any ACLs, I guess.

With a stable IP address, which is portable no matter which ISP is
used, (the aim of Ivip, LISP, APT, TRRP and Six/One Router) there is
no need for changing DNS, no problem with DNS caching times and no
changes to any ACLs anywhere, such as some filtering arrangement
restricting VPN access to remote hosts with particular IP addresses,
or in particular networks.  Furthermore, IP address referrals work fine.

SHIM6 can't do IP address referrals and support multihoming at the
same time, since by referring a single IP address to some other
host, by the time that other host uses that IP address, the ISP link
that host relies upon might be down.

Address referrals with SHIM6 would involve two or more - an unknown
number - of IP addresses, so the host which receives this list would
try them until it found one which worked.  This would require
application changes.

> Sure, and it's perfectly reasonable to go live in the desert and pump up
> water from deep beneath the surface to water your lawn and run an air
> conditioner 24/7 to keep cool.
> 

You and some other folks can't imagine the Internet running
indefinitely with an architectural enhancement such as Ivip which
gives each end-user network its own, stable, portable, IP address or
subnet of addresses.  I can.

I think it is the proper role of the routing system - with proposed
enhancements - to give each host or network stable address space,
including for portability between ISPs, for multihoming and for
mobility.  I think it can be done and that it would be desirable to
do so, compared to the alternatives.

You think it can't work out this way - that there is no
architectural enhancement which could or should remove from hosts
and end-user networks the need to cope with arbitrarily changing
addresses, such as due to multihoming failure, the need to use
another ISP or mobility.

> doing it. Loc/id is painful for every packet and for every session...

Renumbering is disruptive, costly and in many cases completely
impractical.  A core-edge separation scheme such as Ivip does
involve extra bits of equipment and a mapping system.  It doesn't
necessarily involve encapsulation overhead and the PMTUD problems
this brings.  My page:

  http://www.firstpr.com.au/ip/ivip/

lists two approaches to forwarding, without any overhead or PMTUD
problems, for IPv4 and IPv6 - but these require upgrades to DFZ and
many other routers.  Maybe that won't be so hard in the time we have
available - and would be easier than creating more complex ITRs and
ETRs for encapsulation.

My approach is to upgrade the routing system (broadly defined) to do
what it currently does - give hosts and end-user networks stable IP
addresses - but to be able to do so scalably for 10^10 or whatever
separate end-user networks having portable, multihomable "Scalable
PI" address space.

Your approach is a retreat from the original division of labour in
the Net, on the basis that we can't possibly expect the "routing
system" to do this - therefore all hosts must change.  Host
operating systems and applications must change so they can cope with
 having a changed IP address at any time, or at least with
multihoming having two or more addresses, with no guarantee that all
 of them will be reachable.   This does involve application changes,
since with the approach you suggest, there can't be reliable
referrals via a single IP address, which is the current arrangement.

> Sorry, that doesn't make any sense to me.

A multihomed network may have ISP A which is cheap, fast and
generally reliable as its preferred link to the Net.  It may have a
slow, costly, ISP as a backup link.  With SHIM6 the reachability
testing and multihoming service restoration decision making is done
by each correspondent host, in a way which is codified into the
SHIM6 standards.  With LISP, APT, TRRP and Six/One Router, these
things are built into the ITR (or its Six/One Router equivalent -
the translation router).  The mapping system gives the end-user
network some prioritisation of whether to use ISP A's ETR or ISP B's
ETR, but it doesn't give any sophisticated control over how these
ITRs probe for connectivity, or how quickly they make a decision to
use ISP B's ETR.  Nor is there a way the end-user network can
specify how quickly to go back to using ISP A's ETR, and the probing
and decision making criteria for doing that.

With SHIM6 and the non-Ivip core-edge separation schemes, the
processes of multihoming reachability testing and service
restoration decision making are monolithically integrated into these
schemes.  With Ivip, these are completely outside the Ivip system -
a modular approach more in keeping with good engineering practice.

With Ivip, the remote hosts and the ITRs are not at all involved in
probing reachability, or making any decisions about multihoming
service restoration.  That is done by the end-user, or more likely
by some company they hire to probe their network and send mapping
changes in the event of an outage.  With Ivip, the probing and
decision making arrangements are not at all part of the Ivip system
and so can be customised without limit to the desires of each
end-user network.  The end-user network administrator could choose
the frequency of probe packets, the number of vantage points in the
Net from where these probes were sent etc.

The administrator could precisely determine the decision criteria.
For instance, if the link from ISP A occasionally became congested,
they could have the decision process adjusted so it did not change
the mapping to use (expensive) ISP B's ETR instead unless there was
a more prolonged absence of probe packet acknowledgements.

With SHIM6 or the non-Ivip core-edge separation techniques, this
sort of sophistication would not be possible.

OK.

> There will be if everyone updates... The disruption will come to IPv4 as
> it runs out of addresses. At that point there will be people who are on
> v4 and not of v6 and people who are on v6 but not on v4. That's not
> good. People who add v6 today can keep running v4 so there is no
> disruption.

I don't think there is going to be a sudden crunch with IPv4 - just
an increasingly high economic value placed on address space,
together with more economic pressure to use it efficiently, and to
trade it back and forth so none of it goes unused.  I agree it can't
go on forever, but I think we are a long way from a situation where
IPv6 is attractive to most end-users.  The exceptions would be
cell-phone end-users, who have built-in applications and who can
access selected IPv4 things via HTTP gateways etc.

I think that if mobile operators do choose en-masse to give
cell-phone users public IP addresses (they may not, wanting to keep
them on NAT and make it hard for them to run their own VoIP
programs), then this will be the first substantial adoption of IPv6.

> routine if this is really necessary to make our efforts succesful. Maybe
> renumbering once a month is asking too much, but rejecting a solution
> just because it does some stuff in a decentralized manner is the other
> extreme. (I'm not arguing that shim6 solves everything for everyone,
> though, just that IPv6 PI was adopted way too easily.)

OK - the debate continues!

  - Robin

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 23 Sep 2008 09:49:17 +0200

On 23 sep 2008, at 5:27, Robin Whittle wrote:

>> it's unimportant whether this control is an inherent part of a

> important.

A good solution that is somewhat complex is still a good solution,  
especially if there's no good simple solution. Things like SNMP and  
IPsec are so insanely complex that the IETF really can't play the  
complexity card anymore. (It seems any protocol with the S for simple  
in its name is cursed in this regard...)

> and therefore decide it doesn't need to use keepalives?

Applications don't need keepalives.

> How could
> of actual traffic packets?

Why would you need to probe if there is no actual traffic? Suppose you  
have an IMAP session and you go drink coffee at 10 until 10.20 and  
there's no new mail in the interim. At 10.01 the connection goes down  
and at 10.19 it comes back up. Any time spent repairing that  
connection would have been wasted because connectivity wasn't  
necessary at that point.

The shim6 REAP protocol doesn't send keepalives when the session is  
progressing without trouble and it also doesn't send keepalives when  
the session is idle. So it basically only sends keepalives when there  
is only traffic in one direction or around the time a session  
transitions from active to idle.

> The recent paper:

>  http://conferences.sigcomm.org/hotnets/2008/papers/18.pdf

Hm, I'll have to read that when my connectivity comes back. (Things  
are rather quiet here at the university with the connection to the  
rest of the world severed...)

> the reachability problem is only discovered some time
> after it occurs, when the sending host has data to send.

Right.

> Then,

Right. Obviously it's always good if things can be fast but a few  
seconds delay in this relatively rare case isn't a problem, in my  
opinion.

> I don't understand this.

In many cases it's better for higher layers to actually know what's  
going on than to just assume that everything is alright and get  
confused when it isn't. But on the other hand interfaces between  
layers, especially layers implemented in different systems, are rather  
expensive. For instance, congestion control works better if the  
routers and hosts have a common understanding of how much buffering  
will happen in the network, but there is no way to communicate this  
information so there is sort of an arms race between larger router  
buffers and larger send/receive buffers.

> place where IP addresses are used.  This includes any ACLs, I guess.

Yes, that is correct for shim6 is it's currently defined. But you  
could easily (ok, maybe not so easily) add a mapping system to shim6  
and then it COULD support all of this.

> system" to do this - therefore all hosts must change.

shim6 was created with MANY multihomers in mind (potentially all  
internet users). There is no way all of those people can run BGP or  
something like it. Now obviously there is a point at which shim6  
starts becoming impractical. There is also a level of PI prefixes that  
the routing system can support without trouble. However, the current  
rules for PI don't take either of these boundaries into account so  
EVERYONE is going after PI which we know won't scale and the interest  
in shim6 has dried up so shim6 may not become available to those for  
which it would be the best solution, either. This is all a result of  
ARIN jumping the gun on IPv6 PI.

> Host
> of them will be reachable.

Get over it, this is a result of choices made in the IPv6 design a  
long time ago, regardless of the presence of shim6.

> This does involve application changes,

Since when are referrals reliable in the first place? They're a big,  
big mess and the IETF hasn't even considered starting a cleanup effort.

>> The disruption will come to IPv4 as

> trade it back and forth so none of it goes unused.

There is plenty of IPv4 address space for the forseeable future, it's  
just not distributed evenly. Redistribution efforts haven't had  
spectacular results in the past, and I don't expect them to be  
succesful enough in the future to continue to meet current demands  
after we run out of free IPv4 address space. So a crunch is coming. I  
agree that it's probably not going to be a very sudden one (although  
we can't rule that out, no telling what strange things people will do  
when crunch time arrives) but that doesn't really matter: as soon as  
the unused space is gone, it's a given that getting any non-trivial  
amount of IPv4 address space is going to be harder / more expensive  
every successive time.

> but I think we are a long way from a situation where
> IPv6 is attractive to most end-users.

IPv6 is the ultimate NAT traversal mechanism. It will happen,  
especially as it becomes easier for end-users to run IPv6 because it's  
available in more of their equipment and even turned on by default.

---

From: Robin Whittle <[email-masked]>
Date: Wed, 24 Sep 2008 13:24:40 +1000

Hi Iljitsch,

You wrote:

> especially if there's no good simple solution. 

I agree, but I don't think that SHIM6 is a good solution anyway,
even if there was some centralised control of all the hosts.

> Things like SNMP and
> complexity card anymore. (It seems any protocol with the S for simple in
> its name is cursed in this regard...)

It is probably getting harder and harder to design anything simply
and still satisfy the increased number of people involved, or to
work properly in the variety of circumstances which exist.
Sometimes, in an effort to avoid overlapping enhancements to the
Net, we try to do several things at once.  This might be a good
thing if two birds really can be killed with one stone - but only if
the stone is not inordinately complex.

> Applications don't need keepalives.

I am confused by this but it is probably not worth pursuing.

> and at 10.19 it comes back up. Any time spent repairing that connection
> would have been wasted because connectivity wasn't necessary at that point.
> 
> progressing without trouble and it also doesn't send keepalives when the
> session is idle. So it basically only sends keepalives when there is
> only traffic in one direction or around the time a session transitions
> from active to idle.

OK - if there was a temporary loss of connectivity, then this has
saved a lot of trouble.   However, if you want to use IMAP when the
connection for the last used address is down, then you have to wait
while your host's SHIM6 layer figures out it is dead and tries
another address instead.  With Ivip, there would be no such delay or
extra traffic testing connectivity etc, because the end-user
provided multihoming monitoring system would already have detected
the problem and fixed it already by changing the mapping.

> seconds delay in this relatively rare case isn't a problem, in my opinion.

OK - if it is a multihoming link failure, hopefully this won't
happen too often.

> going on than to just assume that everything is alright and get confused
> when it isn't. But on the other hand interfaces between layers,
> especially layers implemented in different systems, are rather
> expensive. For instance, congestion control works better if the routers
> and hosts have a common understanding of how much buffering will happen
> in the network, but there is no way to communicate this information so
> there is sort of an arms race between larger router buffers and larger
> send/receive buffers.

OK - this seems to be an example of it being desirable for the host
applications to know something about the network.

Still, for connectivity restoration in a multihoming link failure
situation, I still prefer the idea of the end-user having their own,
potentially highly customised, outage detection system with their
own preferred decision making system to restore connectivity with a
single action: changing the mapping to send packets to an
alternative ETR which is known to work.  I much prefer this fast,
end-user controlled, arrangement to having thousands of ITRs or
sending hosts mucking around, making their own probings and
decisions, all independently, with little or no ability of the
end-user network's administrator to control, view or debug these
actions due to the probe and decision algorithms being
monolithically encoded into the core-edge separation system.

> Yes, that is correct for shim6 is it's currently defined. But you could
> easily (ok, maybe not so easily) add a mapping system to shim6 and then
> it COULD support all of this.

Then it wouldn't so much resemble SHIM6 as Ivip's option for ITR
functions in the sending hosts!

> something like it. Now obviously there is a point at which shim6 starts
> becoming impractical. There is also a level of PI prefixes that the
> routing system can support without trouble. However, the current rules
> for PI don't take either of these boundaries into account so EVERYONE is
> going after PI which we know won't scale and the interest in shim6 has
> dried up so shim6 may not become available to those for which it would
> be the best solution, either. This is all a result of ARIN jumping the
> gun on IPv6 PI.

Perhaps the fact that SHIM6 has not been widely supported is in an
indication that in the judgement of most end-user networks, it is
not and cannot be an adequate solution, or the best.

> Get over it, this is a result of choices made in the IPv6 design a long
> time ago, regardless of the presence of shim6.

But those choices only make sense if SHIM6 is widely adopted.

I think there are a number of valid reasons for keeping the routing
system (broadly defined, including a future core-edge separation
scheme) doing its current job of giving each host a stable IP
address, rather than giving up on this and expecting hosts to
maintain reliable communications with other hosts in an environment
of multiple IP addresses, any one of which could become unusable
without prior notice.

> Since when are referrals reliable in the first place? They're a big, big
> mess and the IETF hasn't even considered starting a cleanup effort.

For host A to be able to tell host B to send packets to host C,
based solely on host C's IP address is a perfectly simple and robust
technique - as long as host IP addresses are relatively stable.
With the non-PI approach of IPv6, which can only be made to work at
all with SHIM6 being widely adopted, you can only do this robustly
via either:

1 - A specifying C to B by way of C's DNS name.

2 - A specifying C to B by way of C's potentially multiple
    addresses, of which A may only know one.

Both are a lot more complex than using a single IP address - and for
any applications which use a single IP address now, there would have
to be significant application and protocol changes to handle either
of the two options above.   Also, option 1 assumes that C has a DNS
name.  Conventional IP address referrals work independently of DNS -
for any host at all.

> spectacular results in the past, and I don't expect them to be succesful
> enough in the future to continue to meet current demands after we run
> out of free IPv4 address space. 

Past redistribution attempts, of which I know little, were not
motivated by the increasing financial impetus to gain precious IPv4
space - so I expect there will be considerable business and
technical innovation in making the most of the increasingly
fragmented and populated IPv4 space, since IPv6 remains a bleak and
empty Internet for most end-users.

The most obvious technical solution is a core-edge separation scheme
which can slice and dice address space down to single IP addresses
in a scalable manner: LISP with PTRs or Ivip.  I think APT can only
do this if the current proposal is changed so there is a single APT
island - see: http://psg.com/lists/rrg/2008/msg02569.html .  I am
not sure how TRRP would support packets from non-upgraded hosts with
micronets longer than /24.

> So a crunch is coming. I agree that it's
> probably not going to be a very sudden one (although we can't rule that
> out, no telling what strange things people will do when crunch time
> arrives) 

. . . such as innovation and trade . . .

> but that doesn't really matter: as soon as the unused space is
> gone, it's a given that getting any non-trivial amount of IPv4 address
> space is going to be harder / more expensive every successive time.

I agree.  It is hard to predict how this will work out.

> IPv6 is the ultimate NAT traversal mechanism. It will happen, especially
> as it becomes easier for end-users to run IPv6 because it's available in
> more of their equipment and even turned on by default.

I still think we are a long way from this, and that in the meantime,
there will be an increasingly serious routing scaling problem in IPv4.

I don't have a clear idea about how many DFZ routes constitutes an
unworkable problem.  Probably there is no clear point where things
get unacceptably bad - it just gets more expensive and slows down
BGP, due to the increasing number of changed advertisements which
typically must be sent, received, interpreted and perhaps propagated
further (loosely speaking) whenever one link goes down, due to there
being far more prefixes advertised in total, than at present.

  - Robin

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 24 Sep 2008 11:54:31 +0200

> I am confused by this but it is probably not worth pursuing.

:-)

Life is easier if you just go with the flow and don't want to control  
everything. As an app, when you have data, just send it. Either it  
will get there or not. Having sent keepalives doesn't make the former  
more likely or even accurately predict the latter.

> With Ivip, there would be no such delay or

How does ivip know I have an IMAP session that has been idle for 20  
minutes?

> I still prefer the idea of the end-user having their own,
> single action

Right, that's exactly what REAP does.  :-)

>> could

> functions in the sending hosts!

The shim6 control messages could be reused with very few changes.

>> long

> But those choices only make sense if SHIM6 is widely adopted.

Many hosts that run IPv6 will also run some kind of IPv4, so that  
means that they'll have at least an IPv6 link local address, a global  
IPv6 address and an IPv4 address. So apps must be prepared to cycle  
through all addresses and know which one to use for referrals anyway,  
regardless of whether the host has multiple global IPv6 addresses.

> rather than giving up on this and expecting hosts to

This is current practice today because DHCP servers may give you a  
different IPv4 address at any time. When that happens applications  
like Apple's Mail simply reconnect to the server without interrupting  
the user's work.

Other apps need some kind of manual restart, though.

> 1 - A specifying C to B by way of C's DNS name.

That doesn't happen in practice because home users have no control  
over their DNS names.

>    addresses, of which A may only know one.

Current protocols are unable to do this.

> of the two options above.

Yes. But like I said, hosts have multiple addresses and/or don't know  
their own address because they're behind NAT today so there is no easy  
way out.

>> succesful

> Past redistribution attempts, of which I know little,

"Please return unused addresses or at least sign the ARIN contract".

> were not
> space

I believe that freeing up considerable amounts of address space is  
more expensive than the price the big address users are willing to pay  
so if a market is allowed it will only work for small blocks where  
there isn't much of an issue in the first place.

> so I expect there will be considerable business and
> fragmented and populated IPv4 space,

If the big users aren't going to string together lots of small blocks  
when they can't get big blocks anymore the fragmentation won't  
increase beyond what's already happening now.

> in a scalable manner: LISP with PTRs or Ivip.

No, the most obvious technical solution is RFC 2460. But apparently  
obviousness isn't much of a consideration.

I'm fine with having host level mobility in a map and encap scheme as  
long as there is a hierarchy, I don't want to see 3.7 billion /32s at  
the top of the addressing hierarchy.

> unworkable problem.

We cleared the 250k hurdle without too many problems, so I guess we'll  
be ok until around 500k if the growth doesn't happen too fast.

But big ISPs have lots of internal prefixes, so they may run into some  
limit at any time and start filtering longer prefixes, like Sprint 10  
years ago.

> Probably there is no clear point where things
> get unacceptably bad

Because BGP is a real time system, when you do X and you break it, you  
quickly undo X and it will probably work again.

---

From: Robin Whittle <[email-masked]>
Date: Fri, 26 Sep 2008 20:23:49 +1000

Hi Iljitsch,

You wrote:

> everything. As an app, when you have data, just send it. Either it will
> get there or not. Having sent keepalives doesn't make the former more
> likely or even accurately predict the latter.

I understand keepalives are there for the benefit of NAT - to keep a
session in the NAT cache even if there is no actual data to be sent.

> How does ivip know I have an IMAP session that has been idle for 20

Ivip's heuristics have already classified your traffic pattern and
scored it in the 99th percentile in the "advanced technology geek"
dimension.  Such folks are never separated from their IMAP server
for long.

The last 20% of: http://psg.com/lists/rrg/2008/msg02585.html has a
description of how an end-user network would probably hire a company
to continually monitor reachability of their network, and make
decisions about changing mapping.

> Right, that's exactly what REAP does.  :-)

OK - I see this is your work with Jari Arkko and others, which I
will read:

   http://tools.ietf.org/html/draft-ietf-shim6-failure-detection-13
   http://www.shim6.org/47120443.pdf

The approach I describe for Ivip is very different, since it does
not work with hosts which are part of any communication.  That may
be a problem in terms of it not detecting every possible constraint
on packets being sent between two communicating hosts.  However, I
think it scales a lot better than having thousands of hosts all
doing reachability testing with one host.

If the barrier to reachability is close to the sending host and its
ITR, rather than the destination ETR, it is probably not much good
tunneling to an ETR in another ISP, since at the sending host's end,
the encapsulated packets would probably follow the same path.  This
is not absolutely true, of course.  There would be some outages
which a purely host-based or ITR based reachability system would
detect and successfully solve by selecting a different destination
address or ETR address - and which would not be detected, or should
not be detected, by the separate scheme I propose.

> The shim6 control messages could be reused with very few changes.

Maybe - I am keen to pursue a system which does the work in the
network, and leaves the hosts alone, except for consenting hosts
which want to integrate an ITR function in themselves.

> Many hosts that run IPv6 will also run some kind of IPv4, so that means
> that they'll have at least an IPv6 link local address, a global IPv6
> address and an IPv4 address. So apps must be prepared to cycle through
> all addresses and know which one to use for referrals anyway, regardless
> of whether the host has multiple global IPv6 addresses.

You are saying that applications will have to change to work with
IPv6.  But I am saying no-one uses IPv6 much because many
applications, printers and other devices don't work with it, and
because there is no-one much home on the IPv6 Internet yet.

> different IPv4 address at any time. When that happens applications like
> Apple's Mail simply reconnect to the server without interrupting the
> user's work.

In practice, I understand DHCP servers typically don't give out a
different IP address for no good reason.  I understand many DSL
customers often have the same IP address for months or maybe years.

It is one thing for DHCP to be capable of giving a host a different
IP address from time to time.  It is another to try to get everyone
to change to an Internet in which multihomed hosts have two or more
addresses, one of which could fail at any time, with the host stack
and applications being expected to work with the other one.

OK.

> That doesn't happen in practice because home users have no control over
> their DNS names.

Yes, and C doesn't necessarily have a DNS name.  NAT makes a mess of
referrals, of course.

> Current protocols are unable to do this.

Yes.

> their own address because they're behind NAT today so there is no easy

I agree.

>> were not motivated by the increasing financial impetus to gain 
>> precious IPv4 space
> 
> I believe that freeing up considerable amounts of address space is more
> expensive than the price the big address users are willing to pay so if
> a market is allowed it will only work for small blocks where there isn't
> much of an issue in the first place.

The big players already have vast tracts of IPv6 space - but it is
currently no use for creating services which customers want to pay
for.   There is only one kind of address space which has any
commercial utility at present - IPv4 space.  The big ISPs may not
want to pay X dollars or get the space in smallish chunks compared
to what they got for lower costs in bigger chunks in the past.
However, until most end-users have IPv6 connectivity, stacks and
applications, IPv4 space is the only kind which can be used for
saleable services - not counting cellphones and customers in captive
situation where the only available ISP only offers IPv6, which would
be a rare or non-existent situation.

> when they can't get big blocks anymore the fragmentation won't increase
> beyond what's already happening now.

I can't predict what will happen - but I think there is a long way
to go before IPv6 space can be used for a saleable Internet service.

> obviousness isn't much of a consideration.

If "technical" means not worrying about unfortunate details such as
users needing to communicate with all other users, all users being
on IPv4 and IPv6 not being able to inter-operate with IPv4, then I
agree.

> the top of the addressing hierarchy.

It may well come to that.

The barriers to mass migration from IPv4 are extraordinarily high.

> limit at any time and start filtering longer prefixes, like Sprint 10

I think this was discussed in the last few months.  I can't see how
an ISP would willingly make itself unreachable to some subset of the
Net rather than buy newer routers.  Such actions would be seized
upon by competitors and the customers would remember it for years
afterwards.

> quickly undo X and it will probably work again.

My guess is that the pressure of rising DFZ routing table numbers
would be felt as having to either buy a new router, or limit the
number of neighbours it has, depending on its RAM, CPU power and
number of non DFZ routes it handles - assuming the FIB is not the
limiting factor.

 - Robin